Remote
Remote
Mid Level
Full Time
about 1 month ago
💰$130,000 - $140,000
AIPlatform EngineerRemotePythonRuby on RailsRAGLLMEvaluationObservabilityA/B Testing
Requirements
- •Experience building and evaluating AI systems in production including RAG pipelines, search/retrieval systems, LLM-powered applications, and evaluation frameworks
- •Strong proficiency in Python for prototyping and experimentation
- •Openness to learning Ruby on Rails and integrating with existing codebase
- •Comfortable building infrastructure and tooling such as evaluation pipelines, dashboards, and data processing
- •Deep understanding of RAG architecture including chunking strategies, embeddings, retrieval, re-ranking, and context management
- •Strong experimentation mindset with experience designing and running A/B tests and iterating based on results
- •Strong data analysis skills to interpret results, identify patterns, and communicate findings
- •Desire to work in a fast-paced environment valuing speed of iteration, autonomy, accountability, and collaboration
- •Comfortable with ambiguity and learning new technologies as needed
- •Strong alignment with company values
- •Proficient in English at CEFR Level C2 / ILR Level 5
What You'll Do
- •Build evaluation infrastructure to measure AI system speed and accuracy both offline and online
- •Create observability tooling and dashboards that surface quality metrics week-over-week
- •Diagnose quality gaps and trace causes such as retrieval, ranking, or prompting
- •Experiment with different models and agent configurations using data to guide decisions
- •Prototype and validate improvements to the RAG pipeline including chunking strategies, retrieval methods, and re-ranking approaches
- •Analyze customer usage of AI features to identify improvements or new development areas
- •Work closely with other engineers to ensure quality improvements
- •Stay up-to-date with cutting-edge AI research, techniques, and tools
Nice to Have
- •Experience with evaluation frameworks such as Braintrust or LangSmith
Benefits
- •Fully remote work from anywhere in the world
- •Autonomy and trust to focus on outcomes
- •35 days of paid time off annually
- •Paid sabbatical after 5 years
- •Generous U.S. benchmarked compensation and startup equity regardless of location
- •Awesome medical coverage with 100% coverage for employee and family or medical reimbursement options
- •Parental leave
- •Home office stipend
- •Learning and development stipend
- •Annual bonus potential for eligible roles
- •Company retreats twice a year in various international locations
About Circle
Circle is a community platform that brings together the creators' discussions, memberships, and content.
New York, NY, US
100 - 250
Community