Canada (Remote); United States (Remote)•Canada - AB (Remote)•Canada - BC (Remote)•Canada - ON (Remote)•US (Remote)
Remote
Mid Level
Full Time
AILLMevaluationsoftware engineeringremote
Requirements
- •3+ years professional experience in software engineering, applied ML, or data-intensive research roles
- •1-2 years hands-on experience with LLM-based systems in production
- •Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or related field, or equivalent experience
- •Experience building and operating end-to-end evaluation systems for production LLM products
- •Deep understanding of evaluation methodology including golden sets, judges, regression suites, and calibration
- •Fluent understanding of LLM agents including prompts, tool calls, context handling, multi-turn state
- •Strong programming skills in Python and experience with LLM APIs
- •Ability to write consistent rubrics and measure inter-rater reliability
- •Ability to design tests from open-ended quality questions
- •Willingness to perform non-scalable work such as labeling conversations and building failure taxonomies
- •Professional fluency in at least one language other than English
- •Strong written communication skills
What You'll Do
- •Define and maintain the evaluation rubric for the AI agent across adherence, outcome, and conversation experience axes
- •Write annotation guidelines and failure taxonomy; run calibration sessions and measure inter-rater agreement
- •Classify and prioritize failure modes by severity and frequency
- •Review production conversations weekly, open-code failures, cluster them, and create evaluations for automation
- •Trace failures to causes and route to appropriate owners with evidence
- •Evaluate quality across languages and report by segment
- •Build and operate evaluation pipeline combining automated checks, LLM-based judges, and human review
- •Calibrate automated judges against human labels
- •Design sampling strategies for quality measurement
- •Build and maintain golden set and regression suite for validation
- •Maintain failure-mode coverage map and close gaps
- •Design evaluations for high-stakes conversation types and edge cases
- •Define quality and latency thresholds for the agent
- •Report quality trends to stakeholders and customers clearly
- •Collaborate with engineers, product managers, and language teams to prioritize recommendations
- •Document methodology for reproducibility and continuity
Benefits
- •Comprehensive medical, dental, and vision coverage
- •Flexible PTO
- •Paid parental leave
- •Retirement savings plan
- •Remote work setup budget
- •Monthly wellness and communication stipend
- •In-office meal program and commuter benefits for onsite employees
