Cresta logo
    C

    AI Agent Evaluation Engineer

    Cresta
    Canada (Remote); United States (Remote)•Canada - AB (Remote)•Canada - BC (Remote)•Canada - ON (Remote)•US (Remote)
    Remote
    Mid Level
    Full Time
    AILLMevaluationsoftware engineeringremote

    Requirements

    • •3+ years professional experience in software engineering, applied ML, or data-intensive research roles
    • •1-2 years hands-on experience with LLM-based systems in production
    • •Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or related field, or equivalent experience
    • •Experience building and operating end-to-end evaluation systems for production LLM products
    • •Deep understanding of evaluation methodology including golden sets, judges, regression suites, and calibration
    • •Fluent understanding of LLM agents including prompts, tool calls, context handling, multi-turn state
    • •Strong programming skills in Python and experience with LLM APIs
    • •Ability to write consistent rubrics and measure inter-rater reliability
    • •Ability to design tests from open-ended quality questions
    • •Willingness to perform non-scalable work such as labeling conversations and building failure taxonomies
    • •Professional fluency in at least one language other than English
    • •Strong written communication skills

    What You'll Do

    • •Define and maintain the evaluation rubric for the AI agent across adherence, outcome, and conversation experience axes
    • •Write annotation guidelines and failure taxonomy; run calibration sessions and measure inter-rater agreement
    • •Classify and prioritize failure modes by severity and frequency
    • •Review production conversations weekly, open-code failures, cluster them, and create evaluations for automation
    • •Trace failures to causes and route to appropriate owners with evidence
    • •Evaluate quality across languages and report by segment
    • •Build and operate evaluation pipeline combining automated checks, LLM-based judges, and human review
    • •Calibrate automated judges against human labels
    • •Design sampling strategies for quality measurement
    • •Build and maintain golden set and regression suite for validation
    • •Maintain failure-mode coverage map and close gaps
    • •Design evaluations for high-stakes conversation types and edge cases
    • •Define quality and latency thresholds for the agent
    • •Report quality trends to stakeholders and customers clearly
    • •Collaborate with engineers, product managers, and language teams to prioritize recommendations
    • •Document methodology for reproducibility and continuity

    Benefits

    • •Comprehensive medical, dental, and vision coverage
    • •Flexible PTO
    • •Paid parental leave
    • •Retirement savings plan
    • •Remote work setup budget
    • •Monthly wellness and communication stipend
    • •In-office meal program and commuter benefits for onsite employees

    About Cresta

    Cresta uses artificial intelligence to assist sales and service agents in the quality of customer service.

    Palo Alto, CA, US
    250 - 500
    AI & Machine Learning