Senior AI Test Automation Engineer

Senior AI Test Automation Engineer 

Summary 

The Senior AI Test Automation Engineer designs, develops, maintains, and executes automated testing and evaluation solutions for both traditional software and LLM-powered applications. Operating in a forward-deployed capacity, this role works directly with users and delivery teams to capture real-world usage patterns and feedback, translating them into a continuously growing evaluation framework that validates LLM behavior against the consistent flows users actually follow. This role partners with delivery teams, QA, development, product, AI engineering, and the QA Center of Excellence (CoE) to establish scalable automation and evaluation practices, increase test and eval coverage, and integrate quality controls throughout the software delivery lifecycle , from pre-deployment regression testing through production observability. 

You must be eligible to work in the US without Visa Sponsorship.

Responsibilities 

LLM Evaluation & Observability (Core) 

• Act as a forward-deployed quality engineer: engage directly with users and stakeholders to collect feedback, observe real usage patterns, and identify the consistent flows users follow through LLM-powered features. 

• Translate user feedback and production traces into curated evaluation datasets in LangSmith, and continuously expand the eval framework as new feedback, edge cases, and failure modes are discovered. 

• Design, build, and maintain offline evaluation suites (regression, benchmarking, and backtesting) that gate prompt, model, and LangGraph workflow changes before deployment. 

• Develop and calibrate evaluators , heuristic/code-based checks, LLM-as-judge evaluators, and pairwise comparisons , and validate judge reliability against human review. 

• Instrument and maintain end-to-end tracing across LangGraph agents and workflows using LangSmith, ensuring trace coverage, quality, and useful metadata for debugging and analysis. 

• Manage annotation queues and human-in-the-loop feedback workflows, routing interesting or problematic production runs to reviewers and feeding results back into datasets and evaluator calibration. 

• Analyze agent trajectories and multi-step LangGraph executions (tool calls, state transitions, retrieval steps) to pinpoint failure points and distinguish nondeterministic LLM variance from genuine product defects. 

• Integrate eval runs into CI/CD pipelines so that dataset versions, experiments, and quality thresholds provide automated feedback on every relevant change. 

• Support the adoption of production auditing and monitoring capabilities — such as online evaluations on live traffic, quality drift detection, and alerting — to help teams detect issues in production (supportive to the role, not its core focus). 

Test Automation (Core) 

• Design, develop, and implement automated test scripts for UI, API, integration, and regression testing, including deterministic E2E coverage of LLM-powered application surfaces. 

• Integrate automated tests into CI/CD pipelines to enable timely feedback and continuous quality validation. 

• Collaborate with QA, development, product, and business teams to translate requirements, acceptance criteria, and expected agent behaviors into effective automated test and eval coverage. 

• Analyze and triage automation test failures, differentiating framework or script issues from valid product defects , including the added dimension of expected LLM nondeterminism. 

• Support test data management (including eval dataset versioning, splits, and provenance) and help identify or resolve test-environment stability issues. 

• Report on test execution results, eval experiment outcomes, automation and eval coverage, quality trends, and risks. 

• Participate in code reviews and contribute to automation and evaluation standards, reusable components, and best practices. 

• Engage with the QA CoE to align automation and AI evaluation practices with enterprise standards while contributing domain-specific feedback, lessons learned, and continuous-improvement opportunities. 

Required Experience 

• 5+ years of experience in test automation engineering, software quality assurance, or a related role. 

• 1–2+ years of hands-on experience testing or evaluating LLM-powered applications, including building eval datasets, defining pass/fail criteria for nondeterministic outputs, and using LLM-as-judge or heuristic evaluators. 

• Hands-on experience with LangSmith for tracing, evaluation, and observability of LLM applications , including creating datasets, running experiments, and configuring evaluators. 

• Hands-on experience with LangGraph, including graph-based agent workflows, state and context management, and tool calling. 

• Hands-on experience building and maintaining automated test suites using Playwright, preferably with TypeScript. 

• Working proficiency in Python and/or TypeScript sufficient to author custom evaluators, tracing instrumentation, and test code. 

• Experience automating UI and API testing, and experience with API testing tools or frameworks. 

• Familiarity with CI/CD tools such as Azure DevOps, Jenkins, GitHub Actions, or equivalent, including integrating eval runs as pipeline gates. 

• Working knowledge of source-code version control, including Git. 

• Understanding of Agile/Scrum delivery practices and participation in Agile ceremonies. 

• Ability to analyze requirements and user feedback, identify test and eval scenarios, and create maintainable automated coverage. 

• Strong problem-solving, troubleshooting, communication, and collaboration skills , including comfort engaging directly with end users to gather feedback in a forward-deployed capacity. 

Preferred / Nice to Have 

• Ability to understand and document end-to-end business processes, user journeys, and operational workflows. 

• Partner with business stakeholders and end users to translate process requirements into test scenarios, acceptance criteria, and LLM evaluation datasets. 

• Identify process exceptions, edge cases, dependencies, and risks that may affect application or agent behavior. 

• Validate that automated workflows and LLM-powered features produce outcomes aligned with defined business rules and user needs. 

• Use production feedback and observed user behavior to continuously refine process coverage, test automation, and evaluation frameworks. 

• Experience with online evaluations, production monitoring, and quality drift detection for LLM applications. 

• Experience with agent trajectory evaluation, RAG evaluation (retrieval relevance, groundedness, hallucination detection), or guardrails validation. 

• Familiarity with prompt engineering and prompt versioning workflows, and evaluating the impact of prompt or model changes. 

• Understanding of statistical approaches to nondeterministic testing (multiple-run sampling, confidence thresholds, summary metrics across datasets). 

• Experience authoring BDD/Gherkin scenarios using Cucumber or a similar framework. 

• Experience with contract testing tools. 

• Familiarity with SAFe and Agile Release Train (ART) practices. 

• Experience with AI-assisted engineering practices (e.g., using AI coding agents to accelerate test and eval development). 

• Experience with performance, accessibility, mobile, or security test automation. 

• Experience with test management and defect-tracking tools, such as Azure DevOps.