QA Engineer, Automation and Evals

The QA Engineer, Automation and Evals will design, implement, and maintain automated test suites across APIs, UIs, and workflows to ensure product reliability and faster release cycles. This role involves building evaluation pipelines for AI and RAG systems, integrating tests into CI/CD pipelines, and driving higher test coverage in multi-tenant environments. The ideal candidate will combine strong technical testing skills with the ability to communicate risks and quality tradeoffs effectively.

Responsibilities

• Design and maintain automated test suites across services and UI.
• Build evaluation pipelines for ML and RAG systems.
• Integrate quality gates into CI pipelines and monitor flake rate.
• Create and manage test data strategies for multi-tenant environments.
• Drive high automation coverage on APIs and UI.
• Reduce regression rates and improve release confidence.

Requirements

• 3+ years of QA automation experience in SaaS environments.
• Strong skills with testing frameworks such as Playwright, Cypress, or PyTest.
• Familiarity with CI/CD test integration.
• Proven ability to design test strategies for APIs, UIs, and workflows.
• Experience building reliable test harnesses and CI integrations.
• Strong communication skills for articulating risk and quality tradeoffs.
• Ability to analyze logs and traces to triage failures quickly.

Nice to Haves

• Knowledge of evals for AI and LLM applications.
• Experience in security or load testing.
• Background in regression testing for RAG pipelines.

Test data management in multi-tenant environments.