Nimbus · 23 hours ago
QA Manager, AI Agent Testing & Quality Assurance
Seattle, WA
Full-time
Onsite
Mid, Senior Level
$70K/yr - $110K/yr
5+ years expNimbus AI is a company that builds conversational and workflow agents for businesses. They are seeking a QA Manager to establish testing frameworks and ensure the reliability of their AI systems across customer deployments.
Computer Software
Responsibilities
QA strategy & framework development for conversational agents, workflow automations, and partner-specific models across multiple verticals
Test planning and execution—designing test cases, evaluation rubrics, regression suites, and automated testing pipelines for agent behavior
Quality metrics and monitoring to track agent accuracy, consistency, guardrail effectiveness, and performance degradation over time
Cross-functional collaboration with prompt engineers, product, and engineering teams to identify, document, and resolve quality issues
Agent validation processes to ensure new releases, prompt changes, and training updates maintain reliability standards
Team building and leadership as we scale—hiring, mentoring, and growing the QA function
Qualification
Required
5+ years of QA experience, with at least 2 years in a management or lead role
Experience testing AI/ML products, LLM applications, or conversational systems—you understand non-deterministic behavior and how to test it
Strong analytical skills—comfortable evaluating agent outputs, identifying patterns in failures, and defining measurable quality standards
Ability to build testing frameworks from scratch, including test case libraries, evaluation criteria, and automation strategies
Experience with technical testing tools (APIs, JSON, test automation frameworks, monitoring systems)
Excellent communication skills—you can clearly document bugs, write test plans, and explain quality issues to technical and non-technical stakeholders
Leadership experience building or managing QA teams, processes, and culture
Company
Nimbus
Designing conversational flows is tedious, error-prone, and rarely scalable.
11-50 employees
H1B Sponsorship
Nimbus has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2022 (2)
Funding
Current Stage
Early StageCompany data provided by crunchbase