ChatGPT Jobs · 11 hours ago
AI Quality Assurance Analyst
Seattle, WA
Full-time
Onsite
Senior Level
7+ years expAllied Benefit Systems is seeking an AI Quality Assurance Analyst to be the quality champion for their emerging AI capabilities. The role involves ensuring that AI models, integrations, and automations work reliably and safely, by designing tests and collaborating with developers and product owners.
Computer Software
Responsibilities
Test AI features integrated into core systems (e.g., Dynamics CRM, QicLink claims platform, Docuvantage) to ensure they function correctly in end-to-end workflows
Write and automate test cases for AI outputs and traditional software features; define clear pass/fail criteria (using statistical sampling when appropriate) to validate AI results
Own regression testing for AI capabilities: continuously re-run critical test scenarios as models are updated or data changes, to catch issues early
Contribute to the overall test strategy and perform adversarial or edge-case testing (e.g., stress tests, prompt injections) to evaluate the robustness and security of AI components
Document and recommend solutions for issues found; coordinate with the AI Product Owner and SMEs to clarify requirements for fixes and improvements
Collaborate closely with developers and the Product Owner to report bugs, verify fixes, and improve requirements. Emphasize shared accountability for quality across the team
Qualification
Required
Bachelor's degree in related field or equivalent work experience required
7+ years in software Quality Assurance/testing roles, with significant experience in enterprise systems or complex integrations required
Proven expertise in test automation (UI and API) and building regression test suites in CI/CD environments
Familiarity with AI/ML system testing principles. Comfortable validating outputs that may vary statistically and determining acceptable accuracy thresholds
Strong analytical skills to identify edge cases and pinpoint defects across interconnected systems (frontend, backend, data, model)
Excellent documentation and communication skills for writing test plans, reporting issues, and guiding teams on quality best practices
Test automation tools such as Selenium or Cypress for web interface testing, and Postman or REST-assured for API testing
Testing frameworks like PyTest or JUnit, integrated with CI/CD pipelines (Azure DevOps, Jenkins) for continuous testing
Version control (Git) for maintaining test scripts, and monitoring/logging tools (Azure Monitor, Splunk) to investigate issues
Performance testing or security testing tools for evaluating AI systems under load or threat scenarios
Preferred
Experience in industries like insurance or healthcare is a plus (provides domain context for Allied's processes)
Benefits
Medical
Dental
Vision
Life
PTO
Tuition Reimbursement
EAP
Technology Stipend
Company
ChatGPT Jobs
We find the best job offers for experts in ChatGPT and related technologies.
2-10 employees
Funding
Current Stage
Early StageCompany data provided by crunchbase