Devoted Health · 2 weeks ago
Senior Manager, AI Performance & Operations
United States
Full-time
Remote
Mid, Senior Level
$110K/yr - $172K/yrDevoted Health is on a mission to improve the health and well-being of older Americans by providing integrated healthcare solutions. The Senior Manager, AI Performance & Operations will oversee the performance and continuous improvement of AI workflows, ensuring they meet operational outcomes and quality standards.
Family & CareHealthcareProperty & Casualty InsuranceWellnessElder CareElderlyHealth CareInsurance
Responsibilities
Post-Launch Accountability: Own the health of major voice and text AI workflows, ensuring business, quality, safety, and operational outcomes are met
Portfolio Oversight: Maintain a current view of all major workflows, including status, maturity, failure modes, and next actions
Performance & Dashboards: Monitor weekly indicators (containment, escalation, complaints) and oversee production dashboards to track behavior independently
Diagnosis & Incident Response: Lead response for AI regressions; distinguish between model, prompt, or orchestration issues and drive post-mortems to closure
Continuous Improvement & Optimization: Run structured cycles (hypothesis, testing, rollout/rollback) for "mini-launches" like model swaps, prompt changes, and tool updates. Recommend evidence-based accelerations, such as model migrations or deeper tool-orchestration changes
Scalable Evaluation: Design eval systems using golden datasets, LLM-as-judge workflows, and human review queues for production and pre-production
Expert Review: Guide new workflows during incubation to ensure they launch with proper instrumentation, eval coverage, and safety models
Executive Communication: Translate technical changes into plain language for leadership, explaining what broke, why it matters, and who is fixing it
Qualification
Required
Strong production experience with LLMs, RAG, voice AI, agentic orchestration, or equivalent ML systems
Demonstrated experience designing or operating rubrics, scoring frameworks, regression tests, and production-sampling workflows
A clear mental model for distinguishing failure modes, such as model reasoning, tool failure, retrieval issues, and workflow logic
Comfort with active experimentation—including prompt/context tuning, retrieval redesign, and safe model migrations—beyond simple issue triage
Excellent cross-functional judgment and the ability to communicate clearly in ambiguous, high-accountability environments
Deep personal accountability with a drive to own results rather than just producing recommendations
Preferred
Experience in healthcare, regulated workflows, contact center operations, or high-stakes service quality/reliability functions
Benefits
Employer sponsored health, dental and vision plan with low or no premium
Generous paid time off
$100 monthly mobile or internet stipend
Stock options for all employees
Bonus eligibility for all roles excluding Director and above; Commission eligibility for Sales roles
Parental leave program
401K program
Our total rewards package is for full time employees only. Intern and Contract positions are not eligible.
Company
Devoted Health
Devoted Health is a healthcare company serving seniors and giving them a health care plan with personal guides and world-class technology.
Founded in 2017
1001-5000 employees
H1B Sponsorship
Devoted Health has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Engineering and Development
Trends of Total Sponsorships
*2021 (1)
Funding
Current Stage
Late StageTotal Funding
$2.62BKey Investors
The Space BetweenFearless Ventures,GIC,Highbury Group,Maverick Ventures,Stardust Equity,The Space BetweenSoftBank Vision Fund,Uprising Ventures
2026-01-30Series F· $366M
2024-08-02Series E· $112M
2023-12-29Series E· $175M
Leadership Team
Todd Park
Co-Founder, Executive Chairman
Adam Baratz
Technologist
Recent News
maverickcapital.com
2026-06-25
2026-06-01
2026-04-30
Company data provided by crunchbase