Evolent · 7 hours ago
ML/LLM Operations Engineer
United States
Full-time
Remote
Entry, Mid Level
$120K/yr - $140K/yr
2+ years expEvolent is a company that partners with health plans and providers to improve healthcare outcomes for complex health conditions. They are seeking an ML/LLM Operations Engineer to ensure AI systems deliver reliable and compliant results in healthcare, working closely with Data Science, Engineering, and other cross-functional teams.
Health CareHospitalInformation Technology
Responsibilities
Develop and maintain standardized evaluation frameworks to consistently measure LLM performance across relevant healthcare metrics
Build monitoring systems using Logfire to track AI model performance, detect drift, and alert the team to anomalies
Create testing infrastructure for prompt versions, model selection, and quality assurance processes
Design and implement audit sampling processes for continuous quality monitoring and clinical review workflows
Oversee regulatory compliance processes, including documentation for bias assessments, model cards, and audit trails required in healthcare
Optimize LLM operations through intelligent model selection, prompt engineering, and cost management strategies
Support the transition from successful POCs to production-ready services with appropriate testing and validation
Partner with DevOps on infrastructure requirements while focusing on AI-specific monitoring and optimization
Create and maintain documentation, runbooks, and operational procedures for all deployed AI systems
Collaborate with Clinical Support Liaison to incorporate clinical feedback into system improvements
Prepare regular reports on AI system quality, performance metrics, and compliance status
Qualification
Required
Bachelor's or master's degree in computer science, data science, or related field
2+ years of experience with Python development and at least one production LLM implementation
Strong proficiency in SQL for complex log analysis and metrics generation
Demonstrated experience with LLM APIs and frameworks (experience with PydanticAI, LangChain, or similar)
Experience with monitoring tools and practices for AI systems, including performance metrics, drift detection, and alerting
Understanding of LLM behavior, prompt engineering, and common failure modes in production
Experience building evaluation or testing frameworks for AI/ML systems
Strong communication skills for cross-functional collaboration
Preferred
Experience with healthcare AI applications and compliance requirements
Familiarity with multiple LLM providers (OpenAI, Anthropic, Google, Azure)
Knowledge of Pydantic ecosystem including PydanticAI and Logfire
Understanding of LLM evaluation metrics and methodologies
Experience building tools for non-technical users
Basic knowledge of containerization (Docker) for local testing and development
Experience with cloud environments (AWS, Azure) as a user
Understanding of API rate limiting, quota management, and cost optimization strategies
Knowledge of CI/CD concepts for ML model deployments
Experience with regulatory compliance and audit processes
Excellent documentation skills and attention to detail
Benefits
Comprehensive benefits (including health insurance benefits)
Company
Evolent
Evolent partners with health plans and providers to achieve better outcomes for people with complex health conditions.
1001-5000 employees
H1B Sponsorship
Evolent has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (7)
2024 (16)
2023 (22)
2022 (15)
2021 (17)
2020 (20)
Funding
Current Stage
Public CompanyTotal Funding
$1.17BKey Investors
TPG Growth
2025-08-19Post Ipo Debt· $145M
2023-12-05Post Ipo Debt· $350M
2022-08-11Post Ipo Equity· $125M
Leadership Team
Sarah Alexander
Vice President, Strategic Accounts
Recent News
2026-01-06
2026-01-03
2025-12-22
Company data provided by crunchbase