Alight ยท 23 hours ago
AI Automation Engineer - Virtual
United States
Full-time
Remote
Mid Level
$120K/yr - $170K/yrAlight is a company that believes in the power of its people and is dedicated to creating a supportive environment. They are seeking an AI Automation Engineer to establish the initial automation framework for AI Quality, collaborating closely with the AI Quality Lead and developers to build and scale automated test suites for AI models and agents.
Health Care
No H1B
Responsibilities
Develop and maintain automated test frameworks for LLM, RAG, agentic, and classic ML systems
Implement golden/adversarial test cases (functional, safety, bias, prompt injection, refusal correctness, tool use)
Integrate evaluation tools (promptfoo, Ragas) into CI/CD pipelines and support batch quality assessments
Author and validate OpenAPI/JSON Schema contracts for model inputs/outputs; enforce schema checks in pipelines
Instrument observability metrics (latency, success rate, cost, drift) using OpenTelemetry and export to dashboards
Support data QA, lineage tracking, and drift monitoring (Evidently, OpenLineage)
Collaborate with AI developers, SDETs, and SRE to troubleshoot, optimize, and document automation processes
Contribute to red-team playbooks and security testbooks mapped to OWASP/ATLAS risks
Qualification
Required
Experience in test automation, software engineering, or quality assurance (AI/ML experience is a plus, but not required)
Proficiency in Python, JavaScript/Node.js, or similar programming languages
Familiarity with CI/CD, version control, and cloud-based deployment environments
Experience with test automation tools (promptfoo, Ragas, Postman)
Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
Basic knowledge of fairness, safety, and security testing for AI systems
Experience with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
Interest in Responsible AI, bias mitigation, and regulatory compliance
Bachelor's degree in Computer Science, Engineering, or related field, or equivalent practical experience
Benefits
A variety of health coverage options
Wellbeing and support programs
Retirement
Vacation and sick leave
Maternity, paternity & adoption leave
Continuing education and training
Several voluntary benefit options
Company
Alight
Alight Motion is a powerful video editing and animation app that empowers users to create stunning visuals right from their mobile devices.
Founded in 1978
1001-5000 employees
Funding
Current Stage
Late StageLeadership Team
Jocelyn Wyatt
CEO
Recent News
2024-10-21
Business Wire
2024-05-08
2024-05-08
Company data provided by crunchbase