AI Automation Engineer - Virtual jobs in United States
cer-icon
Apply on Employer Site
company-logo

Alight Solutions · 1 day ago

AI Automation Engineer - Virtual

positionUnited States
timeFull-time
remoteRemote
seniorityMid Level
money$120K/yr - $170K/yr

Alight Solutions is dedicated to improving lives through a unified benefits ecosystem, and they are seeking an AI Automation Engineer to establish an automation framework for AI Quality. The role involves developing automated test frameworks, integrating evaluation tools, and collaborating with AI developers to enhance automation processes.

ConsultingEmployee BenefitsFinanceHuman ResourcesSoftware
badNo H1Bnote

Responsibilities

Develop and maintain automated test frameworks for LLM, RAG, agentic, and classic ML systems
Implement golden/adversarial test cases (functional, safety, bias, prompt injection, refusal correctness, tool use)
Integrate evaluation tools (promptfoo, Ragas) into CI/CD pipelines and support batch quality assessments
Author and validate OpenAPI /JSON Schema contracts for model inputs/outputs; enforce schema checks in pipelines
Instrument observability metrics (latency, success rate, cost, drift) using OpenTelemetry and export to dashboards
Support data QA, lineage tracking, and drift monitoring (Evidently, OpenLineage)
Collaborate with AI developers, SDETs, and SRE to troubleshoot, optimize, and document automation processes
Contribute to red-team playbooks and security testbooks mapped to OWASP/ATLAS risks

Qualification

Test automationPythonCI/CDJavaScript/Node.jsAI model evaluationTest automation toolsCollaborationProblem-solvingCommunicationAdaptability

Required

Experience in test automation, software engineering, or quality assurance (AI/ML experience is a plus, but not required)
Proficiency in Python, JavaScript/Node.js, or similar programming languages
Familiarity with CI/CD, version control, and cloud-based deployment environments
Experience with test automation tools (promptfoo, Ragas, Postman)
Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
Basic knowledge of fairness, safety, and security testing for AI systems
Experience with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
Interest in Responsible AI, bias mitigation, and regulatory compliance
Bachelor's degree in Computer Science, Engineering, or related field, or equivalent practical experience

Benefits

Health coverage options
Wellbeing and support programs
Retirement
Vacation and sick leave
Maternity
Paternity & adoption leave
Continuing education and training
Several voluntary benefit options

Company

Alight Solutions

twittertwittertwitter
Glassdoor3.4
company-logo
Alight Solutions is a cloud-based human capital technology company that offers HR and financial solutions.
dateFounded in 2017
location
Chicago, Illinois, USA
people-size10001+ employees
web-link
https://www.alight.com

Funding

Current Stage
Public Company
Total Funding
$1.64B
Key Investors
Caterpillar Ventures
2023-08-22Post Ipo Secondary· $22.5M
2021-07-06Post Ipo Equity· $1.55B
2021-07-06IPO

Leadership Team

leader-logo
Donna Dorsey
Chief Human Resources Officer
linkedin
leader-logo
Stephen Rush
Chief Commercial Officer
linkedin

Recent News

Morningstar.com
Soaring Healthcare Costs Lead to Higher Enrollment in Alternate Medical Plans According to Alight's Annual Enrollment Analysis
2025-12-18
Business Wire
Alight Announces Chief Financial Officer Transition
2025-12-18
PR Newswire
Retirement Clearinghouse Surpasses $20B in Account Consolidations
2025-12-16
Company data provided by crunchbase