ChatGPT Jobs · 10 hours ago
AI Automation Engineer - Virtual
Lincolnshire, IL
Full-time
Onsite
Mid Level
$120K/yr - $170K/yrAlight Solutions is a leader in flexible workspaces and they are seeking an AI Automation Engineer to establish an automation framework for AI Quality. The role involves developing automated test frameworks and collaborating with AI developers to enhance automation processes.
Computer Software
No H1B
Responsibilities
Develop and maintain automated test frameworks for LLM, RAG, agentic, and classic ML systems
Implement golden/adversarial test cases (functional, safety, bias, prompt injection, refusal correctness, tool use)
Integrate evaluation tools (promptfoo, Ragas) into CI/CD pipelines and support batch quality assessments
Author and validate OpenAPI/JSON Schema contracts for model inputs/outputs; enforce schema checks in pipelines
Instrument observability metrics (latency, success rate, cost, drift) using OpenTelemetry and export to dashboards
Support data QA, lineage tracking, and drift monitoring (Evidently, OpenLineage)
Collaborate with AI developers, SDETs, and SRE to troubleshoot, optimize, and document automation processes
Contribute to red-team playbooks and security test books mapped to OWASP/ATLAS risks
Qualification
Required
Experience in test automation, software engineering, or quality assurance (AI/ML experience is a plus, but not required)
Proficiency in Python, JavaScript/Node.js, or similar programming languages
Familiarity with CI/CD, version control, and cloud-based deployment environments
Experience with test automation tools (promptfoo, Ragas, Postman)
Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
Basic knowledge of fairness, safety, and security testing for AI systems
Experience with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
Interest in Responsible AI, bias mitigation, and regulatory compliance
Bachelor's degree in Computer Science, Engineering, or related field, or equivalent practical experience
Benefits
Medical
Life
Retirement
PTO
Wellbeing and support programs
Vacation and sick leave
Maternity, paternity & adoption leave
Continuing education and training
Voluntary benefit options
Company
ChatGPT Jobs
We find the best job offers for experts in ChatGPT and related technologies.
Founded in 2024
2-10 employees
Funding
Current Stage
Early StageCompany data provided by crunchbase