Strada · 3 days ago
AI Automation Engineer - Virtual
United States
Full-time
Remote
Mid Level
$120K/yr - $170K/yrAlight is a company that believes a company’s success starts with its people. As the AI Automation Engineer in the AI CoE, you will be instrumental in standing up the initial automation framework for AI Quality, working closely with the AI Quality Lead and developers to build, run, and scale automated test suites for AI models and agents.
AdviceBookkeeping and PayrollBusiness DevelopmentConsulting
No H1B
Responsibilities
Develop and maintain automated test frameworks for LLM, RAG, agentic, and classic ML systems
Implement golden/adversarial test cases (functional, safety, bias, prompt injection, refusal correctness, tool use)
Integrate evaluation tools (promptfoo, Ragas) into CI/CD pipelines and support batch quality assessments
Author and validate OpenAPI/JSON Schema contracts for model inputs/outputs; enforce schema checks in pipelines
Instrument observability metrics (latency, success rate, cost, drift) using OpenTelemetry and export to dashboards
Support data QA, lineage tracking, and drift monitoring (Evidently, OpenLineage)
Collaborate with AI developers, SDETs, and SRE to troubleshoot, optimize, and document automation processes
Contribute to red-team playbooks and security testbooks mapped to OWASP/ATLAS risks
Qualification
Required
Experience in test automation, software engineering, or quality assurance (AI/ML experience is a plus, but not required)
Proficiency in Python, JavaScript/Node.js, or similar programming languages
Familiarity with CI/CD, version control, and cloud-based deployment environments
Experience with test automation tools (promptfoo, Ragas, Postman)
Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
Basic knowledge of fairness, safety, and security testing for AI systems
Experience with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
Interest in Responsible AI, bias mitigation, and regulatory compliance
Bachelor's degree in Computer Science, Engineering, or related field, or equivalent practical experience
Benefits
A variety of health coverage options
Wellbeing and support programs
Retirement
Vacation and sick leave
Maternity, paternity & adoption leave
Continuing education and training
Several voluntary benefit options
Company
Strada
Strada is a payroll and tech company that offers workforce management with cloud solutions and services for multinational organizations.
Founded in 2018
5001-10000 employees
Funding
Current Stage
Late StageLeadership Team
Colin Brennan
Chief Executive Officer
Gautam Mekala
Chief Information Officer
Company data provided by crunchbase