OpenTrain AI ยท 10 hours ago
AI Agent Evaluation Analyst for Autonomous Agents (No coding required)
United States
Contract
Remote
New Grad, Entry LevelOpenTrain AI is hiring detail-oriented, analytical contributors to help test and improve autonomous AI agent evaluations. The role involves reviewing evaluation tasks, identifying inconsistencies, and collaborating with teams to ensure thorough testing of agents.
Artificial Intelligence (AI)FreelanceMarketplaceOnline Portals
Hiring Manager
Akita Sanders
Responsibilities
Review and refine agent evaluation tasks and scenarios for logic, completeness, and realism
Identify inconsistencies, ambiguities, and missing assumptions
Define gold-standard expected behaviors for agents
Annotate reasoning paths, cause-effect relationships, and plausible alternatives
Collaborate with QA, writers, and developers to suggest refinements and expand edge case coverage
Ensure autonomous agents are tested thoroughly and realistically
Qualification
Required
Strong analytical thinking and excellent attention to detail
Fluent written English with clear documentation skills
Comfort reading structured formats such as JSON or YAML (no need to write code)
Ability to reason about complex systems and spot what could break or be misinterpreted
Preferred
Prior exposure to QA/test-case thinking, logic puzzles, or evaluation frameworks
Company
OpenTrain AI
OpenTrain AI connects companies with vetted data labeling experts, supports any annotation tool, and manages escrow payments.
Founded in 2022
2-10 employees
Funding
Current Stage
Early StageLeadership Team
W
Weston Dotson
Founder
Company data provided by crunchbase