Anthropic · 2 hours ago
Research Engineer, Model Evaluations
San Francisco, CA
Full-time
Hybrid
Senior Level
$300K/yr - $405K/yr
5+ years expAnthropic is a public benefit corporation focused on creating reliable and safe AI systems. As a Research Engineer on the Model Evaluations team, you'll lead the design and implementation of evaluation platforms to assess and improve model capabilities and safety, collaborating closely with various teams to ensure high standards before deployment.
Artificial Intelligence (AI)Generative AIInformation TechnologyMachine Learning
Responsibilities
Design novel evaluation methodologies to assess model capabilities across diverse domains including reasoning, safety, helpfulness, and harmlessness
Lead the design and architecture of Anthropic's evaluation platform, ensuring it scales with our rapidly evolving model capabilities and research needs
Implement and maintain high-throughput evaluation pipelines that run during production training, providing real-time insights to guide training decisions
Analyze evaluation results to identify patterns, failure modes, and opportunities for model improvement, translating complex findings into actionable insights
Partner with research teams to develop domain-specific evaluations that probe for emerging capabilities and potential risks
Build infrastructure to enable rapid iteration on evaluation design, supporting both automated and human-in-the-loop assessment approaches
Establish best practices and standards for evaluation development across the organization
Mentor team members and contribute to the growth of evaluation expertise at Anthropic
Coordinate evaluation efforts during critical training runs, ensuring comprehensive coverage and timely results
Contribute to research publications and external communications about evaluation methodologies and findings
Qualification
Required
Experience designing and implementing evaluation systems for machine learning models, particularly large language models
Demonstrated technical leadership experience, either formally or through leading complex technical projects
Skilled at both systems engineering and experimental design, comfortable building infrastructure while maintaining scientific rigor
Strong programming skills in Python and experience with distributed computing frameworks
Can translate between research needs and engineering constraints, finding pragmatic solutions to complex problems
Results-oriented and thrive in fast-paced environments where priorities can shift based on research findings
Enjoy collaborative work and can effectively communicate technical concepts to diverse stakeholders
Care deeply about AI safety and the societal impacts of the systems we build
Experience with statistical analysis and can draw meaningful conclusions from large-scale experimental data
At least a Bachelor's degree in a related field or equivalent experience
Preferred
Experience with evaluation during model training, particularly in production environments
Familiarity with safety evaluation frameworks and red teaming methodologies
Background in psychometrics, experimental psychology, or other fields focused on measurement and assessment
Experience with reinforcement learning evaluation or multi-agent systems
Contributions to open-source evaluation benchmarks or frameworks
Knowledge of prompt engineering and its role in evaluation design
Experience managing evaluation infrastructure at scale (thousands of experiments)
Published research in machine learning evaluation, benchmarking, or related areas
Benefits
Equity
Benefits
Incentive compensation
Optional equity donation matching
Generous vacation and parental leave
Flexible working hours
Company
Anthropic
Anthropic is an AI research company that focuses on the safety and alignment of AI systems with human values.
501-1000 employees
H1B Sponsorship
Anthropic has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (105)
2024 (13)
2023 (3)
2022 (4)
2021 (1)
Funding
Current Stage
Late StageTotal Funding
$33.74BKey Investors
Lightspeed Venture PartnersGoogleAmazon
2025-09-02Series F· $13B
2025-05-16Debt Financing· $2.5B
2025-03-03Series E· $3.5B
Leadership Team
Dario Amodei
CEO & Co-Founder
Daniela Amodei
President and co-founder
Recent News
Deccan Chronicle
2025-12-31
globalventuring.com
2025-12-31
2025-12-31
Company data provided by crunchbase