69 applicants
Anthropic · 2 days ago
Technical Program Manager, Evaluations
San Francisco, CA
Full-time
Hybrid
Mid Level
$300K/yr - $320K/yr
Wonder how qualified you are to the job?
Maximize your interview chancesArtificial Intelligence (AI)Generative AI
Insider Connection @Anthropic
Responsibilities
Partner with teams like Frontier Risk Evaluations, Security, and Trust & Safety to develop and implement comprehensive evaluation protocols for our latest frontier AI models
Build a single source of truth for tracking all types of model evaluations as required by our Responsible Scaling Policy, AI safety institutes, the White House, and others
Develop and maintain procedures for conducting evaluations, including designing test suites, coordinating red team exercises, and analyzing results
Create and manage dashboards and reporting systems to track model performance, safety metrics, and evaluation outcomes across different AI systems and versions
Lead cross-functional workshops to identify potential risks and edge cases for evaluation, ensuring thorough coverage of AI capabilities and limitations
Coordinate with external partners and industry standards bodies to align our evaluation practices with emerging best practices in responsible AI development
Provide detailed status reports, identifying technical risks, dependencies, and areas requiring additional support
Facilitate communication and coordination between technical workstreams and stakeholders
Continuously identify opportunities for technical process improvements and implement changes as needed
Stay up-to-date with the latest developments in AI safety, ML engineering, and related fields to ensure the program remains at the forefront of responsible AI development
Qualification
Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.
Required
Several years of experience in technical program management, with a track record of successfully delivering complex technical programs, preferably in AI development, ML engineering, or related fields
Experience executing technical programs that require systems and engineering-level knowledge
Exceptionally strong interpersonal and communication skills that enable you to influence without authority, build cross-organizational support, cooperation and action around initiatives and process adoption
Experience prompt engineering on language models
Experience designing and/or running evaluations on Large Language Models
Knowledge of emerging AI governance frameworks and best practices
High threshold for navigating ambiguity and able to balance setting strategic priorities with rapid, high-quality execution
Benefits
Optional equity donation matching
Comprehensive health, dental, and vision insurance for you and all your dependents
401(k) plan with 4% matching
22 weeks of paid parental leave
Unlimited PTO – most staff take between 4-6 weeks each year, sometimes more!
Stipends for education, home office improvements, commuting, and wellness
Fertility benefits via Carrot
Daily lunches and snacks in our office
Relocation support for those moving to the Bay Area
Company
Anthropic
Anthropic is an AI safety and research company that focuses on increasing the safety of large-scale AI systems.
Founded in 2021
51-200 employees
H1B Sponsorship
Anthropic has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Trends of Total Sponsorships
2023 (2)
2022 (5)
2021 (1)
Funding
Current Stage
Late StageTotal Funding
$7.55BKey Investors
GoogleAmazonSK Telecom
2024-01-31Series D· Undisclosed
2023-10-27Corporate Round· $2B
2023-09-25Corporate Round· $4B
Leadership Team
Krishna Rao
Chief Financial Officer
Mike Krieger
Chief Product Officer
Recent News
2024-06-05
2024-06-04
Company data provided by crunchbase
