Salesforce · 3 hours ago
Director of Engineering - AI Evaluations & Experimentation
New York - New York
Full-time
Onsite
Director/Executive
$238K/yr - $345K/yr
10+ years expSalesforce is the #1 AI CRM, where innovation and technology drive customer success. They are seeking a Director of Engineering to lead the AI Agent Evaluation and Experimentation Platform team, responsible for overseeing the evaluation and experimentation lifecycle for AI systems and traditional ML models.
Agentic AIArtificial Intelligence (AI)Cloud ComputingCRMSaaSSales EnablementSoftware
Responsibilities
Define and execute the technical vision for evaluation and experimentation across AI agents and traditional ML models
Own offline evaluation, regression testing, scenario-based simulations, and multi-turn agent testing infrastructure
Build automated evaluation systems including LLM-as-Judge, rule-based scoring, and hybrid evaluation approaches
Design and operate online evaluation, observability, and continuous performance monitoring for agent behavior
Lead development of self-service evaluation and experimentation tooling for agent workflows, tool use, memory, and planning
Support experimentation for both real-time agents and batch or online traditional ML models
Integrate evaluation and experimentation pipelines into CI/CD workflows and release quality gates
Drive adoption of evaluation and experimentation best practices across engineering and AI teams
Set technical direction, review designs, and raise the bar on engineering quality
Lead and develop a senior engineering team, fostering innovation and excellence
Partner with AI research, product, security, and Responsible AI teams on evaluation and experimentation strategy
Qualification
Required
A related technical degree required
10+ years of engineering experience, with 5+ years leading AI/ML teams
Proven ability to lead senior engineers and engineering managers
Experience building and operating experimentation platforms for AI systems or ML products
Strong understanding of LLM-based agentic architectures and traditional ML systems
Experience designing experimentation frameworks for online and offline ML workflows
Experience building evaluation systems for models and agents, including offline tests, regression suites, online monitoring, and LLM-as-a-Judge-style approaches
Strong background in AI agents and LLM systems, including tool use, multi-step workflows, RAG, prompt and policy management, and common agent failure modes
Experience evaluating agent behavior across multi-step workflows and tool-using systems
Hands-on experience designing evaluation frameworks for AI systems
Experience with offline benchmarking, regression testing, and scenario-based evaluation
Experience with automated evaluation approaches such as LLM-as-Judge and hybrid scoring systems
Experience with online experimentation methods including A/B testing, shadow testing, and canary deployments
Experience integrating evaluation and experimentation into CI/CD pipelines and release gating
Experience with data pipelines, metrics systems, and observability tooling
Strong cross-functional communication and stakeholder alignment skills
Preferred
A master's or Ph.D. degree in computer science, machine learning, artificial intelligence, or related field
Experience with data and ML platforms (e.g., Snowflake-centric workflows, feature stores, training pipelines)
Experience working in high-scale production AI/ML environments
Benefits
Wellbeing reimbursement
Generous parental leave
Adoption assistance
Fertility benefits
Time off programs
Medical
Dental
Vision
Mental health support
Paid parental leave
Life and disability insurance
401(k)
Employee stock purchasing program
Company
Salesforce
Salesforce is a cloud-based software company that provides customer relationship management software and applications.
10001+ employees
H1B Sponsorship
Salesforce has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (1883)
2024 (2296)
2023 (1850)
2022 (2849)
2021 (2124)
2020 (1960)
Funding
Current Stage
Public CompanyTotal Funding
$65.38MKey Investors
Starboard ValueEmergence CapitalHalsey Minor
2022-10-18Post Ipo Equity
2004-06-23IPO
2003-01-01Series Unknown· $1M
Leadership Team
Arundhati Bhattacharya
Chairman & Ceo Salesforce India
Kendall Collins
CEO, GovCloud - Salesforce
Recent News
2026-01-25
Dallas Morning News
2026-01-25
2026-01-24
Company data provided by crunchbase