Apply on Employer Site

Demand.io · 6 days ago

Staff Data Scientist: AI Evaluation & Context Systems

Santa Monica, CA

Full-time

Onsite

Lead/Staff

$250K/yr - $380K/yr

Demand.io is a profitable, founder-led consumer AI commerce player driving over $1B in annual GMV. They are seeking a Staff Data Scientist to architect the Evaluation & Context Systems, focusing on creating rigorous evaluation frameworks for AI agents and ensuring they accurately interpret reality.

E-CommerceRetail TechnologyWeb3

H1B Sponsor Likely

Responsibilities

You will not be writing tickets for minor features. You will be architecting the strategic engines of the company

You will architect the Proprietary Evaluation Harness for our ecosystem

You will design the 'Golden Sets' and adversarial loops that measure our AI's ability to distinguish a marketing claim from a verified fact

You define the metrics that prevent us from shipping hallucinations

You will architect a Neuro-Symbolic Retrieval System that fuses our massive Commerce Knowledge Graph with vector search

You ensure the Agent retrieves the logic of the entity, not just the semantics of the text

You will own the Unit Economics of Intelligence

You will model the trade-offs between expensive 'Neural Reading' and efficient 'Symbolic Logic,' optimizing our stack for Risk-Adjusted Value per Token

You are a Scientific Engineer. You don't just 'run experiments'; you design rigorous Protocols

You obsess over 'The Tails.' You are not satisfied with 90% accuracy. You dig into the distribution tails to understand exactly why the 10% failed

You are an Anti-Academic. You do not want to publish papers. You want to ship systems that survive contact with 10 million users

You treat Evaluation as Code. You automate the generation of test cases

Qualification

Evaluation ArchitectureNeuro-Symbolic RetrievalUnit Economics of IntelligenceStatistical significanceAutomated testing

Required

You are a Scientific Engineer. You don't just 'run experiments'; you design rigorous Protocols

You understand statistical significance, p-hacking, and the danger of averages

You obsess over 'The Tails.' You are not satisfied with 90% accuracy

You dig into the distribution tails to understand exactly why the 10% failed

You are an Anti-Academic. You do not want to publish papers

You want to ship systems that survive contact with 10 million users

You treat Evaluation as Code. You automate the generation of test cases

You don't write 1,000 unit tests; you write the Generator that creates them

Benefits

100% premium coverage for you and your family

Daily catered lunches

Unlimited PTO that we actually expect you to use to recharge

Company

Demand.io

Demand.io is a network of shopping platforms & experiences that create greater alignment between brands, consumers and creators.

Founded in 2009

Los Angeles, California, USA

51-200 employees

https://demand.io

H1B Sponsorship

Demand.io has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)

Distribution of Different Job Fields Receiving Sponsorship

Represents job field similar to this job

Trends of Total Sponsorships

2023 (1)