Elastic · 1 month ago
Search - Agent Builder - Senior Data Scientist
United States
Full-time
Remote
Senior Level
$160K/yr - $253K/yr
5+ years expElastic, the Search AI Company, enables businesses to leverage all their data through advanced search and AI solutions. The Senior Data Scientist will be part of a cross-functional team focused on enhancing conversational search quality by designing evaluation pipelines and driving product decisions based on data insights.
AnalyticsCloud ComputingOpen SourceSaaSSearch EngineSoftware
Responsibilities
Design and maintain offline/online evaluation pipelines for conversational search: golden sets, rubric/LLM-as-judge calibration, groundedness/citation checks, and A/B tests
Build and compare retrieval & re-ranking baselines (sparse + dense), query understanding, and semantic rewrites; land improvements with clear metrics
Use results to drive product decisions: model selection, efficient agent routing, tool gating, and agent customization for Elastic use cases in search and beyond
Instrument dashboards and telemetry so helpfulness, faithfulness, latency, and cost trade-offs are visible and trustworthy; guard against regressions in CI
Collaborate tightly with backend engineers on contracts (ES|QL, citations, telemetry), and with PM/UX to translate findings into shipped features
Share outcomes clearly (docs, notebooks, PRs) and mentor peers in experiment design and evaluation craft
Qualification
Required
5–8 years in applied DS/ML with strong IR/NLP experience (RAG, dense/sparse retrieval, re-ranking, vector search)
Proficiency in Python, PyTorch/Transformers, Pandas; reproducible experiments (e.g., MLflow), versioned datasets, and clean, reviewable code
Hands-on evaluation expertise: offline metrics (nDCG/MRR/Recall@k), LLM-as-judge calibration, groundedness/citation scoring, and online A/B testing
Experience turning experimental results into clear product calls (models, routing, tools) and communicating them crisply to cross-functional partners
Practical Elasticsearch experience (or similar); ES|QL familiarity is a plus
Comfort working in a distributed, async-first environment; strong written communication; low-ego collaboration
Benefits
Health coverage for you and your family in many locations
Ability to craft your calendar with flexible locations and schedules for many roles
Generous number of vacation days each year
Increase your impact - We match up to $2000 (or local currency equivalent) for financial donations and service
Up to 40 hours each year to use toward volunteer projects you love
Embracing parenthood with minimum of 16 weeks of parental leave
Company
Elastic
Elastic builds software to make data usable in real time and at scale for search, logging, security, and analytics use cases.
1001-5000 employees
H1B Sponsorship
Elastic has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2024 (1)
2022 (2)
2021 (1)
Funding
Current Stage
Public CompanyTotal Funding
$162MKey Investors
New Enterprise AssociatesIndex VenturesBenchmark
2018-10-04IPO
2018-05-03Secondary Market
2016-07-01Series D· $58M
Leadership Team
Ashutosh Kulkarni
CEO
Shay Banon
Chief Technology Officer
Recent News
2025-12-29
2025-12-10
Company data provided by crunchbase