Elastic · 2 days ago
Search - Agent Builder - Senior Data Scientist
United States
Full-time
Remote
Senior Level
$133K/yr - $211K/yr
5+ years expElastic, the Search AI Company, enables everyone to find the answers they need in real time using all their data at scale. As a Senior Data Scientist on the Search Conversational Experiences team, you will design and run evaluation pipelines, improve chat quality, and collaborate with cross-functional teams to drive product decisions.
AnalyticsCloud ComputingOpen SourceSaaSSearch EngineSoftware
Responsibilities
Design and maintain offline/online evaluation pipelines for conversational search: golden sets, rubric/LLM-as-judge calibration, groundedness/citation checks, and A/B tests
Build and compare retrieval & re-ranking baselines (sparse + dense), query understanding, and semantic rewrites; land improvements with clear metrics
Use results to drive product decisions: model selection, efficient agent routing, tool gating, and agent customization for Elastic use cases in search and beyond
Instrument dashboards and telemetry so helpfulness, faithfulness, latency, and cost trade-offs are visible and trustworthy; guard against regressions in CI
Collaborate tightly with backend engineers on contracts (ES|QL, citations, telemetry), and with PM/UX to translate findings into shipped features
Share outcomes clearly (docs, notebooks, PRs) and mentor peers in experiment design and evaluation craft
Qualification
Required
5–8 years in applied DS/ML with strong IR/NLP experience (RAG, dense/sparse retrieval, re-ranking, vector search)
Proficiency in Python, PyTorch/Transformers, Pandas; reproducible experiments (e.g., MLflow), versioned datasets, and clean, reviewable code
Hands-on evaluation expertise: offline metrics (nDCG/MRR/Recall@k), LLM-as-judge calibration, groundedness/citation scoring, and online A/B testing
Experience turning experimental results into clear product calls (models, routing, tools) and communicating them crisply to cross-functional partners
Practical Elasticsearch experience (or similar); ES|QL familiarity is a plus
Comfort working in a distributed, async-first environment; strong written communication; low-ego collaboration
Benefits
Health coverage for you and your family in many locations
Ability to craft your calendar with flexible locations and schedules for many roles
Generous number of vacation days each year
We match up to $2000 (or local currency equivalent) for financial donations and service
Up to 40 hours each year to use toward volunteer projects you love
Embracing parenthood with a minimum of 16 weeks of parental leave
Company-matched 401k with dollar-for-dollar matching up to 6% of eligible earnings
Company
Elastic
Elastic builds software to make data usable in real time and at scale for search, logging, security, and analytics use cases.
1001-5000 employees
H1B Sponsorship
Elastic has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2024 (1)
2022 (2)
2021 (1)
Funding
Current Stage
Public CompanyTotal Funding
$162MKey Investors
New Enterprise AssociatesIndex VenturesBenchmark
2018-10-04IPO
2018-05-03Secondary Market
2016-07-01Series D· $58M
Leadership Team
Ashutosh Kulkarni
CEO
Shay Banon
Chief Technology Officer
Recent News
2025-12-29
2025-12-10
Company data provided by crunchbase