Apply on Employer Site

Exa · 3 months ago

Research, Evals

San Francisco, California

Full-time

Onsite

Mid Level

$150K/yr - $300K/yr

Exa is building a cutting-edge search engine for the AI age, focusing on massive scale indexing and advanced embedding models. They are seeking an ML evals engineer to design and construct their evaluation stack, ensuring effective measures of search quality in a complex environment.

InternetSearch EngineSoftware

H1B Sponsored

Responsibilities

Investigate how to evaluate search engines in an LLM world and then build the most comprehensive, creative, and effective eval suite

Decide the future of search through the evals we choose to optimize for

Write a manifesto of what perfect search means

Design and implement evaluation frameworks that probe the limits of search

Build scalable, reliable eval pipelines that track regressions, drift, and quality signals across billions of documents

Create golden datasets, synthetic benchmarks, agentic tasks, and real-world test suites that reflect how developers, agents, and humans actually use Exa

Partner closely with ML researchers, data engineers, infra engineers, and product to shape the feedback loops that improve our search models

Qualification

ML experiencePythonRustDistributed pipelinesGPU/cluster jobsData inspectionCreative measurement strategies

Required

Have hands-on ML experience (training, finetuning, or evaluating models (bonus if related to embeddings or LLMs)

Have strong engineering fundamentals and can build reliable systems (Python, Rust, distributed pipelines, GPU/cluster jobs, etc.)

Enjoy diving into data via building eval sets, inspecting edge cases, designing creative measurement strategies

Company

Exa

Exa was built with a simple goal — to organize all knowledge.

Founded in 2021

New York, New York, USA

11-50 employees

https://metaphor.so

H1B Sponsorship

Exa has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)

Distribution of Different Job Fields Receiving Sponsorship

Represents job field similar to this job

Trends of Total Sponsorships

2025 (2)

2024 (1)

Funding

Current Stage

Early Stage

Total Funding

$2.12M

2023-04-01Seed

2021-10-15Seed· $2M

2021-08-27Pre Seed· $0.12M

Leadership Team

Will Bryk

Co-Founder and CEO

Company data provided by crunchbase