Apply on Employer Site

Sully.ai · 1 day ago

Applied Research Scientist

Mountain View, California, United States

Full-time

Hybrid

Senior Level

$180K/yr - $220K/yr

Sully.ai is on a mission to revolutionize healthcare by building AI teammates that support clinicians. They are seeking an Applied Research Scientist to develop and scale automated evaluation pipelines for their AI models, focusing on improving clinical outcomes and operational efficiency.

Artificial Intelligence (AI)Health CareHospitalMachine LearningSoftware

Responsibilities

Build and scale automated evaluation pipelines (LLM-as-judge + human review) with clinical-grade benchmarks

Audit existing evaluation approaches for clinical and agentic tasks

Define initial benchmarks and build early automated pipelines

Partner with engineering to land first set of CI gates for accuracy, factuality, and safety

Deliver a repeatable evaluation framework with automated pipelines in production

Demonstrate measurable improvements in robustness, hallucination reduction, or safety

Publish or present internal research findings that directly shape product reliability

Qualification

LLM evaluation frameworksPythonMachine LearningPyTorchTensorFlowHugging FaceLangChainLlamaIndexExperiment designResearch publicationTechnical writingCommunication skills