CloudBees · 2 days ago
Data Scientist – AI & Agentic Applications & Benchmarking
United States
Full-time
Remote
Mid Level
$166K/yr - $180K/yr
3+ years expCloudBees is a company that empowers developers to deliver scalable and secure software. They are seeking a Data Scientist to help define and measure the impact of Agentic Applications, working closely with engineers and product teams to prototype and measure AI experiences and communicate insights effectively.
Business DevelopmentCloud ComputingCloud Data ServicesDeliveryDevOpsSaaS
Responsibilities
Partner with our platform team to develop and prototype telemetry, eval frameworks, and benchmarks for emerging agentic systems
Partner with product and engineering teams to measure AI outcomes and usage across customers and teams
Help define KPIs and success metrics for AI and LLM-driven features and workflows
Use Python notebooks to explore data, visualize insights, and test hypotheses rapidly and share insights
Tell the story behind the numbers: Write internal documentation, performance summaries, and thought leadership around outcomes
Enable engineering teams to instrument, log, and evaluate agent performance effectively
Stay up to date with evolving metrics and evaluation techniques in the LLM and agentic AI ecosystem
Qualification
Required
3+ years of experience in data science or ML analytics roles, ideally in startup or high-growth environments
Proficiency in Python, including building and sharing analysis via Jupyter notebooks
Experience working with evals, telemetry, A/B testing, and evaluating user-facing ML systems
Experience with AI/ML tools such as MLFlow, Hugging Face, or other Model / LLM tools
Ability to partner with technical teams to define meaningful metrics and benchmarks
Clear communication skills—capable of writing about outcomes, sharing learnings, and influencing stakeholders
Comfort working in fast-paced, ambiguous environments where speed and clarity matter
Preferred
Experience with agentic or LLM-based applications (e.g., evaluating AI copilots, autonomous workflows)
Familiarity with tools like LangSmith, OpenInference, or custom eval stacks
Background in developer tools, DevOps, or platform engineering environments
Benefits
Health Insurance
Dental Insurance
Vision Insurance
Short & Long Term Disability
Life Insurance
HSA/FSA
Remote Work Environment
Flexible Time Off
Paid Company Holidays
Parental Leave
Variable Bonus Plan dependent on your role
Stock grant opportunities dependent on your role
401(k) with Company Match
Company
CloudBees
CloudBees enables enterprises to deliver scalable, compliant, and secure software, empowering developers to do their best work.
501-1000 employees
H1B Sponsorship
CloudBees has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (2)
2023 (1)
2022 (3)
Funding
Current Stage
Late StageTotal Funding
$356.2MKey Investors
Goldman Sachs Asset ManagementDelta-v CapitalGolub Capital
2021-12-09Series F· $150M
2021-12-09Debt Financing· $95M
2018-06-27Series E· $37M
Leadership Team
Eyal Keren
Director Of Engineering
Richa Gupta
Chief Human Resources Officer
Recent News
2026-01-03
DevOps.com
2025-07-29
Research and Markets
2025-07-25
Company data provided by crunchbase