Senior AI Engineer, GenAI & ML Evaluation Frameworks - Grafana Ops, AI/ML | USA | Remote jobs in United States
cer-icon
Apply on Employer Site
company-logo

Grafana Labs · 23 hours ago

Senior AI Engineer, GenAI & ML Evaluation Frameworks - Grafana Ops, AI/ML | USA | Remote

positionUnited States
timeFull-time
remoteRemote
senioritySenior Level
money$154K/yr - $185K/yr

Grafana Labs is a remote-first, open-source powerhouse with a mission to help users understand and improve their systems through observability tools. They are seeking a Senior AI Engineer to design and implement evaluation frameworks for Generative AI and Large Language Models, focusing on automated evaluation and metric definition.

AnalyticsCloud Data ServicesData VisualizationOpen SourceSoftware
check
Comp. & Benefits
check
H1B Sponsor Likelynote

Responsibilities

Design and implement robust evaluation frameworks for GenAI and LLM-based systems, including golden test sets, regression tracking, LLM-as-judge methods, and structured output verification
Develop tooling to enable automated, low-friction evaluation of model outputs, prompts, and agent behaviors
Define and refine metrics for both structure and semantics, ensuring alignment with realistic use cases and operational constraints
Lead the development of dataset management processes and guide teams across Grafana in best practices for GenAI evaluation

Qualification

Evaluation frameworks for AI/MLLarge Language Models (LLMs)Prompt engineeringAutomated evaluation toolingDataset management processesPragmatic mindsetCollaboration skills

Required

Experience designing and implementing evaluation frameworks for AI/ML systems
Familiarity with prompt engineering, structured output evaluation, and context-window management in LLM systems
High autonomy to collaborate and translate team goals into clear, testable criteria supported by effective tooling

Preferred

Experience working in environments with rapid iteration and experimental development
A pragmatic mindset that values reproducibility, developer experience, and thoughtful trade-offs when scaling GenAI systems
A passion for minimizing human toil and building AI systems that actively support engineers

Benefits

Equity
Bonus (if applicable)
Restricted Stock Units (RSUs)
100% Remote, Global Culture
Scaling Organization
Transparent Communication
Innovation-Driven
Open Source Roots
Empowered Teams
Career Growth Pathways
Approachable Leadership
In-Person onboarding
Balance is Key
30 days per annum annual leave

Company

Grafana Labs

twittertwittertwitter
Glassdoor4.1
company-logo
Grafana Labs is an open-source software platform built to support monitoring, visualization, and metric analytics.
dateFounded in 2014
location
New York, New York, USA
people-size1001-5000 employees
web-link
http://grafana.com

H1B Sponsorship

Grafana Labs has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (2)
2022 (1)
2020 (1)

Funding

Current Stage
Late Stage
Total Funding
$805.23M
Key Investors
Lightspeed Venture PartnersGIC
2024-08-21Series D· $270M
2024-08-21Secondary Market
2022-04-06Series D· $240M

Leadership Team

leader-logo
Raj Dutt
co-founder/ceo
linkedin
leader-logo
Anthony Woods
Co-Founder
linkedin

Recent News

lsvp.com
James Ephrati Investments
2025-11-28
Business Wire
Grafana Labs Named a Leader and Outperformer in 2025 GigaOm Radar Report for Cloud Performance Testing
2025-11-14
Business Wire
New: Grafana Labs Launches Mimir 3.0, Expanding Open Observability at Scale at KubeCon + CloudNativeCon North America 2025
2025-11-05
Company data provided by crunchbase