Microsoft · 1 day ago
Senior AI Engineer
Redmond, WA
Full-time
Hybrid
Mid, Senior Level
$120K/yr - $235K/yr
4+ years expMicrosoft is a company where passionate innovators come to collaborate and envision possibilities in a cloud-enabled world. They are seeking a Senior AI Engineer to build and scale the evals platform for real-time, agentic solutions on operational data, focusing on creating pipelines and telemetry to measure AI performance.
Agentic AIApplication Performance ManagementArtificial Intelligence (AI)Business DevelopmentDevOpsInformation ServicesInformation TechnologyManagement Information SystemsNetwork SecuritySoftware
Responsibilities
Build and evolve the Real-Time Intelligence evaluations platform: implement offline and online eval pipelines, including golden datasets, human review workflows, and LLM-as-judge / auto-raters for agents, anomaly detectors, and decisioning systems
Instrument agentic solutions for observability by wiring up telemetry, tracing, structured logging, and dashboards so quality, safety, latency, and cost are easy to monitor and debug
Integrate evals into the development lifecycle by connecting pipelines to CI/CD, canary and A/B experiments, and phased rollouts, making it simple for partner teams to run and interpret evaluations
Collaborate and mentor across product, research, and engineering teams, sharing best practices on eval design, LLM-as-judge usage, and Responsible AI, and providing code reviews and guidance that raise the bar for the AI features
Qualification
Required
Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
Preferred
Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience
Demonstrated experience shipping LLM-powered features or agentic systems in production (e.g., copilots, assistants, workflow agents, tool-calling systems)
Experience with at least one major cloud platform (Azure, AWS, or GCP) and modern infrastructure practices (microservices, containers, orchestration, CI/CD)
Proven ability to define architectures, make trade-offs, and lead multi-team projects from design through launch
Background in evaluation of ML/LLM systems: designing test sets, rubrics, human and auto-rater evaluation pipelines
Excellent communication skills and the ability to explain complex trade-offs to both technical and non-technical stakeholders
Benefits
Certain roles may be eligible for benefits and other compensation.
Company
Microsoft
Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services.
10001+ employees
H1B Sponsorship
Microsoft has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (9192)
2024 (9343)
2023 (7677)
2022 (11403)
2021 (7210)
2020 (7852)
Funding
Current Stage
Public CompanyTotal Funding
$1MKey Investors
Technology Venture Investors
2022-12-09Post Ipo Equity
1986-03-13IPO
1981-09-01Series Unknown· $1M
Leadership Team
Satya Nadella
Chairman and CEO
Vukani Mngxati
Chief Executive Officer - Microsft South Africa
Recent News
2026-01-16
Morningstar.com
2026-01-16
Company data provided by crunchbase