Microsoft · 9 hours ago
Member of Technical Staff, LLM Evaluation
Mountain View, CA
Full-time
Onsite
Senior Level
$140K/yr - $275K/yr
5+ years expMicrosoft is a leading technology company that empowers individuals and organizations to achieve more. The Member of Technical Staff, LLM Evaluation will develop methodologies to evaluate Copilot's performance in real-world scenarios, focusing on user needs and improving AI systems.
Agentic AIApplication Performance ManagementArtificial Intelligence (AI)Business DevelopmentDevOpsInformation ServicesInformation TechnologyManagement Information SystemsNetwork SecuritySoftware
Responsibilities
Leverage expertise to measure the performance of Copilot, identify failure modes and novel mitigation strategies, including data mining, prompt engineering, LLM as a judge, and classifier training
Creative problem solving, navigating complexity with clarity, independently shaping direction and delivering results even when the path isn’t obvious
Create and implement comprehensive evaluation frameworks across diverse scenarios, edge cases, and potential failure modes
Build automated testing systems, generalize solutions into repeatable frameworks, and write efficient code for model pipelines and intervention systems
Maintain a user-oriented perspective by understanding needs from user perspectives, validating approaches through user research, and serving as a trusted advisor on AI matters
Track advances in research, identify relevant state-of-the-art techniques, and adapt algorithms to drive innovation in production systems serving millions of users
Qualification
Required
Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 7+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 10+ years data science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
OR equivalent experience
Preferred
Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 8+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 10+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 12+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
OR equivalent experience
Experience prompting and working with large language models
Experience writing production-quality Python code
Demonstrated interest in Responsible AI
Company
Microsoft
Microsoft is a software corporation that develops, manufactures, licenses, supports, and sells a range of software products and services.
10001+ employees
H1B Sponsorship
Microsoft has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (9192)
2024 (9343)
2023 (7677)
2022 (11403)
2021 (7210)
2020 (7852)
Funding
Current Stage
Public CompanyTotal Funding
$1MKey Investors
Technology Venture Investors
2022-12-09Post Ipo Equity
1986-03-13IPO
1981-09-01Series Unknown· $1M
Leadership Team
Satya Nadella
Chairman and CEO
Vukani Mngxati
Chief Executive Officer - Microsft South Africa
Recent News
MarketScreener
2026-01-06
2026-01-06
Company data provided by crunchbase