Apple · 4 months ago
AI Evaluation Data Scientist
Cupertino, California, United States
Full-time
Onsite
Senior Level, Lead/Staff
$181K/yr - $318K/yr
10+ years expApple is a leading technology company focused on health and wellness technologies. In this role, you will develop and validate evaluation methodologies for Generative AI systems in health applications, ensuring the reliability of human and AI-based assessment systems.
AppsArtificial Intelligence (AI)BroadcastingDigital EntertainmentFoundational AIMedia and EntertainmentMobile DevicesOperating SystemsTVWearables
Responsibilities
Design and analyze human evaluations of AI systems to create reliable annotation frameworks, and ensure validity and reliability of measurements of latent constructs
Develop and refine benchmarks and evaluation protocols, using statistical modeling, test theory, and task design to capture model performance across diverse contexts and user needs
Conduct statistical analysis of evaluation data to extract meaningful insights, identify systematic issues, and inform improvements to both models and evaluation processes
Analyze model behavior, identify weaknesses, and drive design decisions with failure analysis. Examples include, but not limited to: model experimentation, adversarial testing, counterfactual analysis, creating tools to assess model behavior and user impact
Collaborate with engineers to translate evaluation methods and analysis techniques into scalable, adaptable, and reliable solutions that can be reused across different features, use cases, and evaluation workflows
Work cross-functionally to apply methods to real-world applications with designers, clinical experts, and engineering teams across Hardware and Software
Independently run and analyze experiments for real improvements
Qualification
Required
BS and a minimum of 10 years relevant industry experience in a empirical field with emphasis on quantitative methodologies of human behavior, including HCI, Psychometrics, Quantitative or Experimental Psychology, Educational Measurement, Language Assessment, or a relevant field
Proficiency in Python and ability to write clean, performant code and collaborate using standard software development practices (e.g. Git)
Strong statistical analysis skills and experience in crafting experiments, validating data quality and model performance
Experience in building and extending data and inference pipelines to process large scale datasets
Preferred
MS or PhD or equivalent experience in relevant fields
Real-world experience with LLM-based evaluation systems and human annotation and human evaluation methodologies
Experience in rigorous, evidence-based approaches to test development, e.g. quantitative and qualitative test design, reliability and validity analysis
Customer-focused mindset with experience or strong interest in building consumer digital health and wellness products
Strong communication skills and ability to work cross-functionally with technical and non-technical stakeholders
Benefits
Comprehensive medical and dental coverage
Retirement benefits
A range of discounted products and free services
Reimbursement for certain educational expenses — including tuition
Discretionary bonuses or commission payments
Relocation
Company
Apple
Apple is a technology company that designs, manufactures, and markets consumer electronics, personal computers, and software.
10001+ employees
H1B Sponsorship
Apple has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (6998)
2024 (3766)
2023 (3939)
2022 (4822)
2021 (4060)
2020 (3656)
Funding
Current Stage
Public CompanyTotal Funding
$5.67BKey Investors
Berkshire HathawayMicrosoftSequoia Capital
2025-05-05Post Ipo Debt· $4.5B
2025-01-16Post Ipo Debt· $0.31M
2021-04-30Post Ipo Equity
Leadership Team
Tim Cook
CEO
Craig Federighi
SVP, Software Engineering
Recent News
Venrock
2025-12-01
2025-09-25
Mac Daily News
2025-09-25
Company data provided by crunchbase