Amazon Web Services (AWS) · 7 hours ago
Sr Applied Scientist, AI Agent Evaluation, AWS Applied AI Solution – Core Services
Seattle, WA
Full-time
Onsite
Senior Level
$150K/yr - $260K/yr
6+ years expAmazon Web Services (AWS) is a leading cloud platform that aims to provide innovative business applications used globally. They are seeking a Senior Applied Scientist to lead AI agent evaluation efforts, focusing on developing foundational AI components and frameworks that enhance the delivery of AI products across various business applications.
Agentic AIConsultingDevOpsInformation TechnologySoftwareWeb Development
Responsibilities
Design and implement evaluation frameworks for AI agents, including benchmarking tools, annotation systems for RLHF, and standardized patterns for memory orchestration and retrieval that ensure consistent performance across diverse use cases
Develop deep expertise in a strategic research area within AI agent systems, becoming the organization's scientific expert in areas such as agent design and evaluation
Identify and devise new research solutions for ill-defined customer or business problems that require novel methodologies and paradigms to be invented at the product level
Articulate key potential scientific challenges of ongoing or future customer needs or business problems in AI agents, and present interventions to address them
Lead the design, implementation, and successful delivery of solutions for scientifically-complex problems, writing "critical-path" code
Write clear, useful narratives and documentation describing inventions, solutions, and design choices that enable others to understand and reproduce your work
Demonstrate detailed knowledge of your team's solutions and systems while proactively driving utilization and improvement upon the state of the art
Independently assess alternative AI technologies and choose the right approaches to be integrated into your systems
Apply and drive your team to adopt best practices in scientific research, code development, and technical documentation
Influence across multiple teams to build consensus on scientific approaches and architectural decisions
Actively mentor and develop other scientists and engineers, elevating the technical capabilities of the organization
Qualification
Required
3+ years of building machine learning models for business application experience
PhD, or Master's degree and 6+ years of applied research experience
Experience programming in Java, C++, Python or related language
Experience with neural deep learning methods and machine learning
Experience using managed ML/AI solutions
Deep expertise in at least one AI/ML discipline such as NLP, reinforcement learning, etc
Experience with neural deep learning methods and popular frameworks such as PyTorch, TensorFlow, or MxNet
Experience designing and implementing AI/ML systems, including working with LLMs, prompt engineering, retrieval augmented generation (RAG), fine-tuning, or AI agent development
Experience using managed ML/AI solutions such as AWS SageMaker AI or Amazon Bedrock
Demonstrated track record of scientific innovation with measurable business impact
Experience building production systems that operate at scale
Proven ability to drive scientific roadmap and secure management buy-in for new initiatives
Strong collaboration skills with ability to influence across organizational boundaries
Preferred
Experience with modeling tools such as R, scikit-learn, Spark MLLib, MxNet, Tensorflow, numpy, scipy etc
Experience with large scale distributed systems such as Hadoop, Spark etc
7+ years of experience applying scientific methods to solve complex AI problems at scale
Experience mentoring junior scientists and engineers
Advanced knowledge of large language models, including fine-tuning, evaluation, and responsible AI practices
Experience developing bias detection, fairness metrics, and ethical evaluation frameworks for AI systems
Proven track record building reinforcement learning systems with human feedback, including annotation frameworks and reward modeling
Experience creating evaluation frameworks measuring factuality, robustness, and safety across diverse scenarios, comparable to HELM or HealthBench
Expertise in building reusable AI components with well-defined interfaces
Benefits
Equity
Sign-on payments
Full range of medical, financial, and/or other benefits
Company
Amazon Web Services (AWS)
Launched in 2006, Amazon Web Services (AWS) began exposing key infrastructure services to businesses in the form of web services -- now widely known as cloud computing.
10001+ employees
H1B Sponsorship
Amazon Web Services (AWS) has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (22803)
2024 (21175)
2023 (19057)
2022 (24088)
2021 (12233)
2020 (14881)
Funding
Current Stage
Late StageTotal Funding
unknownKey Investors
BIRD Foundation
2025-01-22Grant
Leadership Team
Matt Garman
Chief Executive Officer
Anand Desikan
CTO, CXO Advisor, and Enterprise Technologist
Recent News
2026-01-14
Irish Tech News
2026-01-14
Company data provided by crunchbase