Anyscale · 5 hours ago
Software Engineer - Model Serving Infrastructure
San Francisco, CA
Full-time
Onsite
Mid Level
$170K/yr - $237K/yrAnyscale is on a mission to democratize distributed computing and make it accessible to software developers. They are seeking talented engineers to contribute to the development of next-generation, high-performance machine learning serving systems, focusing on building infrastructure that powers AI applications for millions of users worldwide.
AI InfrastructureArtificial Intelligence (AI)Developer PlatformInformation TechnologyMachine LearningOpen Source
Responsibilities
Design and implement intelligent request routing systems that dynamically balance load across thousands of model replicas while maintaining strict latency SLAs
Build sophisticated traffic management systems that seamlessly transition between model versions at scale, handling terabytes of inference requests without dropping a single query
Create reactive systems that predict traffic patterns and scale model replicas from 1 to 10,000+ instances based on real-time demand signals
Architect frameworks for complex ML pipelines where dozens of models need to communicate, share resources, and maintain end-to-end latency guarantees
Build deep introspection tools that make it trivial to debug distributed ML applications—because "works on my laptop" doesn't cut it at scale
Qualification
Required
Strong Systems Fundamentals: You understand operating systems, networking, concurrency, and distributed systems at a deep level
Production Experience: You've built and maintained systems that serve real users at scale
Code Quality: You write clean, tested, well-documented code that other engineers love to work with
Ownership Mindset: You take responsibility for your code in production—from design to deployment to incident response
Preferred
Experience with distributed systems frameworks (gRPC, Ray)
Background in ML/AI systems or serving infrastructure
Contributions to major open source projects
Experience with performance optimization and profiling
Knowledge of cloud-native technologies (Kubernetes, Istio, etc.)
Company
Anyscale
Anyscale accelerates the development and productionization of any AI app, on any cloud, at any scale.
Founded in 2019
201-500 employees
H1B Sponsorship
Anyscale has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (33)
2024 (14)
2023 (10)
2022 (10)
2021 (4)
2020 (1)
Funding
Current Stage
Growth StageTotal Funding
$259MKey Investors
New Enterprise AssociatesAndreessen Horowitz
2022-08-23Series C· $99M
2021-12-07Series C· $100M
2020-10-21Series B· $40M
Leadership Team
Keerti Melkote
Chief Executive Officer
Robert Nishihara
Co-Founder
Recent News
Dynamic Business
2026-01-23
globalventuring.com
2026-01-12
Company data provided by crunchbase