OpenAI · 2 weeks ago
Software Engineer, Load Balancing - Inference
San Francisco, CA
Full-time
Onsite
Senior Level
$325K/yr - $490K/yr
5+ years expOpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. They are seeking a senior engineer to design and build a load balancer for their research inference stack, focusing on routing AI models with high precision and reliability.
Agentic AIArtificial Intelligence (AI)Foundational AIGenerative AIMachine LearningNatural Language ProcessingSaaS
Responsibilities
Architect and build the gateway / network load balancer that fronts all research jobs, ensuring long-lived connections remain consistent and performant
Design traffic stickiness and routing strategies that optimize for both reliability and throughput
Instrument and debug complex distributed systems — with a focus on building world-class observability and debuggability tools (distributed tracing, logging, metrics)
Collaborate closely with researchers and ML engineers to understand how infrastructure decisions impact model performance and training dynamics
Own the end-to-end system lifecycle: from design and code to deploy, operate, and scale
Work in an outcome-oriented environment where everyone contributes across layers of the stack, from infra plumbing to performance tuning
Qualification
Required
Have deep experience designing and operating large-scale distributed systems, particularly load balancers, service gateways, or traffic routing layers
Have 5+ years of experience designing in theory for and debugging in practice for the algorithmic and systems challenges of consistent hashing, sticky routing, and low-latency connection management
Have 5+ years of experience as a software engineer and systems architect working on high-scale, high-reliability infrastructure
Have a strong debugging mindset and enjoy spending time in tracing, logs, and metrics to untangle distributed failures
Are comfortable writing and reviewing production code in Rust or similar systems languages (C/C++, Java, Go, Zig, etc)
Have operated in big tech or high-growth environments and are excited to apply that experience in a faster-moving setting
Take ownership of problems end-to-end and are excited to build something foundational to how our models interact with the world
Preferred
Experience with gateway or load balancing systems (e.g., Envoy, gRPC, custom LB implementations)
Familiarity with inference workloads (e.g., reinforcement learning, streaming inference, KV cache management, etc)
Exposure to debugging and operational excellence practices in large production environments
Company
OpenAI
OpenAI is an AI research and deployment company that develops advanced AI models, including ChatGPT. It is a sub-organization of OpenAI Foundation.
201-500 employees
H1B Sponsorship
OpenAI has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (1)
2024 (1)
2023 (1)
2022 (18)
2021 (10)
2020 (6)
Funding
Current Stage
Growth StageTotal Funding
$79BKey Investors
The Walt Disney CompanySoftBankThrive Capital
2025-12-11Corporate Round· $1B
2025-10-02Secondary Market· $6.6B
2025-03-31Series Unknown· $40B
Leadership Team
Sam Altman
CEO & Co-Founder
Greg Brockman
President, Chairman, & Co-Founder
Recent News
2026-01-09
The Motley Fool
2026-01-09
Company data provided by crunchbase