engineering
Posted Mar 28Software Engineer, Workload Enablement
at openai
San Francisco, United StatesRemote
Responsibilities
- - Build a suite of benchmarks and stress tests that capture real E2E behavior of our workloads by exercising all aspects of a system, including CPU, GPU, memory subsystem, frontend, scale-up, and scale-out networking (including WAN traffic, NVlink and RDMA collectives), storage, thermals, and any other relevant parts.
Requirements
- We design and deliver advanced systems that support the deployment and operation of cutting-edge AI models.
- Qualifications - BS in CS/EE (or equivalent practical experience). - 5+ years in one or more of: ML systems, performance engineering, distributed systems, or HPC. - Strong hands-on
- experience with: - PyTorch and modern LLM training/inference stacks - Large-scale distributed training concepts (data/model/pipeline parallel, collective comms) -
- Experience with RDMA and debugging/optimizing comms libraries (NCCL or RCCL) and their interaction with hardware/network - Proficiency in Python plus comfort reading/writing performance-critical code (C++/CUDA/HIP is a plus). - Strong profiling/debugging skills (e.g., Nsight, rocprof, perf, flamegraphs; ability to reason from traces/counters). Preferred Skills -
- Experience building workload-shaped benchmarks and stress/fault tests that correlate to production behavior (not just synthetic loops or microbenchmarks). - Familiarity with RDMA networking and transport tuning; understanding of how network topology and congestion impact collectives. -
- Experience running and validating workloads in Kubernetes, and bridging “research code” into robust, repeatable infrastructure. - Hands-on lab
- experience with early hardware (new NICs, new GPUs/accelerators, early racks).
- About OpenAI OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence
Additional details
- About the Team The Scaling team is responsible for the architectural and engineering backbone of OpenAI’s infrastructure.
- Our work spans system software, networking, platform architecture, fleet-level monitoring, and performance optimization.
- About the Role We’re hiring an SW Engineer to enable production workloads and end-to-end testing on new platforms.
- This role will include creating new test harnesses and platform stress benchmarks, porting existing inference and training workloads to new, sometimes early-access, systems/hardware, analyzing performance and bottlenecks, and characterizing the end-to-end behavior of new systems (compute, comms, storage, control plane, and failure modes). Key
- Responsibilities - Port and validate key inference and training workloads on new platforms/SKUs as they arrive; drive correctness, performance, and stability to an internal readiness bar.
- - Deep-dive performance on distributed training/inference: - Collective performance and tuning (across NCCL/RCCL and internal libraries) - Overlap of compute/communication, kernel-level bottlenecks, memory bandwidth and scheduling effects - Create repeatable test harnesses that run in CI / lab environments and produce actionable outputs (pass/fail, performance score, regression detection).
- - Partner with systems + fleet bring-up engineers to ensure the platform is not only stable and performant, but also operationally usable and scalable (containerization, K8s integration, telemetry hooks, failure triage loops).
- - Work cross-functionally with vendors and internal stakeholders by producing clear bug reports, minimal repros, and prioritized issue lists.