engineering
Posted May 11Software Engineer, Hardware Health
at openai
San Francisco, United StatesOn-site
Responsibilities
- - Build and evolve health checks that detect, remediate, and verify failures at scale.
- - Build automation and tooling that enables global cluster management with minimal manual intervention.
Requirements
- ABOUT THE TEAM The Hardware Health and Observability team owns the end-to-end health lifecycle of OpenAI’s global compute fleet.
- We build the systems that observe, detect, remediate, and verify hardware issues across GPUs, CPUs, networking, and platform infrastructure, enabling frontier model training and inference workloads to run reliably at hyperscale.
- you will: - Define and maintain health signals across GPUs, CPUs, networking, and platform infrastructure.
- experience in software or infrastructure engineering. - Strong proficiency with Python and shell scripting. -
- Experience building large-scale distributed systems or infrastructure platforms. - Comfort digging into noisy operational data using SQL, PromQL, or similar tooling. -
- Experience with low-level hardware systems and Linux tooling (e.g. PCIe, InfiniBand, RoCE, networking, power management, kernel performance tuning, FW/SW debugging). -
- Experience operating or debugging large-scale GPU or accelerator clusters. - Expertise in network operations, observability, or systems telemetry. -
- Experience with automated remediation systems or fleet lifecycle management. -
- Experience improving reliability, utilization, or workload uptime in distributed compute environments.
- About OpenAI OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence
Experience
- YOU MIGHT THRIVE IN THIS ROLE IF YOU HAVE: - 7+ years of industry
Benefits
- Experience building reproducible analyses and operational tooling. - Strong systems debugging and operational instincts with an ownership mindset. BONUS IF YOU HAVE: -
Additional details
- Our mission is to maximize healthy, usable compute across accelerator vendors, generations, cloud providers, and regions through reliable health signals, automated remediation, and scalable operational tooling.
- We are the last line of defense for the success of OAI’s production and research workloads.
- ABOUT THE ROLE On the Hardware Health and Observability team, you’ll build critical infrastructure that keeps OpenAI’s largest compute clusters healthy and operational at scale.
- Even small numbers of unhealthy systems can impact large-scale training and inference workloads.
- This team focuses on minimizing downtime, improving fleet efficiency, and ensuring compute resources remain continuously available to researchers and product teams.
- Engineers on this team own problems end-to-end, from defining health signals and debugging failures to building automated remediation systems that operate across millions of GPUs globally. In this role,
- - Ensure critical health checks execute with minimal latency to maximize workload uptime.
- - Investigate hardware failures and system-level issues across large-scale compute environments.
- - Partner with workload, reliability, and provider teams to integrate health signals into training and inference systems.