OpenAI architecture: how OpenAI scaled (5 case studies)
OpenAI runs ChatGPT on a single-primary PostgreSQL database on Azure with nearly 50 global read replicas, serving 800 million users and millions of QPS. Reads are offloaded to replicas, write-heavy shardable workloads move to Cosmos DB, and PgBouncer, cache leasing and multi-layer rate limits keep the primary safe. Its Habitat storage platform grew from a Python client library over Cosmos DB into a central service handling 70M+ requests per second, later rewritten in Rust.
We track 5 public engineering write-ups by OpenAI, published in 2026. Techniques that come up more than once: connection pooling (2) and rate limiting (2). Stack mentioned most: Rust (3), Cosmos DB (2), Kubernetes (2) and Python (2). 5 of the 5 state concrete scale numbers, for example "nearly a petabyte of Git data/day". Each card below links to our short summary and the original post.
Tech used in OpenAI's scaling write-ups
OpenAI scaling timeline: every story, summarized
- 2026-01-22
Scaling PostgreSQL to power 800 million ChatGPT users
OpenAI runs ChatGPT on a single-primary Postgres with nearly 50 global read replicas. Reads are offloaded, write-heavy shardable workloads move to Cosmos DB, and PgBouncer, cache leasing and multi-layer rate limits keep the primary safe.
- 2026-05-04
How OpenAI delivers low-latency voice AI at scale
OpenAI rebuilt its WebRTC stack for ChatGPT voice and the Realtime API after one-port-per-session media termination collided with Kubernetes: large public UDP ranges, sticky ICE/DTLS ownership, and global first-hop latency. A lightweight Go relay exposes a small fixed UDP surface, reads ICE ufrag routing hints from the first STUN packet, and forwards opaque media to the transceiver that owns ICE, DTLS, and SRTP state (with Redis-backed recovery). Cloudflare geo and proximity steering place signaling near users while Global Relay shortens the first media hop, preserving standard WebRTC for clients without making inference backends WebRTC peers.
- 2026-09-11
Rapidly scaling online storage to serve over 1 billion ChatGPT users
OpenAI's Habitat storage platform grew from a Python client library over Cosmos DB into a central service handling 70M+ requests per second. The team ran many low-concurrency Python processes to keep asyncio delay down, switched aiohttp connection reuse from LIFO to FIFO to stop a metastable failure, used Envoy for connection fan-in, kept a deliberately simple NoSQL API, then rewrote the service in Rust.
- 2026-10-23
Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)
OpenAI's Habitat storage platform went from one shared Azure Cosmos DB container, to dedicated single-tenant containers that pushed capacity planning onto every product team, and back to a shared container by default once Azure added hot-key throttling, request-priority QoS and a new serverless mode where each physical partition scales to 10K RU/s and splits on RU usage. After failover and failback outages in late 2024, OpenAI and Azure now co-own testing and monitoring. Replicating small latency-critical datasets widely, a Valkey lookaside cache invalidated through the Cosmos DB change feed, and a Rust rewrite of the proxy cut global read p50 from about 120 ms to about 10 ms.
- 2026-11-02
Scaling Git at OpenAI
OpenAI built Pando, a Git-compatible backend for its monorepo after Codex pushed concurrent branches, merges and fetches past the old system (reliability sometimes below 90%). Pando stores each push as an immutable pack in blob storage and records per-branch ref versions in a horizontally scalable metadata store, giving serializability per ref and causal consistency across the repo so unrelated branches do not contend. Instead of building unique packs per client, it reuses stored packs and serves them via packfile-uri through regional Express caches (over 99% hit rate), with LSM-style background compaction so fetches plan in logarithmic time. Deployed first as a read/write-through proxy, then cut over with a Git CLI wrapper; it now powers all monorepo development, cut median push latency by nearly 5x, and can deliver nearly a petabyte of Git data per day (57+ TB in a single hour).
FAQ
Does ChatGPT use PostgreSQL?
Yes. OpenAI's 2026 post says ChatGPT runs on a single-primary PostgreSQL with nearly 50 global read replicas, for 800 million users and millions of QPS, with write-heavy shardable workloads moved to Cosmos DB.
How does OpenAI protect its Postgres primary?
Per OpenAI's post: reads go to replicas, PgBouncer pools connections, cache leasing reduces load, multi-layer rate limits cap traffic, and write-heavy workloads that can be sharded move to Cosmos DB.
What is OpenAI's Habitat storage platform?
Habitat grew from a Python client library over Cosmos DB into a central storage service handling 70M+ requests per second, using Envoy for connection fan-in and a deliberately simple NoSQL API, before being rewritten in Rust. The post cites 1 billion+ weekly users and 500 PB+.