Storage/compute separation: 5 real scaling case studies
5 of the 94 engineering write-ups in our database are tagged with the technique "storage/compute separation". They come from 5 companies (Datadog, GitHub, GitLab, Neon and OpenAI) and were published between 2022 and 2026. Stack that shows up most often in these stories: Git (3) and Object storage (2). 4 of the 5 state concrete scale numbers, for example "nearly a petabyte of Git data/day" (OpenAI). Each card below links to our short summary and the original post.
For a managed-database migration, see Atlassian's move of about four million Jira databases from Amazon RDS Postgres to Aurora.
Scaling Git at OpenAI
OpenAI built Pando, a Git-compatible backend for its monorepo after Codex pushed concurrent branches, merges and fetches past the old system (reliability sometimes below 90%). Pando stores each push as an immutable pack in blob storage and records per-branch ref versions in a horizontally scalable metadata store, giving serializability per ref and causal consistency across the repo so unrelated branches do not contend. Instead of building unique packs per client, it reuses stored packs and serves them via packfile-uri through regional Express caches (over 99% hit rate), with LSM-style background compaction so fetches plan in logarithmic time. Deployed first as a read/write-through proxy, then cut over with a Git CLI wrapper; it now powers all monorepo development, cut median push latency by nearly 5x, and can deliver nearly a petabyte of Git data per day (57+ TB in a single hour).
The bridge to agent-scale Git infrastructure
GitHub has copied 99.99999% of its Git repositories to Azure, where most now keep two of their five replicas and Azure serves about half of Git traffic. The busiest remaining repositories were blocked by a ~25ms cross-region round trip, because Spokes held a repository-wide lock for each write while waiting on the slowest healthy replica. Scoping the quorum to individual Git references lets writes to different branches proceed in parallel (3-4x higher throughput in internal testing), and a virtual write replica that streams writes into Azure Blob Storage lets stateless read replicas scale out on demand.
Scaling Git for the agentic software factory
GitLab is building next-gen source control for thousands of concurrent AI agents working on large repositories. With conventional Git, a single agent invocation can mean 5 to 10 GB of data transfer and more than 30 seconds of setup just to answer one question. Next-gen SCM keeps the Git protocol but runs stateless compute that scales horizontally against shared object storage, including S3-compatible storage customers control, and lets agents request only the files and history they need instead of cloning whole repos. In early internal tests, workloads ran up to 50x faster with up to 1,000x less network traffic, which GitLab calls peak results it is still validating. It also notes CI is the next bottleneck: GitLab.com customers created 40% more pipelines and pushed 50% more code over the past year.
Inside branch-based restores in Lakebase Postgres
Traditional managed Postgres PITR provisions a new compute+disk monolith, pulls a snapshot out of object storage, and replays WAL—hours at large size—while a survey Neon cites of 50 production Postgres developers found 30% down 3+ hours after a critical failure. Lakebase splits compute from durable history: safekeepers quorum-ack WAL, a pageserver materializes page versions, and object storage keeps immutable history, with 99.99999% of reads served from local caches. A restore is then a metadata branch at a timestamp rather than a copy-and-replay, so time-to-restore drops to seconds even for a 100 TB database.
Introducing Husky, Datadog's third-generation event store
Datadog's Husky is a distributed, schemaless column store that keeps event data on object storage.