How real companies scaled.
54 engineering write-ups from 22 companies, searchable by stack, scale and technique.
The best scaling write-ups, weekly
Under the hood, off the hot path: how D1 scales to millions of databases
How Cloudflare's D1 keeps query execution at the edge while a separate control plane manages millions of SQLite databases.
Scaling PostgreSQL to power 800 million ChatGPT users
OpenAI runs ChatGPT on a single-primary Postgres with nearly 50 global read replicas. Reads are offloaded, write-heavy shardable workloads move to Cosmos DB, and PgBouncer, cache leasing and multi-layer rate limits keep the primary safe.
Haydex: From Zero to 178,600,000,000 rows a second in 30 days
Axiom turned a failed filter-index prototype into a production system for its event store, letting queries skip data and scan 178.6B rows per second.
Migrating the Jira Database Platform to AWS Aurora
Atlassian migrated about four million Jira databases from RDS Postgres to Aurora, balancing reliability, downtime and cost.
How we sped up Notion in the browser with WASM SQLite
Notion cached data in the browser with WASM SQLite and OPFS, improving page navigation times.
Building and scaling Notion's data lake
Notion replaced slow warehouse syncs with a CDC pipeline (Debezium to Kafka to Hudi on S3) processed with Spark.
Timeseries indexing at scale
Datadog redesigned its timeseries indexing to handle a large jump in ingested data.
How Meta trains large language models at scale
Meta covers the hardware, networking and reliability work needed to train LLMs across very large GPU clusters.
How we improved push processing on GitHub
GitHub broke one huge push-processing job into smaller, independently retried jobs driven by Kafka events.
How Stripe's document databases supported 99.999% uptime with zero-downtime data migrations
Stripe's DocDB and its Data Movement Platform move data between shards without downtime.
How Figma's Databases Team Lived to Tell the Scale
Figma's nine-month effort to horizontally shard its Postgres stack using a custom query proxy and logical sharding before physical splits.
Horizontally scaling the Rails backend of Shop app with Vitess
Shopify's Shop app adopted Vitess to shard its Rails backend, covering keyspace design and the migration plan.
Upgrading GitHub.com to MySQL 8.0
How GitHub upgraded its whole MySQL fleet to 8.0 with staged replicas and a working rollback path.
How We Scaled PostgreSQL to 350 TB+
Timescale dogfooded its own Postgres extension to build a database-observability product, using hypertables, columnar compression and tiered storage to keep a 350 TB+ dataset queryable.
Supavisor: Scaling Postgres to 1 Million Connections
Supabase load-tested Supavisor, its Elixir Postgres pooler, with a million client connections.
The Great Re-shard: adding Postgres capacity (again) with zero downtime
Notion tripled its Postgres fleet by re-sharding with logical replication and careful PgBouncer cutovers, with no downtime.
Real-time Messaging
An overview of Slack's channel and gateway servers that fan messages out to millions of connected clients in real time.
The growing pains of database architecture
How Figma moved from one Postgres instance to multiple databases by vertically partitioning table groups.
How Discord Stores Trillions of Messages
Discord moved its message store from Cassandra to ScyllaDB and put Rust data services in front of it to coalesce hot-partition requests.
Husky: Exactly-once ingestion and multi-tenancy at scale
A deep dive on how Husky routes, shards and ingests events exactly once across tenants.
How we built Pingora, the proxy that connects Cloudflare to the Internet
Cloudflare replaced Nginx with Pingora, a Rust proxy that reuses connections better and needs less CPU and memory.
Log analytics using ClickHouse
Cloudflare moved its error-log pipeline to ClickHouse and shares schema and query lessons.
One million queries per second with MySQL
PlanetScale benchmarked a horizontally sharded Vitess cluster up to one million QPS.
We are splitting our database into Main and CI
GitLab.com split CI tables into their own Postgres database to relieve its main database.
Introducing Husky, Datadog's third-generation event store
Datadog's Husky is a distributed, schemaless column store that keeps event data on object storage.
Switching from etcd to SQLite: A Database Migration Story
Tailscale replaced etcd with SQLite plus Litestream replication for a simpler control-plane datastore.
LiveGraph: real-time data fetching at Figma
Figma built LiveGraph, which tails the Postgres replication log so its UI can subscribe to live query results.
Herding elephants: Lessons learned from sharding Postgres at Notion
Notion split its Postgres monolith into a fleet of logical shards keyed by workspace, using double-writes and backfills to migrate.
Why we spent the last month eliminating PostgreSQL subtransactions
GitLab traced replica stalls to Postgres subtransaction overflow and removed subtransactions from its code.
Partitioning GitHub's relational databases to handle scale
GitHub defined schema domains and linted cross-domain queries so it could move table groups onto separate MySQL clusters.
Shard Balancing: Moving Shops Confidently with Zero-Downtime at Terabyte-scale
Shopify rebalances MySQL shards by moving individual shops online with its Ghostferry tool.
ZippyDB: Facebook's key value store
Meta's general-purpose distributed key-value store built on RocksDB, with shard management and configurable consistency.
Read Consistency with Database Replicas
Shopify routes reads to MySQL replicas while giving apps consistency guarantees such as monotonic reads.
Capacity Planning at Scale
How Shopify plans and checks capacity for traffic peaks like BFCM using forecasts and scalability tests.
Scaling Datastores at Slack with Vitess
Slack's multi-year migration from active-active MySQL pairs to Vitess, so data could be sharded by any key, not only workspace.
Introducing Quicksilver: Configuration Distribution at Internet Scale
Quicksilver replicates Cloudflare's config key-value store to every edge server in seconds, built on LMDB.
Why Discord is switching from Go to Rust
Discord rewrote its Read States service from Go to Rust to get rid of garbage-collection latency spikes in a large in-memory cache.
How we optimized Magic Pocket for cold storage
Dropbox added a cold tier to Magic Pocket that cuts replication overhead for rarely read data.
Goodbye Microservices
Segment folded more than 100 destination microservices back into a monolith to cut operational overhead.
How Mozilla's Rust dramatically improved our server-side performance
Figma rewrote its multiplayer server in Rust to cut latency and memory use as documents and users grew.
HTTP Analytics for 6M requests per second using ClickHouse
Cloudflare rebuilt its HTTP analytics pipeline on ClickHouse, replacing a Postgres/Citus setup.
How Discord Scaled Elixir to 5,000,000 Concurrent Users
Discord tuned its Elixir guild processes and fan-out paths, building custom libraries to push events to millions of concurrent users.
Flannel: An Application-Level Edge Cache to Make Slack Scale
Slack added Flannel, an edge cache that serves user and channel data lazily so huge teams no longer need giant boot payloads.
Scaling your API with rate limiters
The four kinds of rate limiters and load shedders Stripe uses to protect its API.
How Discord Indexes Billions of Messages
Discord built message search on many small Elasticsearch clusters, sharding by guild and indexing lazily through a queue.
Online migrations at scale
Stripe's four-step dual-write playbook for migrating data models without downtime.
How Discord Stores Billions of Messages
Early story of Discord outgrowing MongoDB and moving messages to Cassandra with a channel+time-bucket partition key.
FollowFeed: LinkedIn's Feed Made Faster and Smarter
LinkedIn moved its feed to a fan-out-on-read system that keeps member timelines in memory and on RocksDB.
Stack Overflow: The Architecture - 2016 Edition
A tour of how Stack Overflow serves a huge site from a small number of powerful servers with heavy caching.
Running Kafka At Scale
How LinkedIn runs Kafka as its central data pipeline, including cluster tiers and mirroring between datacenters.
TAO: The power of the graph
TAO is Facebook's geo-distributed graph cache over sharded MySQL, serving the social graph at more than a billion reads per second.
Scaling to exabytes and beyond
Dropbox built Magic Pocket, its own exabyte-class block store, and moved user data off third-party cloud storage.
How we migrated Dropbox from Nginx to Envoy
Dropbox moved its edge traffic proxy from Nginx to Envoy for better observability and extensibility.
Rewriting the heart of our sync engine
Dropbox rewrote its desktop sync engine in Rust with a new data model built for correctness and testability.