How teams scaled Kafka: 9 real case studies
9 of the 94 engineering write-ups in our database mention Kafka in their stack. They come from 7 companies (Datadog, Netflix, Cloudflare, GitHub, LinkedIn and 2 more) and were published between 2015 and 2026. Techniques that show up most often alongside Kafka: multi-tenancy (2) and queues (2). It's often paired with Flink (2), Go (2), Rust (2) and Spark (2). 6 of the 9 state concrete scale numbers, for example "1B+ users across 40 regions" (OpenAI). Each card below links to our short summary and the original post.
Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)
OpenAI's Habitat storage platform went from one shared Azure Cosmos DB container, to dedicated single-tenant containers that pushed capacity planning onto every product team, and back to a shared container by default once Azure added hot-key throttling, request-priority QoS and a new serverless mode where each physical partition scales to 10K RU/s and splits on RU usage. After failover and failback outages in late 2024, OpenAI and Azure now co-own testing and monitoring. Replicating small latency-critical datasets widely, a Valkey lookaside cache invalidated through the Cosmos DB change feed, and a Rust rewrite of the proxy cut global read p50 from about 120 ms to about 10 ms.
TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads
Netflix moved older data in its multi-petabyte TimeSeries Abstraction out of hot Cassandra storage in two steps. First, Kafka and Flink streamed events to S3 as Parquet files, which a daily job compacted into small, time-sorted row groups with Bloom filters cached in EVCache (p50 ~150 ms, p99 ~500 ms). The second design skips that streaming pipeline: a ReadOnlyCompactionStrategy aligns immutable SSTables to token ranges, and a custom pluggable Memtable (Cassandra 4.1) reads directly from existing Cassandra backups in S3. It now holds over 15 PB of compressed cold data, cut disk usage on affected clusters from 80% to 7%, and improved p90 latency by ~30%.
Behind the Scenes: Evolving Netflix's Ads Event Pipeline for Live, Part II
Netflix's Ads Event Publisher used to enrich each client ad event by looking up metadata the ad server had written to a globally replicated key-value registry. For live events with tens of millions of concurrent viewers, that meant must-succeed database writes on the ad-serving critical path, a store that couldn't safely be scaled mid-event, and late events whenever lookups slowed. Netflix replaced it with an Apache Flink job: the ad server logs its responses to Kafka, and a custom KeyedCoProcessFunction joins them to client events over a 60-minute window, which catches roughly 99.999% of events. An hourly Spark job orchestrated by Maestro recovers the rest from the data warehouse. Revenue-bearing events are de-duplicated in local Flink state keyed by a hashed event ID. Both pipelines ran side by side with automated audits, and a stable per-event key moved publishing to Flink a slice at a time, region by region. When they agreed on well over 99.99% of events, Netflix retired the legacy service and the registry. In a load test, one AWS region handled an ad break at hundreds of thousands of events per second.
Evolving our real-time timeseries storage again: Built in Rust for performance at scale
Datadog rebuilt its real-time timeseries database (RTDB) storage engine in Rust as Monocle after outgrowing Cassandra, Redis, MDBM, and a Go B+ tree plus a separate RocksDB path for DDSketch distributions. The design shards the keyspace early with a shard-per-core model (for example, 24 storage shards on a 32-core machine), ingesting from Kafka partitions over MPSC channels into per-shard LSM trees on single-threaded Tokio schedulers, with tiered compaction, a unified series cache, and permit- and cost-based query throttlers (including CoDel). At peak scale the rewrite delivered a 60x increase in ingestion performance and 5x faster queries, turning extreme outliers that used to take minutes into results that return in seconds.
Building and scaling Notion's data lake
Notion replaced slow warehouse syncs with a CDC pipeline (Debezium to Kafka to Hudi on S3) processed with Spark.
How we improved push processing on GitHub
GitHub broke one huge push-processing job into smaller, independently retried jobs driven by Kafka events.
Husky: Exactly-once ingestion and multi-tenancy at scale
A deep dive on how Husky routes, shards and ingests events exactly once across tenants.
HTTP Analytics for 6M requests per second using ClickHouse
Cloudflare rebuilt its HTTP analytics pipeline on ClickHouse, replacing a Postgres/Citus setup.
Running Kafka At Scale
How LinkedIn runs Kafka as its central data pipeline, including cluster tiers and mirroring between datacenters.