Netflix

Behind the Scenes: Evolving Netflix's Ads Event Pipeline for Live, Part II

Netflix's Ads Event Publisher used to enrich each client ad event by looking up metadata the ad server had written to a globally replicated key-value registry. For live events with tens of millions of concurrent viewers, that meant must-succeed database writes on the ad-serving critical path, a store that couldn't safely be scaled mid-event, and late events whenever lookups slowed. Netflix replaced it with an Apache Flink job: the ad server logs its responses to Kafka, and a custom KeyedCoProcessFunction joins them to client events over a 60-minute window, which catches roughly 99.999% of events. An hourly Spark job orchestrated by Maestro recovers the rest from the data warehouse. Revenue-bearing events are de-duplicated in local Flink state keyed by a hashed event ID. Both pipelines ran side by side with automated audits, and a stable per-event key moved publishing to Flink a slice at a time, region by region. When they agreed on well over 99.99% of events, Netflix retired the legacy service and the registry. In a load test, one AWS region handled an ad break at hundreds of thousands of events per second.

Published
2026-10-14
Scale
hundreds of thousands of events/sec for one ad break in a single AWS region (load test), ~99.999% of events joined within a 60-minute window, old and new pipelines agreed on well over 99.99% of events before cutover, 25,000+ Flink jobs run at Netflix
Techniques
streaming joins deduplication shadow traffic staged rollout zero downtime
Stack
Flink Kafka Spark Maestro

Try this stack yourself

  • Better Stack · Monitor uptime and logs with Better Stack, and get alerted fast if a migration or traffic spike goes sideways on your own system.
    Better Stack isn't part of Netflix's setup in this story. It's a tool for watching your own systems while you try ideas like these.

Some links above are affiliate links: if you sign up, How They Scaled may earn a commission at no extra cost to you. We only list tools that fit the stack in the story.

See also

Netflix architecture

The best scaling write-ups, weekly

One email a week. No spam.

Open subscribe page

Related stories

Netflix · 2026

TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads

Netflix moved older data in its multi-petabyte TimeSeries Abstraction out of hot Cassandra storage in two steps. First, Kafka and Flink streamed events to S3 as Parquet files, which a daily job compacted into small, time-sorted row groups with Bloom filters cached in EVCache (p50 ~150 ms, p99 ~500 ms). The second design skips that streaming pipeline: a ReadOnlyCompactionStrategy aligns immutable SSTables to token ranges, and a custom pluggable Memtable (Cassandra 4.1) reads directly from existing Cassandra backups in S3. It now holds over 15 PB of compressed cold data, cut disk usage on affected clusters from 80% to 7%, and improved p90 latency by ~30%.

15+ PB compressed data in cold storagedisk usage cut from 80% to 7%tiered storagecompactioncost optimizationCassandraKafkaFlink
Notion · 2024

Building and scaling Notion's data lake

Notion replaced slow warehouse syncs with a CDC pipeline (Debezium to Kafka to Hudi on S3) processed with Spark.

change data capturedata lakeincremental ingestionKafkaDebeziumApache Hudi
Netflix · 2026

Single-tenant Titus: VM-level isolation on a multi-tenant container platform

Titus, Netflix's Kubernetes-based container platform, launches millions of containers a week on tens of thousands of EC2 instances, packing many teams onto shared hosts. cgroups can't fence last-level cache, memory bandwidth or the global lock in HTB network shaping, so a small-packet flood from one container could starve its neighbors, and stateful streaming jobs like Flink would fall behind and need minutes to catch up. After ruling out microVMs (Firecracker, Kata), gVisor and moving teams to plain EC2, Netflix added an allowlisted single-tenant mode: the federation layer tags eligible tasks, the autoscaler launches the smallest instance that fits each one just in time, and anti-affinity, taints, no preemption and no repacking keep it alone on its host, all behind the same Titus API. Flink, streaming connectors and telemetry pipelines now report stable operation, and the shared fleet packs more densely because sensitive workloads no longer need anti-affinity spreading. The cost is slower launches, since every task waits for its own instance, and fixed host overhead that can take tens of percent of a small node.

millions of containers launched per weektens of thousands of EC2 instances across multiple AWS regionsworkload isolationmulti-tenancyautoscalingTitusKubernetesAWS EC2
OpenAI · 2026

Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

OpenAI's Habitat storage platform went from one shared Azure Cosmos DB container, to dedicated single-tenant containers that pushed capacity planning onto every product team, and back to a shared container by default once Azure added hot-key throttling, request-priority QoS and a new serverless mode where each physical partition scales to 10K RU/s and splits on RU usage. After failover and failback outages in late 2024, OpenAI and Azure now co-own testing and monitoring. Replicating small latency-critical datasets widely, a Valkey lookaside cache invalidated through the Cosmos DB change feed, and a Rust rewrite of the proxy cut global read p50 from about 120 ms to about 10 ms.

1B+ users across 40 regions500+ PB served by Habitatmulti-tenancyhot key throttlingCDC cache invalidationAzure Cosmos DBValkeyKafka