How teams scaled Spark: 2 real case studies
2 of the 94 engineering write-ups in our database mention Spark in their stack. They come from 2 companies (Netflix and Notion) and were published between 2024 and 2026. It's often paired with Kafka (2). 1 of the 2 state concrete scale numbers, for example "hundreds of thousands of events/sec for one ad break in a single AWS region (load test)" (Netflix). Each card below links to our short summary and the original post.
Behind the Scenes: Evolving Netflix's Ads Event Pipeline for Live, Part II
Netflix's Ads Event Publisher used to enrich each client ad event by looking up metadata the ad server had written to a globally replicated key-value registry. For live events with tens of millions of concurrent viewers, that meant must-succeed database writes on the ad-serving critical path, a store that couldn't safely be scaled mid-event, and late events whenever lookups slowed. Netflix replaced it with an Apache Flink job: the ad server logs its responses to Kafka, and a custom KeyedCoProcessFunction joins them to client events over a 60-minute window, which catches roughly 99.999% of events. An hourly Spark job orchestrated by Maestro recovers the rest from the data warehouse. Revenue-bearing events are de-duplicated in local Flink state keyed by a hashed event ID. Both pipelines ran side by side with automated audits, and a stable per-event key moved publishing to Flink a slice at a time, region by region. When they agreed on well over 99.99% of events, Netflix retired the legacy service and the registry. In a load test, one AWS region handled an ad break at hundreds of thousands of events per second.
Building and scaling Notion's data lake
Notion replaced slow warehouse syncs with a CDC pipeline (Debezium to Kafka to Hudi on S3) processed with Spark.