From Homegrown to Flink: Migrating a Stateful Ad Event Join at Scale

Zalando Marketing Services replaced a 7-year-old homegrown in-memory join that matched Sponsored Products auction bids to clicks/views within 15 minutes. The old Java service read up to 200 MB/s, could not partition both streams cleanly, and lost up to 15 minutes of state on every deploy or rescale. After Interval Join burned 15-30% CPU on RocksDB seeks and held a ~70M-entry timer queue, they rewrote the match with KeyedCoProcessFunction (point lookups, RocksDB TTL expiry), incremental 3-minute checkpoints, and two-way state. A four-week shadow pipeline hit 99.9% enrichment parity, then a one-week A/B test; average pods fell from 20 to about 5, memory from ~320 GB to ~100 GB, and EC2 cost from about €80 to €30 per day, with enrichment up ~0.5%.

Published

2026-07-24

Scale

up to 200 MB/s on the old pipeline, average pods cut from a constant 20 to ~5 (range 1-8), join-operator subtasks 30-60 down to 10 max, memory ~320 GB to ~100 GB; EC2 ~€80 to ~€30/day, shadow match rate 99.9% vs production; +0.5% enrichment from two-way state

Techniques

stream joins checkpointing keyed state shadow traffic autoscaling

Stack

Apache Flink RocksDB Kubernetes Nakadi Kinesis Java

Read the original on Zalando's blog →