Zalando · 2026
From Homegrown to Flink: Migrating a Stateful Ad Event Join at Scale
Zalando Marketing Services replaced a 7-year-old homegrown in-memory join that matched Sponsored Products auction bids to clicks/views within 15 minutes. The old Java service read up to 200 MB/s, could not partition both streams cleanly, and lost up to 15 minutes of state on every deploy or rescale. After Interval Join burned 15-30% CPU on RocksDB seeks and held a ~70M-entry timer queue, they rewrote the match with KeyedCoProcessFunction (point lookups, RocksDB TTL expiry), incremental 3-minute checkpoints, and two-way state. A four-week shadow pipeline hit 99.9% enrichment parity, then a one-week A/B test; average pods fell from 20 to about 5, memory from ~320 GB to ~100 GB, and EC2 cost from about €80 to €30 per day, with enrichment up ~0.5%.
up to 200 MB/s on the old pipelineaverage pods cut from a constant 20 to ~5 (range 1-8)stream joinscheckpointingkeyed stateApache FlinkRocksDBKubernetes