Discord · 2026
How Discord Automates ScyllaDB Clusters at Scale
Discord's 7-person persistence team replaced fragile Python and bash scripts with the Scylla Control Plane, a Rust framework of idempotent tasks with declared preconditions, composed into YAML workflows. Jobs batch nodes by availability zone with concurrency limits to protect quorum, store their state in SQLite so they resume after interruptions, and send a webhook alert on unrecoverable errors. Operations like standing up shadow clusters and rolling upgrades went from a day and a half of babysitting to under two hours.
dozens of clusters, hundreds of database nodes7-person persistence teamdatabase operations automationidempotent taskszone-aware rolloutsScyllaDBRustYAML
OpenAI · 2026
Scaling Git at OpenAI
OpenAI built Pando, a Git-compatible backend for its monorepo after Codex pushed concurrent branches, merges and fetches past the old system (reliability sometimes below 90%). Pando stores each push as an immutable pack in blob storage and records per-branch ref versions in a horizontally scalable metadata store, giving serializability per ref and causal consistency across the repo so unrelated branches do not contend. Instead of building unique packs per client, it reuses stored packs and serves them via packfile-uri through regional Express caches (over 99% hit rate), with LSM-style background compaction so fetches plan in logarithmic time. Deployed first as a read/write-through proxy, then cut over with a Git CLI wrapper; it now powers all monorepo development, cut median push latency by nearly 5x, and can deliver nearly a petabyte of Git data per day (57+ TB in a single hour).
nearly a petabyte of Git data/daymedian push latency cut nearly 5x (~80%)per-ref causal consistencyreusable packfilesstorage/compute separationGitPandoRust