Datadog architecture: how Datadog scaled (9 case studies)
Datadog's Husky is its third-generation event store: a distributed, schemaless column store that keeps event data on object storage, written in Go, with exactly-once ingestion and multi-tenant sharding built on Kafka and FoundationDB. Datadog also redesigned its timeseries indexing in Rust on RocksDB to handle a large jump in ingested data.
We track 9 public engineering write-ups by Datadog, published between 2022 and 2026. Techniques that come up more than once: multi-tenancy (2). Stack mentioned most: PostgreSQL (3), Rust (3), Go (2), Kafka (2) and Kubernetes (2). 6 of the 9 state concrete scale numbers, for example "+53% avg write latency with remote_apply (pgbench TPC-B)". Each card below links to our short summary and the original post.
Tech used in Datadog's scaling write-ups
Datadog scaling timeline: every story, summarized
- 2022-05-17
Introducing Husky, Datadog's third-generation event store
Datadog's Husky is a distributed, schemaless column store that keeps event data on object storage.
- 2023-02-22
Husky: Exactly-once ingestion and multi-tenancy at scale
A deep dive on how Husky routes, shards and ingests events exactly once across tenants.
- 2024-06-28
Timeseries indexing at scale
Datadog redesigned its timeseries indexing to handle a large jump in ingested data.
- 2025-02-19
Achieving relentless Kafka reliability at scale with the Streaming Platform
Datadog treats Kafka as commodity hardware behind a Streaming Platform control layer. Streams span multiple Kafka clusters, AZs, and Kubernetes clusters so producers and consumers talk to stable identifiers while the control plane redirects traffic to healthier topics in seconds—for incidents, rebalancing, decommissioning, or repartitioning—without app redeploys. A custom Assigner replaces Kafka's group coordinator for multi-cluster rebalancing in seconds; Stream lanes isolate QoS and park poison pills in a DLQ; an enriched commit log tracks multiple offset ranges plus timestamps so consumers can process live traffic while replaying backlogs and the control plane can compute wall-clock lag across millions of partitions. After zstd "bombs" crashed stock clients, Datadog wrote libstreaming in Rust (with Java/Go/Python bindings) for unified, high-throughput access without a proxy in the TB/s data path.
- 2025-06-17
Breaking up a monolith: How we're unwinding a shared database at scale
Datadog is splitting its large shared Postgres database into team-owned instances in three phases. Logical separation moves tables into about 30 private domain schemas with explicit ACLs, using query samples and interviews with 30+ teams. Access reduction routes services through per-team PG Proxies so ACLs can be narrowed safely, and physical separation replicates each schema to a managed OrgStore instance, moves reads first and then writes with under five seconds of downtime, and replicates back for rollback. Two workloads have fully migrated and more than 70 dead tables were deleted along the way.
- 2025-08-04
Evolving our real-time timeseries storage again: Built in Rust for performance at scale
Datadog rebuilt its real-time timeseries database (RTDB) storage engine in Rust as Monocle after outgrowing Cassandra, Redis, MDBM, and a Go B+ tree plus a separate RocksDB path for DDSketch distributions. The design shards the keyspace early with a shard-per-core model (for example, 24 storage shards on a 32-core machine), ingesting from Kafka partitions over MPSC channels into per-shard LSM trees on single-threaded Tokio schedulers, with tiered compaction, a unified series cache, and permit- and cost-based query throttlers (including CoDel). At peak scale the rewrite delivered a 60x increase in ingestion performance and 5x faster queries, turning extreme outliers that used to take minutes into results that return in seconds.
- 2025-11-18
Scaling real-time file monitoring with eBPF: How we filtered billions of kernel events per minute
Datadog's eBPF-based file integrity monitoring sees more than 10 billion file events per minute, and at about 5 KB each, shipping them all would mean terabytes per second. Agent-side rules cut what goes upstream to roughly one million events per minute, and moving evaluation into the kernel stops the ring buffer from overflowing: static approvers compiled from detection rules let matching events through, while runtime discarders stored in LRU eBPF maps drop values no rule can match. Together they pre-filter up to 94% of events in the kernel with no dropped events.
- 2026-03-23
When upserts don't update but still write: Debugging Postgres performance at scale
Datadog tracks last-seen time for millions of ephemeral hosts so inactive ones can be deleted after 7 days. An ON CONFLICT DO UPDATE upsert meant to touch each host at most once a day still locked the conflicting row and wrote Heap LOCK plus COMMIT WAL records even when the WHERE clause skipped the update—at 500 upserts/sec that roughly doubled write IOPS and quadrupled WAL syncs, and at 25,000/sec it would overwhelm a single writer. Using pg_walinspect and a debugger on heap_lock_tuple, they replaced it with an ON CONFLICT DO NOTHING CTE plus a conditional UPDATE that generates no WAL on true no-ops, keeps HOT updates when a day has passed, and preserves correctness for coarse host tracking.
- 2026-06-04
When failover isn't safe: Building high-availability PostgreSQL on Kubernetes
A gameday that injected network latency into one availability zone left several of Datadog's Kubernetes-based PostgreSQL clusters stuck: replicas lagged past Patroni's maximum_lag_on_failover, so no standby could be promoted safely. Datadog moved to a hybrid model: standbys in the leader pool take part in synchronous replication (one synchronous standby, synchronous_mode_strict, synchronous_commit=remote_apply) while read replicas stay asynchronous. pgbench showed remote_apply raised average latency by 53% and cut throughput by 34%, yet production clusters saw no measurable impact after a gradual rollout, and synchronous replication is now enabled in all of its clusters.
FAQ
What is Datadog Husky?
Husky is Datadog's third-generation event store, introduced in 2022: a distributed, schemaless column store that keeps event data on object storage, with storage separated from compute.
How does Husky ingest data exactly once?
Datadog's 2023 deep dive explains how Husky routes, shards and ingests events exactly once across tenants, using Go, Kafka and FoundationDB.
How did Datadog scale timeseries indexing?
Datadog's 2024 post describes redesigning its timeseries indexing, with Rust and RocksDB, to handle a large jump in ingested data.