Datadog

Evolving our real-time timeseries storage again: Built in Rust for performance at scale

Datadog rebuilt its real-time timeseries database (RTDB) storage engine in Rust as Monocle after outgrowing Cassandra, Redis, MDBM, and a Go B+ tree plus a separate RocksDB path for DDSketch distributions. The design shards the keyspace early with a shard-per-core model (for example, 24 storage shards on a 32-core machine), ingesting from Kafka partitions over MPSC channels into per-shard LSM trees on single-threaded Tokio schedulers, with tiered compaction, a unified series cache, and permit- and cost-based query throttlers (including CoDel). At peak scale the rewrite delivered a 60x increase in ingestion performance and 5x faster queries, turning extreme outliers that used to take minutes into results that return in seconds.

Published
2025-08-04
Scale
60x increase in ingestion performance, 5x faster queries at peak scale, extreme query outliers from minutes to seconds
Techniques
shard-per-core LSM tree tiered compaction series cache query throttling
Stack
Rust Tokio LSM tree Kafka DDSketch Monocle

See also

Datadog architectureGo vs Rust for backend services at scale

The best scaling write-ups, weekly

One email a week. No spam.

Open subscribe page

Related stories

OpenAI · 2026

Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

OpenAI's Habitat storage platform went from one shared Azure Cosmos DB container, to dedicated single-tenant containers that pushed capacity planning onto every product team, and back to a shared container by default once Azure added hot-key throttling, request-priority QoS and a new serverless mode where each physical partition scales to 10K RU/s and splits on RU usage. After failover and failback outages in late 2024, OpenAI and Azure now co-own testing and monitoring. Replicating small latency-critical datasets widely, a Valkey lookaside cache invalidated through the Cosmos DB change feed, and a Rust rewrite of the proxy cut global read p50 from about 120 ms to about 10 ms.

1B+ users across 40 regions500+ PB served by Habitatmulti-tenancyhot key throttlingCDC cache invalidationAzure Cosmos DBValkeyKafka
Datadog · 2025

Achieving relentless Kafka reliability at scale with the Streaming Platform

Datadog treats Kafka as commodity hardware behind a Streaming Platform control layer. Streams span multiple Kafka clusters, AZs, and Kubernetes clusters so producers and consumers talk to stable identifiers while the control plane redirects traffic to healthier topics in seconds—for incidents, rebalancing, decommissioning, or repartitioning—without app redeploys. A custom Assigner replaces Kafka's group coordinator for multi-cluster rebalancing in seconds; Stream lanes isolate QoS and park poison pills in a DLQ; an enriched commit log tracks multiple offset ranges plus timestamps so consumers can process live traffic while replaying backlogs and the control plane can compute wall-clock lag across millions of partitions. After zstd "bombs" crashed stock clients, Datadog wrote libstreaming in Rust (with Java/Go/Python bindings) for unified, high-throughput access without a proxy in the TB/s data path.

hundreds of trillions of observability events/dayhundreds of Kafka clusters, thousands of topics, millions of partitionsmulti-cluster failovercustom coordinatorcommit log enrichmentApache KafkaKubernetesRust
Datadog · 2024

Timeseries indexing at scale

Datadog redesigned its timeseries indexing to handle a large jump in ingested data.

indexingrewriteRustRocksDB
Datadog · 2023

Husky: Exactly-once ingestion and multi-tenancy at scale

A deep dive on how Husky routes, shards and ingests events exactly once across tenants.

exactly-once ingestionshardingmulti-tenancyGoKafkaFoundationDB