Netflix

Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale

Netflix found a typical regression investigation took about four hours, mostly spent reconstructing relationships between systems like Atlas, Radar, Spinnaker, PagerDuty, incident.io and Slack that each named the same thing differently. It built an Observability Ontology on the W3C stack (RDF, OWL, SHACL, SPARQL), stored as a knowledge graph in QuipuDB, its homegrown distributed quad store built on the Rust key-value store redb. Enrichers, some LLM-based, turn noisy inputs into typed triples, every application is validated against a catalog of about 49,000 entries, and dependency edges are capped. A correlation engine scores chains across 13 signal types within a ten-minute window, discarding anything under 0.6 confidence, and an agentic AutoRCA system reads the graph to hand engineers ranked, evidence-backed hypotheses.

Published
2026-10-16
Scale
~65M concurrent streams (Paul vs. Tyson, Nov 2024), 2B+ requests/day, 38M+ real-time logging events/sec, ~49,000-entry software catalog, 13 signal types in the correlation engine
Techniques
knowledge graphs observability root cause analysis AI agents
Stack
RDF OWL SHACL SPARQL QuipuDB redb Rust LLMs

See also

Netflix architectureGo vs Rust for backend services at scale

The best scaling write-ups, weekly

One email a week. No spam.

Open subscribe page

Related stories

OpenAI · 2026

Scaling Git at OpenAI

OpenAI built Pando, a Git-compatible backend for its monorepo after Codex pushed concurrent branches, merges and fetches past the old system (reliability sometimes below 90%). Pando stores each push as an immutable pack in blob storage and records per-branch ref versions in a horizontally scalable metadata store, giving serializability per ref and causal consistency across the repo so unrelated branches do not contend. Instead of building unique packs per client, it reuses stored packs and serves them via packfile-uri through regional Express caches (over 99% hit rate), with LSM-style background compaction so fetches plan in logarithmic time. Deployed first as a read/write-through proxy, then cut over with a Git CLI wrapper; it now powers all monorepo development, cut median push latency by nearly 5x, and can deliver nearly a petabyte of Git data per day (57+ TB in a single hour).

nearly a petabyte of Git data/daymedian push latency cut nearly 5x (~80%)per-ref causal consistencyreusable packfilesstorage/compute separationGitPandoRust
Meta · 2026

Catching Bugs Before They Land: JiTTesting's Human-on-the-Loop Approach at Full Scale

Meta's Catching Just-in-Time Tests (JiTTests) are now production infrastructure. Overnight, on spare capacity, the system picks risky diffs from the previous day and generates tests designed to fail: a "dodgy diff" workflow treats each change as a mutant, while an intent-aware workflow infers what the diff is meant to do, lists the risks, and builds mutants from them. An ensemble of LLM judges plus a deterministic rule-based assessor (RubFake) ranks the failing "weak catches", and engineers are only contacted for the strongest ones. 589 engineers have engaged, reporting 60% true positives, and in the first half of 2026 it helped catch 74 serious privacy bugs before landing, cutting privacy SEVs by 20%.

hundreds of millions of lines of code589 engineers engaged (64% of outreach)LLM test generationmutation testingfalse-positive filteringLLMsLLM-as-Judge ensemblemutation testing
OpenAI · 2026

Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

OpenAI's Habitat storage platform went from one shared Azure Cosmos DB container, to dedicated single-tenant containers that pushed capacity planning onto every product team, and back to a shared container by default once Azure added hot-key throttling, request-priority QoS and a new serverless mode where each physical partition scales to 10K RU/s and splits on RU usage. After failover and failback outages in late 2024, OpenAI and Azure now co-own testing and monitoring. Replicating small latency-critical datasets widely, a Valkey lookaside cache invalidated through the Cosmos DB change feed, and a Rust rewrite of the proxy cut global read p50 from about 120 ms to about 10 ms.

1B+ users across 40 regions500+ PB served by Habitatmulti-tenancyhot key throttlingCDC cache invalidationAzure Cosmos DBValkeyKafka
OpenAI · 2026

Rapidly scaling online storage to serve over 1 billion ChatGPT users

OpenAI's Habitat storage platform grew from a Python client library over Cosmos DB into a central service handling 70M+ requests per second. The team ran many low-concurrency Python processes to keep asyncio delay down, switched aiohttp connection reuse from LIFO to FIFO to stop a metastable failure, used Envoy for connection fan-in, kept a deliberately simple NoSQL API, then rewrote the service in Rust.

70 million+ requests/sec1 billion+ weekly usersconnection poolingload balancingrate limitingPythonRustCosmos DB