Catching Bugs Before They Land: JiTTesting's Human-on-the-Loop Approach at Full Scale

Meta's Catching Just-in-Time Tests (JiTTests) are now production infrastructure. Overnight, on spare capacity, the system picks risky diffs from the previous day and generates tests designed to fail: a "dodgy diff" workflow treats each change as a mutant, while an intent-aware workflow infers what the diff is meant to do, lists the risks, and builds mutants from them. An ensemble of LLM judges plus a deterministic rule-based assessor (RubFake) ranks the failing "weak catches", and engineers are only contacted for the strongest ones. 589 engineers have engaged, reporting 60% true positives, and in the first half of 2026 it helped catch 74 serious privacy bugs before landing, cutting privacy SEVs by 20%.

Published

2026-10-29

Scale

hundreds of millions of lines of code, 589 engineers engaged (64% of outreach), 60% true-positive rate, 74 SEV privacy bugs caught at diff stage in H1 2026, privacy SEVs down 20%

Techniques

LLM test generation mutation testing false-positive filtering human-on-the-loop

Stack

LLMs LLM-as-Judge ensemble mutation testing RubFake

Read the original on Meta's blog →