Post

How we continuously validate formal specifications in production

Datadog’s FETCh generates Rust runtime monitors from a subset of TLA+ specifications, checking production events against allowed state transitions.

For its routing system, the team split validation into per-process routing checks and global placement checks, using RocksDB-backed state where needed. A monitor detected conflicting placements missed by testing. Reported p90 overhead for routing sidecars was about 2% CPU and under 0.2% memory. This checks observed behavior, not all possible executions, and instrumentation and global state remain challenges.