How Datadog measure data completeness at scale
Datadog developed a system to measure data completeness across its high-volume data pipeline at scale. The approach uses a distributed counting mechanism with idempotent keys to track expected vs. actual data volumes, enabling reliable SLAs and rapid detection of missing or delayed data.
Background
- Datadog is a major cloud monitoring platform that ingests huge volumes of metrics, logs, and traces from customers' systems. Its core business depends on reliable data collection.
- "Data completeness" is the problem: when collecting billions of data points across distributed systems, some inevitably go missing due to network issues, server crashes, or processing delays. Datadog needed to detect silent data loss without inspecting every raw event.
- The article describes a system that measures completeness as a percentage — comparing data actually received against a statistical estimate of what should have arrived. This is hard because no single source has a full view of "expected" data.
- Their approach uses heartbeat markers and a streaming aggregation layer to count expected vs. actual volumes across shards and time windows; this surfaces completeness scores so internal teams can spot silent data loss.