Content-defined chunking in Go: 3.7 GB/s and a third less memory
Plakar released v1.1.0 of its Go content-defined chunking library, achieving up to 3.7 GB/s throughput while using a third less memory. The release also includes formal proofs of correctness for the chunking algorithms.
Background
Content-defined chunking (CDC) splits data at boundaries determined by the content itself, not at fixed byte offsets. This is critical for backup and deduplication: if you insert a byte at the start of a file, fixed-size chunking shifts every subsequent boundary (so everything appears "changed"), but CDC re-syncs naturally, detecting only the truly new parts.
- Plakar is an open-source backup tool written in Go; its author wrote the `go-cdc-chunkers` library used by several Go backup projects.
- This post announces v1.1.0, claiming throughput of 3.7 GB/s on modern hardware with ~33% less memory than v1.0.
- "Provably correct" means the author used model checking (the `porcupine` library) to mathematically verify the chunker behaves identically to a reference on all possible inputs — important because CDC logic errors can silently corrupt backups.
- The bottom line: faster, leaner chunking means cheaper and quicker backups for any tool that depends on this library.