Core dump epidemiology: fixing an 18-year-old bug

This is a postmortem by OpenAI's infrastructure team about a subtle, long-lived bug in their data pipeline for training large language models. - **OpenAI** is the company behind ChatGPT, GPT-4, and the DALL·E image generator. - A "core dump" is a snapshot of a program's memory at the moment it crashes, used for debugging. - The bug was 18 years old — it originated in the open-source Python library `gdbm` (GNU dbm), a key-value database. It caused silent data corruption: when two "core dumps" were stored under the same key, the older entry was incorrectly deleted instead of the newer one. - Because the bug was silent (no crash, no error message), years of crash data were gradually lost. The team eventually traced it via statistical oddities in their crash reports and fixed the upstream `gdbm` library. - **Why it matters**: Data pipelines for AI systems run at massive scale, and silent corruption bugs can degrade model training, safety analysis, or reliability metrics without anyone noticing. Fixing such bugs often requires fixing the open-source dependencies that the entire industry relies on.