Core dump epidemiology: fixing an 18-year-old bug
OpenAI engineers tracked down and fixed an 18-year-old bug in the SQLite database library that caused rare crashes in data infrastructure. The bug, discovered through analysis of core dumps, involved a subtle memory corruption issue in SQLite's query processing logic. OpenAI contributed the fix to the open-source SQLite project.
Background
- OpenAI, best known for ChatGPT and GPT-4, also runs massive AI-training infrastructure. This post is from their reliability engineering team.
- A "core dump" is a snapshot of a program's memory at the moment it crashes — the computing equivalent of an autopsy report.
- The post describes two bugs in widely used Linux tools (systemd-coredump and the kernel's /proc/pid/mem). One was 18 years old, the other 10 years old. Neither was OpenAI-specific.
- The older bug silently threw away core dumps when a crash happened during a memory-compaction routine. The newer one made the dump collector hang permanently under the same conditions, poisoning future dumps.
- Why it matters: AI training runs on thousands of machines for weeks. One crash can cascade; without a core dump, engineers have no forensic data. The post shows how critical system bugs can hide for decades until modern hardware (huge memory, many CPU cores) makes them common.