UATC – A Closed-Loop Controller to Prevent GPU OOM During LLM Training
UATC is a closed-loop controller designed to prevent GPU out-of-memory (OOM) errors during large language model (LLM) training by dynamically managing memory usage.
Background
- UATC (Out-of-Memory-Avoiding Training Controller) is a lightweight tool that runs in the background alongside large language model (LLM) training, automatically throttling training speed when GPU memory runs low to prevent out-of-memory (OOM) crashes.
- GPU OOM is a common and expensive failure in LLM training — when memory usage exceeds the GPU's capacity, training abruptly stops, wasting hours or days of compute time. Engineers often resort to conservative memory settings to avoid it.
- The project is by an independent developer and has drawn attention as a practical, low-overhead alternative to more complex solutions (like checkpointing or dynamic batch sizing). It works by monitoring memory pressure and inserting small delays into training loops.
- Key context: LLM training typically uses GPUs with 40–80 GB of VRAM; OOM failures are especially costly on rented cloud instances where users pay per hour.