It's Always the Learning Rates
The blog post argues that learning rates are often the root cause of training failures in machine learning, emphasizing that tuning learning rates—rather than more complex architectural changes—frequently resolves issues like divergence or slow convergence. The author shares practical advice on learning rate schedules, warmup, and debugging training runs.
Background
- **Learning rate** is a key hyperparameter in machine learning — it controls how big a step a model takes when adjusting its weights during training. Too high, and training diverges; too low, and it stalls or takes forever.
- The post’s title echoes the folk wisdom *“It’s always the learning rates”* — meaning that many seemingly mysterious training failures can be traced back to this one setting, no matter how sophisticated the rest of the system is.
- Ian Barber is a seasoned engineer who previously led engineering at the UK Financial Conduct Authority and has written extensively on practical ML and systems engineering.
- This piece fits into a broader industry debate: as AI models grow more complex, foundational engineering details (data quality, hyperparameters) often matter more than architectural novelty — a counterpoint to hype-driven narratives.