Scaling Laws, Carefully
A careful review of neural network scaling laws covering empirical findings, token-counting nuances, compute-optimal training (Chinchilla law), and loss measurement methodology, with recommendations for avoiding common pitfalls.
Background
- Scaling laws are empirical observations in AI research showing that model performance improves predictably as you increase three resources: model size (parameters), data size (tokens), and compute (flops). These laws guide how companies like OpenAI and DeepMind allocate resources when training large language models (LLMs).
- Lilian Weng is a prominent AI researcher and former Head of Safety at OpenAI, known for clear technical explainers. Her posts are widely read in the ML community.
- Recent debate has questioned whether scaling "is running out" — some researchers argue that continued gains require novel architectures (e.g., "reasoning" models like o1) rather than just bigger versions of the same approach. This post examines those claims carefully.