Scaling Laws, Carefully
Reviews scaling laws in deep learning, emphasizing careful empirical measurement to predict model performance. Discusses how loss scales with compute, data, and model size, but warns violations occur when key assumptions are not met. Provides practical advice for conducting and interpreting scaling experiments.
Background
- **Lilian Weng** is a well-known applied AI researcher at OpenAI; her blog posts are widely read as authoritative, accessible summaries of important ML topics.
- **Scaling laws** are empirical observations that model performance (e.g., accuracy, loss) improves predictably as you increase model size (parameters), training data volume (tokens), and compute budget. They have been the core "recipe" behind large language models (LLMs) like GPT-4.
- This post reviews recent research showing that simple scaling laws are breaking down or need significant revision—larger models now sometimes underperform smaller ones trained on better data, and factors like data quality, training schedule, and inference-time compute (chain-of-thought, test-time scaling) are becoming as important as raw size.
- Why it matters: Practitioners building AI systems rely on scaling laws to decide how to allocate limited compute budgets. If the old rules no longer hold, it changes how companies and researchers will build future models (e.g., prioritizing data curation or inference strategies over simply training bigger models).