Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
A new study shows that a single transformer layer can match the performance of full-parameter reinforcement learning training, questioning the necessity of deep architectures for certain RL tasks.
Background
- This paper challenges a core assumption in reinforcement learning from human feedback (RLHF), the technique used to align large language models like ChatGPT with human preferences. The standard view holds that RLHF requires **full-parameter fine-tuning**—updating billions of weights across all layers of a neural network.
- The authors show that fine-tuning **just a single transformer layer** (the last one) can match the performance of full-parameter RL training on key alignment benchmarks (e.g., MT-Bench, AlpacaEval, TL;DR summarization).
- This is significant because training LLMs is extremely expensive in compute and memory. If RL alignment needs only one layer, it could dramatically reduce the cost and hardware requirements for post-training, making alignment research more accessible.
- The finding also raises scientific questions: if alignment "fits" in one layer, what does that imply about how LLMs store and apply human preferences? The paper is a preprint (July 2025) and has not yet been peer-reviewed.