Smooth-Maximum, the most useful function [video]
This video explains the smooth-maximum function, a differentiable approximation of the maximum function used in optimization and machine learning. It covers how smooth-max works, its mathematical properties, and practical applications where it is useful for gradient-based methods.
Background
The soft maximum (also called LogSumExp or smooth maximum) is a differentiable approximation of the "max" function — it smoothly blends the largest values rather than picking just one. It's widely used in machine learning (e.g., the attention mechanism in transformers, softmax classifiers), control theory, and convex optimization because it preserves differentiability while behaving like max. The video likely explains its mathematical form (log of sum of exponentials), why it beats hard max in gradient-based training, and its connection to other functions like softmin or log-sum-exp.