Combining LLMs Rarely Beats the Best Single Model, I tested 67 frontier models
A study testing 67 frontier large language models found that combining multiple LLMs (ensembling or mixing outputs) rarely outperforms the single best model in the group, challenging the assumption that model collaboration consistently improves performance.
Background
- The paper tests whether combining outputs from multiple large language models (ensembling) reliably beats using just the best single model — and finds it usually does not.
- 67 frontier models were tested (likely including GPT-4, Claude, Gemini, Llama, etc.), and no ensemble consistently outperformed the strongest individual model in the set.
- This challenges a common assumption in ML that ensembling multiple models almost always improves performance. For LLMs, the gains appear marginal or nonexistent.
- The result matters for practitioners deciding whether to route queries across many LLMs (costly, complex) or just use one top model. It also raises questions about diminishing returns in the current LLM landscape.