The False Promise of Imitating Proprietary LLMs
The paper argues that imitating proprietary large language models (LLMs) like GPT-4 through distillation or fine-tuning on their outputs is a flawed strategy. It shows that such imitation models inherit limitations without achieving the same performance, and that open-source development should instead focus on fundamental research and training from scratch using public data and methods.
Background
This paper criticizes the common practice of "model distillation" or "imitation," where open-source AI projects try to copy the behavior of expensive, proprietary large language models (like GPT-4) by training smaller models on text outputs from the big ones.