Updated instruction fine-tuning tests on GPT-2-style models show OpenAI's models performed best. Some custom models with similar test loss scores showed unexpected variations in instruction-following ability, with no clear pattern emerging across all tested models.
The article presents updated results from instruction fine-tuning experiments on a 32-layer language model built from scratch. It discusses interventions and performance improvements achieved through the fine-tuning process.
The author reflects on insights gained from working through appendices in their LLM from scratch series, noting that these supplementary materials provided valuable practical knowledge and deeper understanding of implementation details beyond the main content.