The article presents updated results from instruction fine-tuning experiments on a 32-layer language model built from scratch. It discusses interventions and performance improvements achieved through the fine-tuning process.
The author reviews the appendices of "Build a Large Language Model (from Scratch)" and found useful material on PyTorch basics, DistributedDataParallel training, and LoRA implementation. While these sections could have saved time during their explorations, they believe working through concepts independently provided deeper learning than simply reading explanations.
The author completed training a GPT-2-like model in 44 hours on a local machine, achieving performance close to GPT-2 small. Through systematic testing of various interventions, they identified learning rate adjustments and dropout removal as most effective for improving model loss. The author plans to next implement an LLM from scratch using JAX without reference to their book.
Updated instruction fine-tuning tests on GPT-2-style models show OpenAI's models performed best. Some custom models with similar test loss scores showed unexpected variations in instruction-following ability, with no clear pattern emerging across all tested models.