LLM from scratch, part 33 – what I learned from the appendices

Related stories

• Writing an LLM from scratch, part 33 -- what I learned from finally getting round to the appendices

  2.0

  The author reviews the appendices of "Build a Large Language Model (from Scratch)" and found useful material on PyTorch basics, DistributedDataParallel training, and LoRA implementation. While these sections could have saved time during their explorations, they believe working through concepts independently provided deeper learning than simply reading explanations.

• Writing an LLM from scratch, part 32m -- Interventions: conclusion

  2.0

  The author completed training a GPT-2-like model in 44 hours on a local machine, achieving performance close to GPT-2 small. Through systematic testing of various interventions, they identified learning rate adjustments and dropout removal as most effective for improving model loss. The author plans to next implement an LLM from scratch using JAX without reference to their book.

• Writing an LLM from scratch, part 32l -- Interventions: updated instruction fine-tuning results

  1.5

  Updated instruction fine-tuning tests on GPT-2-style models show OpenAI's models performed best. Some custom models with similar test loss scores showed unexpected variations in instruction-following ability, with no clear pattern emerging across all tested models.