Category: LLM Fine-Tuning
How LLM fine-tuning actually works: training memory, numerics, optimizers, masking, SFT, data, LoRA, QLoRA and preference tuning.
-
Training Memory: The Four Tenants and the 16 Bytes Per Parameter
Why a 3 GB model needs 25 GB to train. The four memory tenants derived from first principles, where the 16 bytes…
-
LLM Fine-Tuning Explained: What Actually Changes Inside the Model
LLM fine-tuning runs the pretraining objective on your data with a loss mask. What actually changes, when to do it, and how…