Tag: AdamW
-
AdamW Explained, Line by Line
AdamW dissected: where it runs, the lineage behind the name, the four update lines, why decoupled weight decay was a real fix,…
-
Gradients and Optimizers: From SGD to Adam, and Why m and v Cost 8 Bytes
What a gradient is, what an optimizer decides, and why Adam’s two running statistics per weight are worth 8 bytes each on…
-
Training Memory: The Four Tenants and the 16 Bytes Per Parameter
Why a 3 GB model needs 25 GB to train. The four memory tenants derived from first principles, where the 16 bytes…