Tag: residual stream
-
The Output Projection: How 64 Attention Heads Become One Thought
Work a full output projection by hand on a 6 x 6 grid, then see why W_O routes head findings and why…
-
Inside One Transformer Block: The Residual Stream and Its Seven Matrices
A transformer block takes 8,192 numbers in and returns 8,192 out. Follow the residual stream, the seven matrices, and what each of…