A transformer block reads 8,192 numbers per token and writes 8,192 numbers per token. Block 1 does it, block 80 does it, and the width never varies anywhere in between. That constraint is forced by the architecture rather than chosen by a designer, and once you see what forces it, the block stops being a list of parts to memorise and becomes a shape you can draw from memory.
This part traces one block of the series running example, a Llama-3-70B class model: 80 layers, d_model 8,192, 64 query heads, 8 key/value heads, head_dim 128, feed-forward width 28,672. Seven weight matrices, two normalisations, two additions. Nothing else.
The most useful thing in here is a correction. Most engineers carry a mental picture in which x flows through every matrix in turn, like water through a series of filters. That picture is wrong in three specific and consequential ways. Fixing it explains the KV cache, the output projection, and why the feed-forward holds most of the model’s weights. By the end you should be able to draw the block, say what each matrix receives, and compute its parameter count by hand.
- Part 1. How an LLM Answers a Question: The Complete Inference Path
- Part 2. Byte-Pair Encoding Explained: How LLMs Turn Text Into Tokens
- Part 3. Inside One Transformer Block: The Residual Stream and Its Seven Matrices (you are here)
- Part 4. Multi-Head Attention Explained: Why 64 Heads Instead of One
- Part 5. The Output Projection: How 64 Attention Heads Become One Thoughtcoming 13 Aug
- Part 6. The Feed-Forward Network: Where a Transformer Keeps What It Knowscoming 14 Aug
- Part 7. Mixture of Experts Explained: Conditional Computation From Zerocoming 15 Aug
- Part 8. Attention Is All You Need, Dissected: The 2017 Figure, Box by Boxcoming 16 Aug
- Part 9. The Roofline Model: Why LLM Decode Is Memory-Boundcoming 17 Aug
- Part 10. Continuous Batching and PagedAttention: How vLLM Keeps a GPU Busycoming 18 Aug
- Part 11. Benchmark Your Own LLM Serving Stack: Two Measurements, One Afternooncoming 19 Aug
The invariant, before any detail
Definition
Transformer block
One repeated unit of the model, holding seven weight matrices, two normalisations and two additions. A Llama-3-70B class model stacks 80 identical copies of it, differing only in the values of their weights.
Origin: set out in the 2017 transformer paper, which deleted the recurrent layer and kept attention plus a per-token feed-forward network. Recurrence forced a sequence to be walked one position at a time, so training could not be parallelised along the sequence and long-range dependencies had to survive many sequential steps. Removing that limit is what the block was for.
Why it matters: a block is the unit of everything you will later budget. It is 855,638,016 parameters, 1.71 GB in bf16, and one slice of KV cache per token. Multiply by 80 and you have the model.
One vector of 8,192 numbers per token enters a block. One vector of 8,192 numbers per token leaves it. All 80 blocks share that shape, which is what lets you reason about one and then multiply.
The invariant is forced rather than chosen. The block adds its result to its input, and addition requires matching shapes. If the block returned 4,096 numbers there would be nothing legal to add them to. So everything that happens inside has to come back to 8,192 before the block ends. The attention branch fans out to 64 heads and comes back. The feed-forward branch widens to 28,672 and comes back. Neither one is permitted to change the width of the thing it hands over.
That single constraint is why blocks stack arbitrarily deep, and why a 70B model and a 405B model look identical at this level of the diagram. Only the block count and the stream width differ, so the drawing you are about to learn is the drawing for every dense transformer in production.
Section takeaways
- A block takes 8,192 numbers per token and returns 8,192 numbers per token, in every one of the 80 layers.
- The width is forced by the residual addition, since adding two vectors requires them to be the same shape.
- Both branches leave the stream’s width alone: attention fans out to 64 heads and returns, the feed-forward widens to 28,672 and returns.
- Model families differ in block count and stream width, so one block diagram covers every dense transformer you will serve.
Draw the spine first: the residual stream
Definition
Residual stream
The unbroken vertical path carrying one 8,192-number vector per token from the embedding table at the bottom of the model to the LM head at the top, straight through all 80 blocks. Each block reads the current state, computes a contribution, and adds that contribution back.
Origin: the skip connection around each sublayer comes from the 2015 residual networks work in computer vision, which was answering the finding that adding layers to a deep network made it worse rather than better, because the gradient had no short path back to the early layers. The transformer adopted it in 2017, and the name residual stream comes from later interpretability work that reads the path as a shared channel every block writes into and reads from.
Why it matters: nothing on the stream is ever overwritten. Information written by block 3 is still present and readable at block 70, which is why features can be built up gradually across depth.
In practice
When you draw a transformer block, draw the vertical line first and hang everything off it. Drawing the matrices first produces the chain picture, which is the wrong topology and the source of most of the confusion in the second half of this article.
Treating the stream as a workspace rather than a pipeline changes what you expect to find in it. A pipeline transforms its payload at each stage, so only the latest version exists. The residual stream accumulates instead, so the vector arriving at block 70 is a sum of the embedding plus 138 separate contributions written by the blocks below it.
Analogy
The stream is a whiteboard that 80 people write on in turn. Each person reads what is already there, decides on a contribution, and adds it. Nobody is allowed to erase, so the last person still sees traces of the first.
Where it breaks: the whiteboard has exactly 8,192 slots and every contribution is added numerically into those same slots. Two blocks writing opposite values in one slot cancel each other out, which a real whiteboard would never do.
Why the addition matters more than it looks
The reason the architecture is built this way is a derivative. The derivative of x + f(x) with respect to x is 1 + f'(x), and that leading 1 is an unobstructed path for gradients to travel back to block 1 no matter what the branch does.
Without it, the gradient reaching block 1 would be a product of 80 Jacobians. Any systematic tendency for those factors to sit below 1 shrinks the signal towards zero, and any tendency above 1 blows it up. With the residual path present the product always contains a term that is exactly 1, so early layers keep receiving usable gradient however deep the stack gets. Eighty blocks would not train without it. This is a training fact showing up in an inference diagram, which is common: most of what looks arbitrary about a transformer block is a training constraint that survived into the deployed weights.
Section takeaways
- The residual stream is a shared workspace that is added to and never overwritten, so early contributions stay readable at the top of the stack.
- Draw the vertical line first. Every branch leaves it and rejoins it, and nothing flows matrix to matrix along it.
- The derivative of
x + f(x)is1 + f'(x), and that constant 1 is the gradient path that makes an 80-block stack trainable. - Without the additive path the gradient to block 1 is a product of 80 Jacobians, which vanishes or explodes.
Branch one: attention
RMSNorm opens the branch, and it works on a copy
Definition
RMSNorm, and pre-norm placement
RMSNorm divides all 8,192 numbers by their root mean square, then multiplies each one by a learned gain. Pre-norm means it sits on the branch rather than on the spine, so the stream itself passes by untouched.
Origin: the 2017 transformer used LayerNorm, applied after the addition. RMSNorm was published in 2019 as a cheaper replacement that drops the mean-subtraction step and keeps only the rescaling, on the finding that the recentring was not what made normalisation work. Pre-norm placement was a separate change: post-norm stacks were unstable to train at depth and leaned on a careful warmup schedule, and moving the norm onto the branch made deep models train reliably.
Why it matters: the 8,192 gains are 16 KB, a rounding error against a 1.71 GB block, and the model stops working without them. Their placement on the branch is what keeps the addition path a pure sum from embedding to LM head.
Everything downstream in this branch is therefore working on a normalised copy, while the original x continues down the spine unchanged and waits to be added to. Two vectors exist at this point in the block, and keeping them apart is what makes the rest of the dataflow readable.
W_Q, W_K and W_V read the same input at the same time
The normalised vector is fed to three separate matrices simultaneously. This is a fan-out, so W_K does not receive q. All three read the identical input and produce three independent outputs, which is also why a serving engine can fuse them into a single matrix multiply.
Analogy
Three analysts receive the same memo at the same moment and each writes a different report from it. What they do not do is pass one report down a hallway for the next analyst to edit.
Where it breaks: the three are not independent readers with opinions. Each is a fixed linear projection of the same vector into a different space, and the spaces are different sizes.
The outputs are not the same size. q comes out 8,192 wide, which is 64 heads of 128. k and v come out 1,024 wide, which is 8 heads of 128.
Definition
Grouped-query attention (GQA)
An attention layout in which many query heads share a smaller number of key/value heads. Here 64 query heads are divided into 8 groups of eight, and each group shares one key/value head.
Origin: multi-query attention came first, in 2019, sharing one key/value head across every query head once it was clear that the KV cache is what makes decoding memory-bound. It cost measurable quality, so grouped-query attention followed in 2023 as the setting in between, keeping most of the memory saving and recovering the quality, and it came with a recipe for converting existing multi-head checkpoints rather than training from scratch.
Why it matters: it makes W_K and W_V eight times smaller, and it makes the KV cache eight times smaller, which is the difference between a serving stack that fits in memory and one that does not. Quality cost is close to zero.
Rotate, then deposit into the cache
Definition
RoPE (rotary position embedding)
A rotation applied to q and k by an angle proportional to the token’s position in the sequence. It touches neither v nor the residual stream, and it has no parameters at all.
Origin: introduced in 2021 in the RoFormer paper. The 2017 transformer added a position signal into the input vector itself, which mixes position into content and is pinned to the sequence lengths seen in training, so nothing sensible happens past them. Rotating q and k instead makes every attention score depend on the distance between two positions rather than on where each one sits, and it costs no weights.
Why it matters: position decides which tokens match each other, and it deliberately does not decide what those tokens contain. That separation is why v is left alone.
Then k and v are copied into this layer’s slice of the KV cache. The direction of that copy is the thing people get wrong.
Definition
KV cache, per layer
The stored k and v values for every token processed so far, held separately for each of the 80 layers. One token costs 2 (for k and v) times 1,024 times 2 bytes, so 4 KB per layer and 320 KB across the model.
Origin: caching became standard practice as soon as decoder-only models were served autoregressively. Without it, generating token n would mean recomputing k and v for every earlier token, in all 80 layers, on every single step, so the cost of a reply would climb with the square of its length instead of linearly.
Why it matters: the cache flows forward in time within one layer, while the residual stream flows upward through layers within one step. Layer 6 never reads layer 5’s cache. Conflating those two axes is the most common error in reasoning about serving memory.
That distinction is worth holding on to if you ever plan to reason about prompt caching and what it actually saves you, since what a cache hit reuses is exactly these per-layer entries.
Attention runs, then W_O merges the heads
Sixty-four heads each attend independently, each producing 128 numbers. Concatenated side by side that is 8,192 again, which is exactly why head_dim was set to 128 in the first place. The arithmetic 64 times 128 equals 8,192 is a constraint rather than a coincidence, and it is what picked the number. Why 64 heads instead of one is the subject of the next part.
Definition
W_O, the output projection
The 8,192 by 8,192 matrix that multiplies the concatenated head outputs. It consumes attention’s output, and never q, k, v or x.
Origin: it has been part of multi-head attention since the 2017 transformer paper. Splitting attention into heads leaves the result as a row of separate slices, each only one head wide, so something has to turn them back into a single vector of model width. Without that step each head would write into its own fixed region of the stream for the model’s whole depth and could never reach any other head.
Why it matters: until this exact moment the 64 heads sat in separate 128-number slices and had never interacted. W_O is the only place in the attention branch where head 12’s finding can reach head 40. Without it the heads stay in their own lanes for the model’s full depth.
The output projection gets a part of its owncoming 13 Aug later in the series, because what it mixes turns out to matter as much as that it mixes.
The first addition, and the death of q, k and v
W_O‘s output is added to the vector that entered the block. Added, and never substituted. The branch computed an adjustment to the stream, and the stream absorbed it.
At that instant q, k and v cease to exist. They were scratch values, created a few microseconds earlier by the three projections and consumed inside attention. The only trace they leave behind is the copy of k and v in the cache, and that copy exists for future decode steps of this same layer.
Section takeaways
- RMSNorm sits on the branch, so attention works on a normalised copy while the original
xwaits on the spine. W_Q,W_KandW_Vare siblings reading one input, which is what lets a serving engine fuse them into one matrix multiply.- Grouped-query attention gives 64 query heads only 8 key/value heads, shrinking both those matrices and the cache eightfold.
- RoPE rotates
qandkonly, has zero parameters, and decides which tokens match rather than what they contain. - The KV cache moves forward in time inside one layer. It is never a hand-off to the layer above.
W_Omixes the 64 head outputs and is the only point in the branch where heads interact.
Branch two: the feed-forward
Definition
SwiGLU feed-forward
Three matrices rather than the classic two. W_gate and W_up both read the same normalised input and both widen it from 8,192 to 28,672. The gate output passes through SiLU, the two are multiplied elementwise, and W_down compresses the product back to 8,192.
Origin: gated feed-forward networks were measured against the classic two-matrix version in a 2020 paper, which found the gated variants consistently better on an equal parameter budget and said plainly that it had no theory for why. Adoption followed the measurements rather than an argument. The hidden width drops from four times the model width to roughly 3.5 times so that adding a third matrix leaves the parameter count where it was.
Why it matters: these three matrices are 704,643,072 parameters, which is 82.4 percent of the block. Any question about where a model’s knowledge lives is mostly a question about them.
A second RMSNorm opens this branch, working on the new stream value produced by the first addition. Every token passes through the branch alone, and there is no interaction between positions anywhere inside it, which is why it parallelises cleanly and why it is a pure matrix-multiply problem at serving time.
Analogy
Think of a mixing desk with 28,672 channels. W_up produces the signal on every channel and W_gate sets a fader on each one, so the multiply decides, dimension by dimension, how much of the up branch survives.
Where it breaks: no engineer sets the faders. The gate is computed from the same input as the signal, so the routing changes with every token that arrives.
Added back, and the block is finished. What leaves is 8,192 numbers per token, the same shape that arrived, updated twice. The normalised copies, q, k, v, the head outputs and the wide intermediate are all gone. What the feed-forward actually storescoming 14 Aug is a big enough question to deserve its own part.
The whole block at once
One spine, two branches, seven matrices, two additions. Reproducing that from memory, starting with the vertical line, means you have the dataflow right and everything remaining is naming.
Section takeaways
- The feed-forward is three matrices, not two.
W_gateandW_uprun in parallel and onlyW_downis downstream of them. - The gate multiplies the up signal elementwise, so routing is decided per dimension and per token.
- Nothing in this branch mixes positions, which makes it embarrassingly parallel and purely compute-bound at prefill.
- The 28,672-wide intermediate is transient. It is allocated and freed inside the block and never outlives it.
- The finished block hands upward exactly what it received: 8,192 numbers per token, updated twice.
Who eats what
Knowing the order of operations in a transformer block is a different thing from knowing what each matrix receives, and the second one is where the mental model usually breaks. The next few sub-sections are the correction, one wrong arrow at a time.
The version to unlearn
The natural assumption is that x is the raw material and each matrix works on it in turn: x through W_O, then through W_up, then W_gate, then W_down. A chain, with x as the thing being processed and each matrix refining it a little further.
Three things are wrong with that picture. W_O never receives x. The feed-forward receives a different vector from the one attention received. And W_gate and W_up run side by side rather than one after the other.
Only three matrices ever touch x
W_Q, W_K and W_V each receive a normalised copy of x, all three reading the same input at the same time. That is the only point in the entire block where anything derived directly from x meets a weight matrix. The other four never see it.
q, k and v go into attention and do not come out
They are consumed. Attention uses q to compare, k to be compared against, and v as the material to blend. What emerges is something new, 64 vectors of 128 numbers each, and q, k and v are finished. They are not passed to W_O, or to any of the three feed-forward matrices. No tuple of (q, k, v) is handed anywhere.
W_O receives the head outputs, never x
The 64 head outputs are laid side by side into one 8,192-number vector, and that concatenation is what W_O multiplies. There is no connection between x and W_O at all. While attention was running, x was sitting on the residual stream, untouched, waiting to be added to.
The addition creates a new vector
W_O‘s output is added to x, producing x'. This is a different vector. It contains everything x contained plus this block’s attention contribution, and both are 8,192 wide, which is the only reason the addition is legal at all. From here on, x is history, and everything in the second half of the block works on x'.
The feed-forward receives x’, and its three matrices are not a chain
A second normalisation runs on x', and the result feeds W_gate and W_up in parallel. Both read the same input. Only W_down comes afterwards, and what it receives is the elementwise product of the other two. It never sees x and it never sees x'. The gate and up pair feeding down is the one and only place in a transformer block where a weight matrix chains directly into another weight matrix.
The corrected picture
One sentence holds it together. x goes into the attention branch and comes back as x'. x' goes into the feed-forward branch and comes back as x''. q, k and v never leave the first branch.
Section takeaways
- Only
W_Q,W_KandW_Vever receive something derived fromx. The other four matrices never see it. q,kandvare consumed inside attention and handed to no matrix. Their only surviving copy is in the cache.W_Oreceives the concatenated head outputs, so there is no arrow at all fromxtoW_O.- The first addition creates a new vector
x', and the whole second half of the block works on that. W_gateandW_upfeedingW_downis the only direct matrix-to-matrix chain in the block.
The seven matrices, with the arithmetic
Every number below follows from four constants: d_model 8,192, 64 query heads, 8 key/value heads, head_dim 128, feed-forward width 28,672. Multiply the two dimensions of each matrix and you have its parameter count, so nothing here needs to be taken on trust.
| Matrix | Shape | Receives | Produces | Parameters |
|---|---|---|---|---|
W_Q |
8,192 x 8,192 | norm(x) |
q, 8,192 wide (64 heads x 128) |
67,108,864 |
W_K |
8,192 x 1,024 | norm(x) |
k, 1,024 wide (8 heads x 128) |
8,388,608 |
W_V |
8,192 x 1,024 | norm(x) |
v, 1,024 wide (8 heads x 128) |
8,388,608 |
W_O |
8,192 x 8,192 | 64 head outputs, concatenated | one mixed 8,192 vector | 67,108,864 |
W_gate |
8,192 x 28,672 | norm(x') |
gate signal, 28,672 wide | 234,881,024 |
W_up |
8,192 x 28,672 | norm(x') |
up signal, 28,672 wide | 234,881,024 |
W_down |
28,672 x 8,192 | SiLU(gate) times up | 8,192 vector | 234,881,024 |
Read the middle column on its own. x appears three times, x' twice, and q, k and v appear nowhere, because no matrix consumes them. That column is the corrected picture in table form.
Now the totals. The attention half is 67,108,864 twice plus 8,388,608 twice, which is 150,994,944. The feed-forward half is 234,881,024 three times, which is 704,643,072. The block is 855,638,016 parameters, and the feed-forward is 82.4 percent of them. The two RMSNorm layers contribute 16,384 parameters between them, which is 0.002 percent of the block and is why nobody counts them.
Grouped-query attention is what makes W_K and W_V eight times smaller than W_Q and W_O. If all 64 heads had their own keys and values, those two matrices would be 8,192 by 8,192 as well, and the attention half would be 268,435,456 parameters instead of 150,994,944. The bigger saving is at runtime. With 8 key/value heads, one token’s cache entry for one layer is 2 times 1,024 times 2 bytes, so 4 KB, or 320 KB across all 80 layers. With 64, the same token costs 32 KB per layer and 2.56 MB across the model, eight times the memory traffic on every single decode step. That trade is the argument running under the roofline model and why decode is memory-boundcoming 17 Aug.
Scale the block up and the familiar headline number falls out. 80 blocks at 855,638,016 parameters is 68,451,041,280. Add the 128,256 by 8,192 embedding table and an untied LM head of the same shape, roughly 1.05 billion parameters each, and you land just above 70 billion. In bf16 the block is 1.71 GB, the 80 blocks are about 137 GB, and the whole model is around 141 GB before you allocate a single byte of KV cache. That is the number that decides whether you need two H100s or eight, and it is the arithmetic behind the decision to self-host at all.
Section takeaways
- One block is 855,638,016 parameters: 150,994,944 in attention and 704,643,072 in the feed-forward.
- The feed-forward is 82.4 percent of the block, so a question about model capacity is mostly a question about three matrices.
- Both RMSNorm layers together are 16,384 parameters, 0.002 percent of the block, and structurally essential anyway.
- Grouped-query attention saves 117,440,512 parameters per block and, more importantly, cuts per-token cache from 32 KB to 4 KB per layer.
- 80 blocks plus embedding and LM head is about 141 GB in bf16, before any KV cache is allocated.
Pre-norm, post-norm, and what survives the block
The 2017 paper put the normalisation after the addition, so the block computed LayerNorm(x + f(x)). That places a normalisation on the spine itself, the clean gradient path is interrupted once per layer, and deep stacks needed learning-rate warmup and careful initialisation to train at all. Modern models moved the norm onto the branch, computing x + f(norm(x)), which leaves the spine a pure sum from the embedding all the way to the LM head. That change is most of the reason an 80-layer stack is now routine. The cost is that activation magnitudes on the stream tend to grow with depth, which is why pre-norm models put one final normalisation just before the LM head to clean up.
Definition
Activation memory
The transient tensors a block allocates and frees while running: both normalised copies, q, k and v as live values, the 64 head outputs, the concatenated vector, and the two 28,672-wide feed-forward intermediates.
Origin: the term comes from training, where every intermediate tensor has to stay alive until the backward pass consumes it, and it became a budget with a name once those tensors started to outweigh the weights. Work published in 2016 on sublinear memory cost made the trade explicit by recomputing activations instead of storing them. Serving has no backward pass, so a tensor is freed the moment it is used, which is why this number is a peak rather than a total.
Why it matters: it is the third category of GPU memory, alongside weights that never change and cache that grows per token. Its peak scales with batch size and sequence length rather than with the model, so it is the term that moves when you raise concurrency.
Two things leave the block alive, on two different axes. Upward, to block n plus 1, goes the 8,192-number residual vector per token, updated twice, and that is the only thing handed up. Forward in time, inside this layer only, go the k and v copies in the KV cache slice, 4 KB per token, held for this layer’s future decode steps. Everything else on the activation list above is freed before the block returns.
In practice
Budget GPU memory in exactly three buckets: weights that never change (141 GB here), cache that grows with every token generated (320 KB per token), and activations that come and go with batch size and sequence length. Most capacity-planning mistakes come from folding two of those buckets into one. The discipline is the same as any other memory-hierarchy problem where the layout decides the throughput.
Section takeaways
- Post-norm normalises after the addition and puts a normalisation on the residual path. Pre-norm keeps that path a pure sum.
- Pre-norm is why very deep stacks train without elaborate warmup, at the cost of stream magnitudes growing with depth.
- Exactly one thing is handed to the next block: the 8,192-number residual vector per token.
- The KV cache entry survives on a different axis, forward in time within this layer, at 4 KB per token per layer.
- Everything else is activation memory, freed inside the block, and its peak scales with batch and sequence rather than with the model.
Key takeaways
- The 8,192 in, 8,192 out invariant is forced by the residual addition. Addition needs matching shapes, so every branch must return to the stream’s width.
- Draw the spine first. The residual stream is a shared workspace that is added to and never overwritten, so what block 3 writes is still readable at block 70.
- The derivative of
x + f(x)is1 + f'(x), and that leading 1 is why an 80-block stack trains at all. - Only
W_Q,W_KandW_Vever receive something derived fromx. They are siblings reading one input, not a chain. q,kandvare consumed inside attention and handed to nothing.W_Oreceives the concatenated head outputs.- The first addition produces a new vector
x'. The feed-forward works onx', andW_gateandW_uprun in parallel with onlyW_downdownstream of them. - The block is 855,638,016 parameters and the three feed-forward matrices are 82.4 percent of them. Grouped-query attention is what shrinks
W_KandW_V, and it shrinks the KV cache eightfold at the same time.
Frequently asked questions
Does each transformer block overwrite the residual stream?
No. Each block computes two contributions and adds them to the stream. Nothing is replaced, which is why a feature written into the stream by an early block is still present and readable by a late block, and why the gradient path back to block 1 stays unobstructed.
Why do W_K and W_V have fewer parameters than W_Q?
Grouped-query attention. The model has 64 query heads but only 8 key/value heads, so W_Q outputs 8,192 numbers while W_K and W_V output 1,024 each. That makes them eight times smaller and, more importantly, makes the KV cache eight times smaller at 4 KB per token per layer instead of 32 KB.
Does W_O receive q, k or v?
No. W_O receives the 64 head outputs concatenated into one 8,192-number vector. q, k and v are consumed inside attention and are never handed to another matrix. The only copies that outlive attention are the k and v entries written into this layer’s KV cache for future decode steps.
Are W_gate and W_up applied one after the other?
No, they run in parallel on the same normalised input, both widening it to 28,672. The gate output passes through SiLU, the two are multiplied elementwise, and only then does W_down compress the product back to 8,192. This is the only place in a transformer block where one weight matrix feeds another directly.
Why is the feed-forward most of a transformer block’s parameters?
Its three matrices are each 8,192 by 28,672, which is 234,881,024 parameters apiece, while the four attention matrices total 150,994,944. That puts the feed-forward at 704,643,072 out of 855,638,016 parameters, or 82.4 percent of the block.
What is the difference between pre-norm and post-norm?
Post-norm, used in the original 2017 transformer, normalises after the addition, so a normalisation sits on the residual path itself. Pre-norm normalises on the branch instead and leaves the residual path a pure sum, which is what makes very deep stacks trainable without elaborate warmup schedules.
Sources and further reading
- Attention Is All You Need, the original transformer, including the post-norm block layout.
- Deep Residual Learning for Image Recognition, where the additive skip connection and its gradient argument were introduced.
- Root Mean Square Layer Normalization, the RMSNorm paper.
- RoFormer: Enhanced Transformer with Rotary Position Embedding, the source of RoPE.
- GLU Variants Improve Transformer, which introduces the SwiGLU feed-forward used here.
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, the grouped-query attention paper.
Part 2 covered how text becomes the tokens that enter this stream. Part 4 opens the attention branch properly and asks why 64 heads instead of one.
Check your understanding
Take the 10 question quiz on this article
You get a full report at the end: your score, the correct answer to anything you missed, why it is correct, and a link straight back to the section it came from. Hints are there if you want them. Nothing is stored, and you can retake it as often as you like.
- 10 questions
- 4 select all that apply
- hint on every question
- timed, no limit
