Attention Is All You Need, Dissected: The 2017 Figure, Box by Box

Attention Is All You Need is a thirteen-page paper about machine translation, and its Figure 1 is the most reproduced diagram in machine learning. It gets pasted into explainers, admired for a sentence, and then abandoned in favour of a paragraph about queries and keys. The boxes themselves stay unexplained.

The previous seven parts of this series traced a modern decoder-only model from raw text to expert routing, which is exactly one of the three architectures that grew out of that figure. This part goes back to the original and takes it apart.

By the end you should be able to open the paper, look at Figure 1, and read it without hesitating: which tower is which, what the crossing arrow carries, why there are three attention blocks and not one, what “shifted right” means, and which boxes a 2026 model still contains.

Start here: it is a translation model

Attention Is All You Need is not about chatbots, or generation, or scaling laws. It is about translating English into German and English into French, and it exists to remove recurrence from the encoder-decoder translation systems of the time. Recurrent networks processed one word per step, so the time to train on a sentence grew with the sentence. That serialisation was the bottleneck, and attention was the way around it.

Read Figure 1 as a translation architecture and it makes immediate sense. Read it as “an LLM” and half of it looks inexplicable, because the task has one property that shapes the whole diagram: the source sentence is fully known before you start, while the target sentence is produced one word at a time and must never see its own future. Two sequences with different rules, so two towers.

How to read the diagram

Data flows upward. The left tower is the encoder, the right tower is the decoder, and the thick arrow crossing between them is the single most important line in the picture. The dashed boxes with “N x” beside them mean the block inside repeats six times, so what you are looking at is one layer drawn once rather than the whole stack.

Count the attention boxes before going further. There are three, not one. Encoder self-attention at the bottom left. Decoder masked self-attention at the bottom right. And decoder cross-attention in the middle right, the one fed by the crossing line. They run the same arithmetic and differ only in where their queries, keys and values come from.

The model that was actually trained

The base configuration is small enough to hold in your head, which is worth doing before you look at anything modern. Set it beside the Llama-3-70B class model this series has been using as its running example.

You can check the parameter count by hand, which is a good way to confirm you have understood the shapes. Each attention block holds four square matrices of 512 x 512, so about 1.05M parameters. Each feed-forward block holds 512 x 2048 plus 2048 x 512, so about 2.10M. An encoder layer has one attention block and one feed-forward block, giving 3.15M, and six of them give 18.9M. A decoder layer has two attention blocks plus a feed-forward block, giving 4.20M, and six of them give 25.2M. The shared embedding table is 37,000 x 512, another 18.9M. Add those and you land at roughly 63M, which is the paper’s 65M once you count the biases and layer norms.

The whole base model trained in about 96 GPU-hours on Pascal cards, three GPU generations before an H100. The big model took three and a half days. That is the entire compute budget behind the architecture every model in production today descends from.

The bottom of each tower

Input embedding, and the multiply by the square root of d_model

A lookup table, exactly as you would expect: token ID in, 512 numbers out. The detail most redrawings drop is that one matrix does three jobs. The encoder’s embedding table, the decoder’s embedding table and the final Linear layer before the softmax are all the same weights, which is why a single 18.9M-parameter table covers all three in a 65M-parameter model.

The second dropped detail is that the embeddings are multiplied by sqrt(d_model), which for d_model 512 is about 22.6. The reason is a scale mismatch. Embeddings initialised with variance 1/d_model have entries around 0.04, while the positional encodings that get added next are sinusoids bounded between minus one and one. Without the scaling, position would swamp identity. Modern models dropped this along with the weight tying, because they normalise the residual stream at the entrance to every block anyway.

Sinusoidal positional encoding, not learned

Attention has no notion of order. Permute the input and the output permutes with it. Something has to carry position, and the 2017 paper does not learn it. It computes it from a fixed formula: alternating sine and cosine waves at geometrically spaced frequencies, added once, at the bottom, and never again.

The frequency ladder is the part worth understanding. Dimension pair 0 has a wavelength of about 6.3 positions, so it cycles every few tokens. Each successive pair divides the frequency by a constant factor, so by the last pair the wavelength is around 62,000 positions and the wave is effectively a straight ramp. Every position therefore gets a unique fingerprint: fast dimensions resolve local order, slow dimensions resolve where you are in the document.

The authors had a specific hope for this. Because the encoding is a smooth function of position rather than a lookup table, it is defined at position 5,000 even if training only ever saw 100. And for any fixed offset, the encoding at pos + k is a linear function of the encoding at pos, so a model could in principle learn to attend by relative distance.

It did not extrapolate well in practice. Modern models use rotary embeddings instead, which Part 4 covered alongside the head layout. Rotation beat addition for two reasons. Added position occupies dimensions of the residual stream, where it competes with content and gets smeared by every matrix above it, whereas rotation only turns q and k inside the attention score, leaving the residual stream untouched. And because rotating both vectors by their absolute positions makes their dot product depend only on the difference, relative distance falls out of the arithmetic instead of having to be learned.

Three attention blocks, not one

Encoder self-attention, unmasked

The encoder’s attention is the operation you already know with the causal mask removed. Every source token attends to every other source token, including the ones to its right. That is legitimate here, because the whole source sentence exists before translation begins and nothing is being predicted. Bidirectional context is free.

Decoder masked self-attention

The decoder’s first attention block is the same operation with the future set to negative infinity before the softmax. Position 3 may see positions 1, 2 and 3, and nothing beyond. This is the block that survived into every generative model, unchanged. Its job is to track what has been written so far.

Cross-attention, where the two towers meet

The middle block in the decoder is the only place in the architecture where the queries and the keys come from different sequences. Be precise about this, because it is the thing most summaries get vague about:

• Queries come from the decoder. They encode “what does the German word I am about to write need to know?”
• Keys and values come from the encoder’s final output. They encode “here is everything the English sentence contains.”
• There is no mask. The decoder may look at the entire source at every step. Masking applies only to the decoder’s own past, in the block below.

The encoder output is computed once per sentence and reused by all six decoder layers, which is the 2017 ancestor of caching. Cross-attention is where translation actually happens: the query for the position about to emit “Katze” lands mostly on the encoder vector for “cat”.

All three blocks run the same arithmetic, so the differences fit in three columns: where Q comes from, where K and V come from, and whether a mask applies.

One primitive, wired three ways. That is the paper’s real claim.

Add and Norm, and the feed-forward block

Why the order was wrong

Each “Add and Norm” box does two things in sequence: add the sub-block’s input back in, then normalise the sum. That ordering is post-norm, and it is the paper’s most consequential mistake.

Putting the normalisation after the add means the normalisation sits on the residual path. A gradient travelling back down the encoder tower crosses two of them in every layer, twelve in all, and each one rescales what passes through. The decoder tower has three per layer, so eighteen more. Six layers a side survived that. Eighty would not have.

The placement is also why the original recipe needed a learning-rate warmup schedule to train at all. Without the ramp, the early updates near the output layer are large enough to destabilise the run.

Pre-norm moves the normalisation inside the branch, so the residual spine becomes a clean sum from input to output and gradients pass through it untouched. Warmup becomes optional. Depth becomes cheap. Nearly every model since 2019 made this switch, and it is the single change most responsible for transformers that are 80 layers deep instead of 6.

Feed forward: two matrices and a ReLU

The 2017 feed-forward network is the ungated ancestor of the block Part 6 pulled apart. Widen 512 to 2,048, apply ReLU, narrow back to 512. Two matrices, two bias vectors, one nonlinearity, applied independently at every position.

Two details date it. The biases are there, and modern models drop them because they buy nothing measurable. And d_ff is exactly 4 x d_model, a ratio that held for years. SwiGLU broke it by adding a third matrix, a gate that multiplies the up-projection elementwise. To keep the parameter budget equal against three matrices instead of two, the inner width came down to roughly 3.5 x d_model, which is exactly where the 70B model’s 28,672 sits against its 8,192.

“Shifted right”, the phrase that confuses everyone

The decoder’s input in Figure 1 is labelled “Outputs (shifted right)”, which reads like a typo the first several times you see it. It means something simple. Prepend a start token, drop the last target token, and feed that as the decoder input. Position i of the input is then the token that came immediately before the token position i has to predict.

If the target is “die Katze sass <eos>”, the decoder receives “<sos> die Katze sass”. Position 1 sees <sos> and must produce “die”. Position 2 sees “die” and must produce “Katze”. Without the shift, position 2 would be handed the very token it is being asked to predict, and the task would be a copy.

Teacher forcing is a training-time construct. During training the ground-truth target exists, so the whole sentence can be fed at once and every position supervised in one forward pass. At inference there is no target to shift, so the decoder feeds itself, one token at a time, which is why streaming exists and why time to first token behaves so differently from total latency. That asymmetry is the origin of the prefill and decode split that dominates serving economics later in this series.

The training recipe holds one more counterintuitive piece. The paper trains against a smoothed target distribution rather than a one-hot one, and reports that this made perplexity worse while making accuracy and BLEU better.

Figure 2: the mechanism itself

Scaled dot-product attention

Figure 2 left is five boxes stacked bottom to top: MatMul, Scale, an optional Mask drawn as a dashed rectangle, SoftMax, MatMul. The dashed box is the whole encoder-decoder difference, compressed into one optional step in one shared diagram.

The scale factor deserves an honest explanation, because “for numerical stability” is not one. Take a query and a key whose components are independent with mean 0 and variance 1. Their dot product sums d_k such products, so it has mean 0 and variance d_k, giving a standard deviation of sqrt(d_k). At d_k = 64 that is 8, so raw scores routinely land at plus or minus 16 or 24. Feed a 16-point logit gap into a softmax and you get a ratio of about nine million to one: the distribution collapses onto a single token, and the gradient through a saturated softmax is close to zero. Dividing by 8 pulls the spread back to roughly plus or minus 2, where the softmax is still soft and still differentiable. The scale is there to keep training alive rather than to keep floats in range.

Multi-head attention

Figure 2 right shows something the modern implementation hides. The paper draws h separate Linear projections, one per head, feeding h independent attention runs, then a Concat and a final Linear. In code those per-head projections are fused into one wide matrix and the result is reshaped, which is why heads are usually explained as slices of a single vector. The two are mathematically identical. The paper’s version is the conceptual one and the fused matrix is the efficient one.

Two boxes here have names from earlier in this series. Concat is where 8 x 64 = 512 reassembles, and the Linear on top is W_O, the mixing matrix that Part 5 was entirely about. The paper’s own argument for multiple heads is one sentence long: a single head averages, and averaging inhibits attending to several positions at once.

The three descendants, and the scorecard

Which tower each family kept

Encoder only. BERT and its descendants kept the left tower. Bidirectional attention, no causal mask, trained by masking tokens and predicting them. These models understand text and cannot generate it left to right. They are not historical: if you run a retrieval pipeline, your embedding model and your reranker are almost certainly from this family, as anyone who has built a retrieval augmented generation stack has discovered.

Encoder and decoder. T5 and BART kept both towers. This still makes sense when there genuinely are two distinct sequences with different modalities or rules: speech to text, image to text, and machine translation itself. Niche for general text work, entirely correct where it applies.

Decoder only. GPT, Llama and everything else this series has traced kept the right tower and deleted cross-attention with it. This family won generation outright.

Why the encoder was dropped

The reason is not that encoders are bad. It is that in general text generation there is no separate source sequence to encode. There is one stream, and everything before the current token is context. The decoder’s own masked self-attention over a growing context already builds a representation of all of it, so it does the encoder’s job in-band, with the same weights.

Then serving economics finished the argument. One tower means one set of weights, one training objective, one KV cache, and a uniform execution path. Look at what a modern server actually does and you will notice the split came back anyway, in a cheaper form: prefill processes the whole prompt in parallel, exactly as an encoder would, and decode emits one token at a time, exactly as the decoder did. Same two phases, one stack of weights. Anyone running these models in production is paying for that distinction on every request, and the next part explains why the two phases have completely different cost profiles.

What 2017 got right and what it got wrong

Six decisions carry the architecture, and nine years of practice has settled the verdict on each of them.

What survived untouched is a short list: scaled dot-product attention, multiple heads, residual connections, the attention-then-feed-forward block ordering, and Q, K, V as three projections of the same input. That core is nine years old and unamended. Everything in the table above is a fix to the packaging around it.

Key takeaways

• Figure 1 of Attention Is All You Need is a translation architecture. The two towers exist because translation has two sequences with different rules, and the source can be read bidirectionally while the target cannot.
• There are three attention blocks, not one. They run identical arithmetic and differ only in where Q, K and V come from and whether a mask applies.
• Cross-attention takes queries from the decoder and keys and values from the encoder output, with no mask. It is the only asymmetric block in the paper, and it does not exist in decoder-only models.
• “Shifted right” means prepend a start token and drop the last target token, so every position is predicting something it cannot see. It is teacher forcing, and it applies at training time.
• Dividing by sqrt(d_k) exists because a dot product of d_k unit-variance terms has standard deviation sqrt(d_k). Without the division the softmax saturates and the gradient dies.
• Post-norm was the paper’s costliest mistake. It forced learning-rate warmup and made depth beyond a dozen layers unreliable, which pre-norm fixed.
• The encoder was dropped for generation because there is no separate source to encode. One tower is simpler to train, cheaper to serve, and does the encoder’s job in-band.

Frequently asked questions

What is Attention Is All You Need actually about?

It is a machine translation paper. It proposes replacing the recurrent networks used in encoder-decoder translation systems with attention alone, so that training parallelises across all positions in a sentence instead of stepping through them one at a time. The title is a claim about subtraction: recurrence and convolution can be deleted.

Why are there three attention blocks in the transformer diagram?

Encoder self-attention reads the source sentence bidirectionally. Decoder masked self-attention tracks what has been written so far without seeing the future. Cross-attention lets the decoder consult the encoded source. They are the same operation with different wiring for queries, keys and values.

What does “outputs shifted right” mean in the transformer diagram?

It means the decoder input is the target sequence with a start token prepended and the final token dropped, so each position receives the token before the one it must predict. This is teacher forcing. It lets an entire target sentence be supervised in one parallel forward pass during training.

Why divide by the square root of d_k in scaled dot-product attention?

A dot product of two vectors with d_k independent unit-variance components has variance d_k, so a standard deviation of sqrt(d_k). At d_k = 64 that is 8, large enough that the softmax saturates onto one token and its gradient vanishes. Dividing by sqrt(d_k) restores a workable spread.

Why did decoder-only models replace the encoder-decoder transformer?

In general text generation there is no separate source sequence to encode, so the decoder’s masked self-attention over a growing context already does the encoder’s job. One tower means one set of weights, one objective, one KV cache and a simpler serving path, and it scales more cleanly.

Why do modern transformers use pre-norm instead of post-norm?

The 2017 paper normalises after the residual addition, which puts the normalisation on the residual path and forces every gradient to pass through it. That required a learning-rate warmup schedule and made deep stacks unstable. Pre-norm moves the norm inside the branch, leaving a clean residual spine and making 80-layer models trainable.

Sources and further reading

• Attention Is All You Need, Vaswani et al., 2017. The source for every configuration number and formula above.
• On Layer Normalization in the Transformer Architecture, Xiong et al., 2020. The analysis of why post-norm needs warmup and pre-norm does not.
• RoFormer: Enhanced Transformer with Rotary Position Embedding, Su et al., 2021. The positional encoding that replaced sinusoids.
• GLU Variants Improve Transformer, Shazeer, 2020. The gated feed-forward that replaced ReLU.
• BERT: Pre-training of Deep Bidirectional Transformers, Devlin et al., 2018. The encoder-only branch.
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, Raffel et al., 2019. The T5 encoder-decoder branch.

Part 7 covered conditional computation and expert routing, the last structural change to the block itself. Part 9 turns from architecture to hardware and explains why decode is memory-bound while prefill is notcoming 17 Aug, which is where the training-time parallelism this paper won stops helping you.