The Feed-Forward Network: Where a Transformer Keeps What It Knows

A transformer block holds seven weight matrices. Three of them belong to the feed-forward network, and between them those three carry about 82 percent of the block’s parameters. Attention gets the papers, the talks and the diagrams. The feed-forward network gets the weights.

Most explanations of it fit in one sentence: widen, apply a nonlinearity, narrow. That sentence is accurate and it tells you nothing. It does not tell you why the widening helps, why there are three matrices in a modern model instead of two, or why the people who edit facts into a language model go straight for these weights and ignore attention entirely.

Part 5 closed out the attention half of the block with W_O. This part takes the other half. By the end you will be able to trace six channels of a real gated FFN by hand: raw gate score, SiLU, elementwise multiply, weighted sum of output rows, final vector. Every number is checkable with a calculator. You will also know why the width is 28,672 and not 32,768, why widening d_model instead would cost roughly five times as much for the same gain, and why this is the one slot in the block that Mixture of Experts can replace.

The running model is the same as everywhere else in this series: 80 layers, d_model 8,192, 64 query heads, 8 KV heads, FFN width 28,672.

Orientation: W_gate and W_up are siblings

W_gate and W_up are siblings. Both receive the same normalised vector, both run at the same time, and both widen 8,192 to 28,672. They are not a chain, and nothing flows from one into the other. They meet later, at an elementwise multiply, and only W_down is downstream of anything. If your mental picture is x into W_gate into W_up, throw it away now, because every later step will confuse you.

On a GPU the two are often fused into a single matmul against a concatenated weight tensor, which is a performance detail rather than a change of structure. It works precisely because they are independent.

W_gate supplies permission

One raw score per channel

Mechanically it is an ordinary matrix multiply: 8,192 numbers in, 28,672 numbers out, one per channel of the wide space. W_gate is 8,192 x 28,672, which is 234.9 million parameters, or about 470 MB at bf16.

Those raw numbers are not fader positions yet, and they can be large, small or negative. Our toy has six channels instead of 28,672 so the arithmetic stays visible, and the raw scores are [2.1, -3.0, 0.4, -0.2, 4.0, 1.1].

SiLU turns scores into fader positions

Run our six raw scores through it and you get [1.87, -0.14, 0.24, -0.09, 3.93, 0.83]. Check one: sigmoid(2.1) is 0.8909, and 2.1 * 0.8909 = 1.871. The score of -3.0 collapses to -0.14, so that channel is now effectively shut. The score of 4.0 passes through almost intact at 3.93.

One honest point about SiLU. Because it goes slightly negative, a closed channel is not exactly zero, and it can push a small amount of inverted signal into the output. That is a real property of the function and not a rounding artefact, and it shows up in the worked arithmetic later in this article.

The multiply is where the gate acts

Now W_up‘s content arrives and the two vectors are multiplied position by position. Channel 1’s gate multiplies channel 1’s content, and there is no mixing across channels at all. Say the content values are [1.0, 5.0, 2.0, 3.0, 1.5, 2.0]. The gated result is [1.87, -0.70, 0.48, -0.27, 5.90, 1.66].

Two channels are worth pausing on. Channel 2 carried content of 5.0, the largest value on the row, and a gate of -0.14 crushed it to -0.70. Channel 5 carried only 1.5, and a gate of 3.93 lifted it to 5.90, the largest result. The loudest content lost and the quietest content won, which is the point of the whole mechanism: what survives is decided by the gate and not by the content’s own magnitude.

Two questions, asked by two matrices

Notice what the split bought. W_up answers “what could this channel contribute?” and W_gate answers “should it, for this token?”. Two different questions, two different matrices, combined only at the end, so neither matrix has to compromise to do the other’s job.

W_gate is fixed after training, and it is multiplied by a different normalised vector for every token, so it produces a different set of 28,672 fader positions every time. One matrix, an unlimited number of filters.

Without a gate you get the 2017 arrangement: widen, apply a nonlinearity, narrow. Every token receives the same treatment, and the only way to suppress a channel is for the content itself to be small.

W_up supplies content, and every channel is a detector

W_up sits beside W_gate, takes the same input, and produces the same shape. Its numbers mean something completely different: the gate produced levels, and W_up produces the material those levels act on.

Look at a single column of W_up. It is a vector of 8,192 numbers, and computing channel 5 means taking the dot product of that column with the token’s vector, summed over 8,192 terms. A dot product measures alignment, so the column is a learned pattern and the channel’s value says how strongly this token matches it. Large means a strong match, near zero means no match, and negative means the token looks like the opposite of the pattern.

That reframes the wide space. It is a bank of 28,672 independent pattern detectors, all evaluated on every token, rather than a bigger version of the residual stream.

Why widen at all

The token arrives as 8,192 numbers, so why blow it out to 28,672? Because you cannot have more clearly distinguishable detectors than you have dimensions. An 8,192-dimensional space gives you room for roughly 8,192 patterns that do not sit on top of each other, and widening to 28,672 gives you room for about 3.5 times as many, with space left between them.

The nonlinearity then runs in the wide space, where the patterns are already separated. Patterns that overlap and interfere at 8,192 dimensions can be pulled apart at 28,672. The width here buys capacity to recognise rather than capacity to represent.

Why not just make d_model bigger instead

If width helps, the obvious move is to widen everything. The answer is that d_model is paid for in every matrix in every block, plus the embedding table, plus the LM head, plus the KV cache. FFN width is paid for in three matrices, and the wide vector never leaves the block, so the cache never sees it.

Make it concrete. Add 1,024 to each width and count what it costs, keeping head_dim at 128 and the GQA ratio at 8 query heads per KV head.

Same 1,024 of extra width, roughly five times the weight cost, and 320 MiB more cache for every concurrent sequence you want to serve at 8k context.

W_down and the forced return to 8,192

The block’s last act is x + FFN_output, and addition demands matching shapes. You cannot add a 28,672-wide vector to an 8,192-wide one, so whatever the FFN produces must be 8,192 wide. The widening was always temporary, and W_down is a structural obligation imposed by the residual stream rather than a design preference. Part 3 walked through the residual stream and why every sublayer has to hand back the same shape it was given.

Each channel owns an output vector

Turn W_down on its side and it becomes readable. It has 28,672 rows, each 8,192 numbers long, and row 5 is what channel 5 writes into the residual stream when it fires. The output is those rows added up, each weighted by how strongly its channel fired, so a channel that fired hard contributes a lot of its row and a channel that was gated shut contributes almost nothing.

The arithmetic, carried all the way through

Our gated values were [1.87, -0.70, 0.48, -0.27, 5.90, 1.66]. Give each channel a short output row of 4 numbers instead of 8,192, and add them up.

Check the first component yourself: 1.87(1) + 0.48(2) + 5.90(3) + 1.66(1) = 1.87 + 0.96 + 17.70 + 1.66 = 22.19. Channels 2 and 4 had zeros in the first position and drop out.

Check the second: -0.70(3) + 0.48(1) + 1.66(2) = -2.10 + 0.48 + 3.32 = 1.70. Here channel 2, the one the gate shut, does contribute, and it contributes with a flipped sign. That is the SiLU negative dip showing up in the output. It is small, and it is not zero.

Scale that up to 28,672 rows of 8,192 numbers and you have the vector this block adds to the residual stream for this token. Nothing else about the computation changes.

The payoff: the FFN is a key-value memory

Model-editing work followed the same road. ROME changes a specific factual association by making a rank-one edit to a middle-layer FFN weight matrix, and it works well enough to be a standard baseline.

Be careful how far you push the storage picture. “Facts live in the FFN” is a good working model and a bad literal claim. Hase and colleagues showed that you can successfully edit a fact by changing weights in a layer quite different from the one causal tracing points at, which means editability and localisation are not the same property. A given fact is usually spread across several layers and many channels.

The division of labour across a block is cleaner than the localisation question. Attention moves information between positions and the feed-forward network stores it. Edit attention weights and the model routes differently. Edit FFN weights and the model believes differently.

Where the parameters actually go

Here is the whole block, counted. Every matrix in the feed-forward network is the same size, which follows from all three touching the same two widths.

Across 80 layers that is 112.7 GB of FFN weights against 24.2 GB of attention weights, on a model whose bf16 checkpoint is roughly 140 GB. When you generate a token at batch size 1, the GPU reads essentially all of that from HBM to produce a single token, which is why decode is memory-bound. Part 9 puts numbers on thatcoming 17 Aug, and the same reasoning shows up whenever you are thinking about memory rather than arithmetic as the scarce resource.

Classic FFN versus modern SwiGLU

The 2017 FFN had two matrices and a ReLU, with an inner width of four times d_model. SwiGLU adds a third matrix, so at the same width it would cost 50 percent more parameters for the same slot. The standard fix is to scale the width down by two-thirds, which makes the swap parameter-neutral: three matrices at two-thirds the width cost exactly what two matrices at full width cost.

So the honest answer to “why 28,672 and not 32,768” has two halves. The two-thirds rule is why the number is not simply 4 times 8,192, since adding a third matrix without shrinking the width would inflate the FFN by half for no structural reason. Early Llama models followed the rule closely, with Llama-2-7B using 11,008 against a d_model of 4,096, which is 2.69 times and near the 8/3 the rule prescribes.

Why this is the slot Mixture of Experts replaces

One property has been quietly true throughout: the feed-forward network processes each token entirely independently. Token 4 knows nothing about token 3 inside the FFN, because all the cross-token work happened in attention, upstream.

That independence is what makes the slot swappable. You cannot do the same to attention, because attention’s entire job is to mix across positions, so routing tokens to separate attention modules would sever the connections it exists to make. The FFN is the only slot in the block where conditional computation is structurally free. Part 7 builds an MoE layer from zerocoming 15 Aug on exactly this foundation, including what it does to your serving memory when all the experts have to be resident whether they fire or not.

Key takeaways

• W_gate and W_up run in parallel from the same input and meet at an elementwise multiply. Only W_down is downstream.
• W_gate supplies permission and no content. SiLU converts its raw scores into fader positions, dipping to about -0.278 at worst, so a closed channel leaks a small inverted signal rather than exactly zero.
• Each column of W_up is a learned pattern and each channel value is a match score. Widening to 28,672 buys room for about 3.5 times as many distinguishable detectors.
• Widening d_model by the same 1,024 would cost roughly five times the weights and 12.5 percent more KV cache per token. FFN width costs three matrices and nothing in the cache.
• Each of the 28,672 rows of W_down is an output vector, and the FFN’s result is those rows summed with the gated activations as weights. The residual add is what forces the return to 8,192.
• W_up holds keys and W_down holds values, so the FFN behaves like a soft key-value lookup. Attention moves information between positions; the FFN stores it.
• Three matrices at 234.9M parameters each hold 82.4 percent of a block, and the feed-forward network’s per-token independence is what lets Mixture of Experts replace it.

Frequently asked questions

Is W_up applied after W_gate?

No. They are siblings. Both read the same normalised residual vector, both run at the same time, and both project 8,192 dimensions up to 28,672. They meet only at the elementwise multiply, after SiLU has been applied to the gate branch. Many inference kernels fuse the two into one matmul precisely because neither depends on the other.

What does SiLU do that ReLU does not?

SiLU is x * sigmoid(x). It is smooth everywhere, so it has gradient below zero, and it dips to about -0.278 near an input of -1.28 before flattening toward zero. ReLU has a hard corner and a completely flat dead region, so a channel that lands there gets no gradient and can stay dead for the rest of training. The smoothness also lets the gate express partial openness rather than a binary on or off.

Why is the FFN width 28,672 and not 4 x 8,192 = 32,768?

SwiGLU uses three matrices where the classic FFN used two, so keeping the 4x width would inflate the FFN by 50 percent. The standard fix is to scale the width by two-thirds, which for d_model 8,192 gives 21,845 and makes the swap parameter-neutral. Llama-3 chose 3.5 times d_model, or 28,672, which takes most of the discount and spends part of it back on extra width.

How many parameters are in the FFN compared to attention?

For a Llama-3-70B class block, the three FFN matrices hold 704,643,072 parameters and the four attention matrices hold 150,994,944. That is 82.4 percent of the block in the FFN. Across 80 layers it is about 112.7 GB of FFN weights against 24.2 GB of attention weights at bf16.

Does the FFN really store the model’s facts?

The key-value memory reading is well supported, and model-editing methods like ROME work by changing FFN weights. Be careful about stronger claims. Later work found you can edit a fact successfully in a layer other than the one causal tracing identifies, individual facts are usually spread across several layers, and channels are polysemantic. The FFN is where storage happens; clean per-neuron localisation of a given fact is not established.

Why can Mixture of Experts replace the FFN but not attention?

The FFN processes every token independently, with no cross-token dependency inside it. That means you can route different tokens to different copies of the three matrices without breaking anything. Attention exists to mix information across positions, so splitting tokens across separate attention modules would destroy the connections it is there to build.

Sources and further reading

• Attention Is All You Need (Vaswani et al., 2017) sets out the original two-matrix ReLU feed-forward sublayer and the 4x width convention.
• GLU Variants Improve Transformer (Shazeer, 2020) introduces SwiGLU and the two-thirds width adjustment that keeps the three-matrix version parameter-comparable.
• Language Modeling with Gated Convolutional Networks (Dauphin et al., 2016), where the gated linear unit was introduced.
• Transformer Feed-Forward Layers Are Key-Value Memories (Geva et al., 2020) is the paper behind the keys-and-values reading of W_up and W_down.
• Locating and Editing Factual Associations in GPT (Meng et al., 2022) introduces ROME, which edits facts by a rank-one change to middle-layer FFN weights.
• Does Localization Inform Editing? (Hase et al., 2023) is the useful counterweight: successful edits and causal localisation do not line up as neatly as you would hope.
• The Llama 3 Herd of Models (Meta, 2024) documents the configuration used as the running example here, including the 8,192 model dimension and 28,672 FFN width.