How Transformers Actually Work (No Hand-Waving)

A transformer is not a database of sentences, a symbolic reasoner, or a mysterious machine that “understands attention.” At its core, it is a differentiable program that repeatedly mixes information across token positions and transforms each position independently. Training adjusts billions of numeric parameters so that the resulting program predicts useful continuations.

That description is simple, but it leaves plenty of room for hand-waving. What exactly enters the model? Where do queries, keys, and values come from? Why divide by a square root? What changes between training and generation? This article follows one tensor through a decoder-only transformer, the architecture behind most generative large language models, and keeps the shapes visible along the way.

From Text to Positioned Vectors #

Neural networks consume numbers, not strings. A tokenizer first maps text to a sequence of integer token IDs. Modern tokenizers usually operate on subword or byte-derived units: common fragments may occupy one token, while rare words split into several. Tokenization is deterministic for a fixed vocabulary, but it is not linguistically neutral. A model can find arithmetic, unusual names, or some languages harder partly because their text requires more tokens.

For a vocabulary of size V, the model owns an embedding table $E \in \mathbb{R}^{V \times d_{\text{model}}}$. Looking up T token IDs produces

where $B$ is batch size, $T$ is sequence length, and $d_{\text{model}}$ is the model width. An embedding is not a dictionary definition. It is a learned coordinate vector whose usefulness comes from how later layers process it. During training, gradients move vectors for tokens that need similar downstream behavior into useful geometric relationships.

Self-attention alone does not know order: permuting the input rows would permute its output rows in the same way. The model therefore injects position information. The original transformer added fixed sinusoidal vectors. Other models learn absolute position embeddings or rotate query and key coordinates with RoPE. These choices differ in extrapolation behavior, but they solve the same problem: make “dog bites person” distinguishable from “person bites dog.”

With additive positions, the first layer receives

RoPE instead modifies queries and keys inside attention. In either case, token identity and position become available to every block.

Q, K, V, and Scaled Dot-Product Attention #

Given hidden states $X$, a learned linear projection creates three views:

The names come from information retrieval, but there is no separate query sentence or key database. Every token produces all three. A query encodes what the current position wants to find; a key encodes what each source position offers; a value carries the information to retrieve. Their meanings are learned, not assigned by a programmer.

For one attention head, similarity is the dot product $QK^T$. Softmax turns each row into nonnegative weights summing to one, and the weighted sum selects from $V$:

Here $d_h$ is head width and $M$ is a mask. Without scaling, the variance of dot products grows with $d_h$, pushing softmax toward extremely sharp regions with small gradients. Dividing by $\sqrt{d_h}$ keeps logits at a trainable scale.

A decoder uses a causal mask: position $i$ may attend only to positions $j \le i$. Future logits receive $-\infty$ before softmax, so their probability is zero. Padding masks can additionally hide placeholder tokens when variable-length examples share a batch.

Multi-head attention runs this operation in parallel subspaces. If the model width is 768 and there are 12 heads, each head commonly has width 64. Heads can learn different routing patterns, although assigning a crisp human role to every head is usually an oversimplification. Their outputs are concatenated and mixed by another learned projection.

The following PyTorch implementation makes the dimensions explicit. The boolean mask is broadcastable to [B, H, Tq, Tk], where True means visible.

import math

import torch
from torch import Tensor, nn


def scaled_dot_product_attention(
    query: Tensor,
    key: Tensor,
    value: Tensor,
    mask: Tensor | None = None,
) -> tuple[Tensor, Tensor]:
    # query: [B, H, Tq, Dh], key/value: [B, H, Tk, Dh]
    scores = query @ key.transpose(-2, -1)
    scores = scores / math.sqrt(query.size(-1))

    if mask is not None:
        scores = scores.masked_fill(~mask, float("-inf"))

    weights = torch.softmax(scores, dim=-1)
    output = weights @ value
    return output, weights


class CausalMultiHeadAttention(nn.Module):
    def __init__(self, d_model: int, num_heads: int) -> None:
        super().__init__()
        if d_model % num_heads != 0:
            raise ValueError("d_model must be divisible by num_heads")

        self.num_heads = num_heads
        self.head_dim = d_model // num_heads
        self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)
        self.output = nn.Linear(d_model, d_model, bias=False)

    def forward(self, hidden: Tensor) -> Tensor:
        # hidden: [B, T, D]
        batch, tokens, width = hidden.shape
        qkv = self.qkv(hidden)
        qkv = qkv.view(batch, tokens, 3, self.num_heads, self.head_dim)
        qkv = qkv.permute(2, 0, 3, 1, 4)
        query, key, value = qkv.unbind(dim=0)  # each [B, H, T, Dh]

        causal_mask = torch.ones(
            tokens, tokens, dtype=torch.bool, device=hidden.device
        ).tril()
        causal_mask = causal_mask.view(1, 1, tokens, tokens)

        attended, _ = scaled_dot_product_attention(
            query, key, value, causal_mask
        )
        attended = attended.transpose(1, 2).contiguous()
        attended = attended.view(batch, tokens, width)
        return self.output(attended)

This is mathematically correct reference code, not a fast production kernel. Real systems use fused implementations such as FlashAttention to avoid materializing the full $T \times T$ score matrix in high-bandwidth memory.

The Rest of a Transformer Block #

Attention communicates across positions, but it is only half the block. A position-wise multilayer perceptron, or MLP, transforms each token independently. A common form expands width by roughly four times, applies a nonlinear activation such as GELU or SwiGLU, then projects back to $d_{\text{model}}$. Attention answers “which earlier information should reach this position?” The MLP answers “what computation should happen to the information now present here?”

Residual connections add a sublayer’s input to its output. In a pre-normalization decoder block, the computation is approximately

The residual path gives gradients a direct route through deep networks and lets each sublayer learn a refinement instead of reconstructing the whole representation. LayerNorm or RMSNorm controls activation scale. Pre-norm architectures normalize before each sublayer and generally train deep stacks more reliably than the original post-norm arrangement.

After dozens or hundreds of blocks, a final normalization feeds an output projection with one logit per vocabulary token. Often this projection shares weights with the input embedding table. Softmax would convert logits into probabilities, although training code usually passes logits directly to a numerically stable cross-entropy function.

Training Is Parallel; Generation Is Sequential #

During pretraining, the objective is usually next-token prediction. Given tokens $x_1,\ldots,x_T$, the model predicts each token from its prefix and minimizes average negative log-likelihood:

The inputs and targets are the same sequence shifted by one position. Thanks to the causal mask, all positions can be processed in parallel during training without leaking future tokens. Backpropagation computes how every parameter contributed to the loss; an optimizer such as AdamW updates those parameters over enormous datasets. Instruction tuning and preference optimization can later reshape behavior, but they operate on the same basic next-token machinery.

Inference has no known future sequence. The model processes a prompt, obtains logits for the next token, chooses one, appends it, and repeats. Greedy decoding selects the largest logit. Temperature rescales logits before sampling; top-k and top-p restrict the candidate set. These strategies change selection, not the transformer’s learned knowledge.

Teacher forcing creates an important mismatch: during training, each position sees the correct earlier tokens from the dataset; during generation, it sees its own earlier choices. One weak choice can alter every later conditional distribution. The model is not retrieving a finished answer. It is repeatedly extending a prefix.

KV Caching, Context Cost, and Hard Limits #

Naive generation reruns the entire prefix for every new token. That needlessly recomputes keys and values for old positions. In a causal decoder, old hidden states do not change when a token is appended, so each layer can cache their projected $K$ and $V$ tensors. At the next step, the model computes a query, key, and value only for the new token, appends the new key and value, and lets the new query attend over the cached history.

For batch size $B$, layers $L$, heads $H$, cached length $T$, and head width $d_h$, KV storage is proportional to

The factor 2 represents keys and values. Multiply by bytes per element to estimate memory. Grouped-query and multi-query attention reduce cache size by sharing fewer key/value heads across query heads. Quantized caches trade some precision for capacity.

Caching makes each decode step much cheaper than recomputing the prefix, but it does not make long context free. The new query still interacts with all cached positions, cache memory grows linearly with context, and prefill over the original prompt remains expensive. Standard attention’s training and prefill work grows quadratically with sequence length, even when fused kernels reduce memory traffic.

Transformers also have conceptual limits. A finite context window means old information must be truncated, summarized, or retrieved again. Next-token training rewards plausible continuation, not truth, so confident fabrication is possible. Token-level computation can be brittle on exact arithmetic and long algorithms. Training data creates knowledge cutoffs and biases. More parameters and context improve capability, but do not remove these objective-level constraints.

Takeaways #

• A tokenizer converts text into model-specific IDs; embeddings and position mechanisms turn those IDs into ordered vectors.
• Learned projections produce queries, keys, and values. Scaled dot products compute routing weights, while causal masking prevents access to future positions.
• Multiple heads route information in parallel subspaces; MLPs then transform each position. Residual connections and normalization make deep stacks trainable.
• Training predicts all shifted targets in parallel under a causal mask. Inference must choose one token and feed it back before computing the next.
• KV caching reuses old keys and values, reducing repeated work while consuming memory that grows with context length.
• A transformer is a powerful conditional sequence model, not an oracle. Its architecture, objective, tokenizer, context window, and data all shape what it can and cannot do.

Once the tensor shapes are explicit, the apparent magic becomes an engineered loop: represent tokens, route context, transform features, score the vocabulary, and repeat. The scale is extraordinary; the operations are concrete.