attention is all you need

in one sentence

before this paper, a machine read a sentence the way a spider crawls its web, one thread at a time, left to right, half-forgetting the first thread by the time it reaches the last. the transformer throws that out. it lets the spider sit still and feel the whole web at once.

the one idea

the spider doesn't watch, it feels

a spider doesn't watch its web. it sits at the center, legs resting on the silk, and it feels. prey lands, the threads carry a vibration home, and the spider weighs which threads are buzzing, how hard, and from where. that weighing is the entire idea behind the transformer. every word sits on the web and feels every other word pulling on it.

what every thread carries

a question, an offer, a signal

each word on the web carries three things. a query: the question it sends out, who here relates to me? a key: what it advertises to everyone else, the tension it holds on its thread. and a value: the signal it actually passes along when plucked. attention measures how strongly each word's key answers another word's query, turns that into weights that sum to one, and pulls back a blend of the values, mostly from the threads that sing loudest.

those three aren't handed to you. first, every word gets turned into a long list of numbers, its meaning written out as coordinates. that list is called a vector, and the whole sentence, stacked into rows, is X. then the spider spins the three threads from X by passing it through three grids of numbers it has slowly tuned (a grid of numbers like that is a matrix): one grid makes the query, one the key, one the value. those three grids, written W_Q, W_K, W_V, are the only thing training actually learns.

why eight

a spider has eight legs

so far the spider feels the web one way, and one way is thin. a spider doesn't trust a single leg. each one rests on a different part of the web and feels for a different thing: one for distance, one for direction, one for rhythm. the transformer copies this exactly. it runs h = 8 attentions in parallel, eight independent reads of the same web, each in its own slice of the numbers, then pools them into one decision. the paper really does use eight.

how it knows order

the shape of the web is the memory

eight legs feel the web from eight angles, but they all assume the web has a shape. on its own, it doesn't. a web with no shape is just tangled silk, and its geometry is what lets the spider tell a near thread from a far one. attention has the same blind spot: on its own it sees a bag of words with no order, and shuffling them changes nothing. so each word gets stamped with its position, a pattern of waves at different speeds, like coordinates woven into the web. now near and far carry meaning.

why it's fast

feel it all at once, don't crawl it

the spider now feels the web, in eight ways, and tells near from far. the last question is speed. the old networks crawl the web one thread at a time, checking each in turn, slow, and forgetting the start of the sentence by the end. attention feels every thread in a single pulse. one step, not n steps. and because any two words touch through the web directly, distance stops mattering. that is the whole reason this trains at internet scale and holds a long thought.

the whole web

two stacks, one block repeated

stack that one move, layer on layer, and you get the whole model: two towers. the encoder (left) feels the input and builds a rich version of it. the decoder (right) writes the output one word at a time, allowed to feel what it has already written and to reach across to the encoder. each tower is the same block repeated N = 6 times, and each block is just two moves: feel, then think. the feel is the attention you already met. the think is every word stepping aside on its own to chew on what it just gathered (that step is the feed forward). wrapped around both is a safety net the diagrams call “add & norm”: keep a copy of what went in so nothing gets lost, and gently rescale the numbers so they never blow up. one rule on the decoder: it can't feel a thread that hasn't been spun yet. no peeking at the future.

build it from scratch

the whole web, in numpy

you've now seen every thread of it: feeling the web, in eight ways, with position woven in, stacked six passes deep. so here's my real test for whether it clicked. can you write it with no framework? everything above collapses into one forward pass. read these three blocks and you've read a transformer. no autograd, no magic, just matrix multiplies and the line from the top of this page.

import numpy as np

def softmax(x):
    e = np.exp(x - x.max(-1, keepdims=True))
    return e / e.sum(-1, keepdims=True)

def attention(Q, K, V, mask=None):        # Q,K,V: (seq, d_k)
    d_k     = Q.shape[-1]
    scores  = Q @ K.T / np.sqrt(d_k)       # how hard each thread pulls
    if mask is not None:
        scores = np.where(mask, scores, -1e9)   # can't feel the future
    weights = softmax(scores)              # each row sums to 1
    return weights @ V                     # the weighted blend

↳ scores is how hard every thread pulls, weights is how loud each vibration comes back. the mask is the decoder's one rule: blank the threads not yet spun, before the softmax.

def multi_head(X, Wq, Wk, Wv, Wo, h, mask=None):   # X: (seq, d)
    seq, d = X.shape
    d_k    = d // h                        # 512 / 8 = 64
    Q, K, V = X @ Wq, X @ Wk, X @ Wv       # spin the three threads

    def split(M):                          # one read per leg: (h, seq, d_k)
        return M.reshape(seq, h, d_k).transpose(1, 0, 2)

    Q, K, V = split(Q), split(K), split(V)
    legs = [attention(Q[i], K[i], V[i], mask) for i in range(h)]
    out  = np.concatenate(legs, axis=-1)   # pool the eight reads
    return out @ Wo

↳ eight legs, eight reads of the same web, pooled and mixed by W_o.

def layer_norm(X, g, b, eps=1e-5):
    mu  = X.mean(-1, keepdims=True)
    var = X.var(-1, keepdims=True)
    return g * (X - mu) / np.sqrt(var + eps) + b

def feed_forward(X, W1, b1, W2, b2):       # think about each word on its own
    return np.maximum(0, X @ W1 + b1) @ W2 + b2

def block(X, p, mask=None):                # feel, then think
    X = X + multi_head(layer_norm(X, *p["ln1"]), *p["attn"], p["h"], mask)
    X = X + feed_forward(layer_norm(X, *p["ln2"]))
    return X                              # the + keeps the original silk taut

def transformer(token_ids, P):
    X = P["emb"][token_ids] + P["pos"][:len(token_ids)]   # words + positions
    for layer in P["layers"]:              # N = 6 passes
        X = block(X, layer)
    return X @ P["emb"].T                  # scores over the vocabulary

↳ this is the whole web, end to end: words and positions in, six passes of feel-then-think, a final projection to the next word. give those matrices random numbers, train them by guessing the next word a few trillion times, and you have gpt, you have claude.

that is the entire architecture. give those matrices random numbers, train them by guessing the next word a few trillion times, and the same web becomes gpt, becomes claude. nothing past here changes the shape; it only grows it.

where these webs are

the spider doesn't care what it feels

the paper starts with translation, but the web doesn't care what's on it. words, pixels, sound, the rungs of a protein, turn it into a sequence and the same spider finds the shape. so it ends up under almost everything you touch, usually without saying so.

do you even need this?

it depends who you are

not everyone has to know the web. here's the honest split.

why it matters

it's the shape of the wave

the deeper point: one architecture quietly became the floor under this entire era. text, images, sound, biology, all of it runs through the same web from the same paper. knowing the transformer is not a party trick.

it's knowing the shape of the thing reshaping whatever you do. you don't have to weave the web yourself. you do have to know it's a web, and roughly how it feels. that's the difference between riding this wave and getting caught under it.