the mechanism behind modern ai
attention
claude, the autocomplete in cursor, the translator on your phone, all of them run on one idea: let every word feel every other word, and decide what matters.
what you're seeing
each word reaches for the others
ten words sit on a web. every glowing thread is one word feeling another, pulling in a little of its meaning. brighter and thicker means it leans on that word harder.
the model does this for every word at once, in a single pulse. that parallelism is the whole trick.
specialization
a leg learns one kind of feeling
a spider doesn't trust one set of threads. it runs many attention heads in parallel, eight legs. one tracks the previous word, one finds the subject, one stays local.
here the colours are different heads, each with its own habit. real models stack dozens.
the payoff
who is “it”?
watch the rose head from the word it. its strongest thread lands on cat, the model has resolved the pronoun across the whole sentence.
nobody wrote a rule for grammar. the pattern is just what attention learned.
one dial
sharp focus, or a soft haze
before the threads are drawn, the scores get divided by a temperature. low temperature collapses attention onto a single word; high temperature spreads it into an even wash.
it's the same dial you set on an llm when you turn the temperature up or down.
a concept in nature
a flock has no leader
watch a murmuration of starlings. no bird is in charge. each one simply feels its nearest neighbours, weighting the close ones more, and adjusts.
that's attention exactly: simple local rules, run by everyone at once, with the shape of the whole emerging from them. no conductor, no center.
in the wild
where this lives today
the same primitive, everywhere you've seen ai work.
code that knows your file
attention reaches back across hundreds of lines to reuse something you defined far above.
holding a conversation
every reply feels the whole chat, how it still knows what “it” meant ten messages ago.
aligning languages
the original 2017 use: match words across languages even when the order flips.
pixels and proteins
image patches feel each other (vit); amino acids feel each other to fold proteins (alphafold).
why it matters
it's why this era happened
long-range memory
old networks crawl left-to-right and forget. attention connects any two words in one step.
fully parallel
every word at once, not in sequence, which maps perfectly onto gpus, so you can train on the whole internet.
it just scales
the same block works at one layer or a hundred billion parameters. stack it and capability keeps climbing.
the whole idea
it's four lines of python
strip away the engineering and self-attention is one short function.
def attention(Q, K, V): d = Q.shape[-1] scores = Q @ K.T / np.sqrt(d) # how hard each thread pulls weights = softmax(scores) # each thread's brightness return weights @ V # gather along the threads
the raw pull of every thread.
the temperature dial.
how bright each thread glows.
the signal flows home.
that's it
dots, threads, a softmax.
everything else, the billions of parameters, the layers, the training, is this one move, repeated.