writing

notes from muhib

essays on the things i build and the way i try to build them: transformers, fast systems, and the faith underneath all of it.

essay · interactive · 9 min attention is all you need, in my words the transformer explained end to end through one image, a spider feeling its web. everything defined in plain words before the jargon, and built so you could code it from scratch. comes with a 3d visualization you can pull on. essay · interactive · 8 min flash attention the same attention, made fast. why the bottleneck is memory traffic not math, and how flashattention streams the scores through on-chip memory in tiles with an online softmax. exact, not approximate, with the numpy to prove it.