Attention
The mechanism that lets a model weigh which earlier tokens matter for the next one
Architecture
In plain English
When you finish someone's sentence, you don't give every word they said equal importance — you focus on the few that matter. Attention is the model doing that, scoring every previous word for how much it should care.
Technically
For each token, three vectors are projected from its embedding: a query, a key and a value. The query is compared against the keys of all preceding tokens via scaled dot product, softmaxed into weights, and used to produce a weighted sum of the values. Cost grows with the square of sequence length, and the keys and values of earlier tokens must be retained for every subsequent token — which is what the KV cache stores and what makes long contexts expensive in memory. Most efficiency work in local inference is an attack on one of those two costs.