Sliding Window Attention (SWA)
Restricting attention to a fixed local token window to save memory
Architecture
In plain English
Reading a 1000-page book by focusing your eyes on the current paragraph while remembering earlier chapters through summary chapters.
Technically
An attention pattern (used in Gemma 2 and Mistral) where tokens only attend to the preceding W tokens (e.g. W=4096) in alternating layers. Bounds per-layer KV cache memory to O(W) rather than O(N), enabling high throughput over long contexts.