Learn · Speed Hacks · KV Cache
KV Cache Memory Cost
In plain English
The longer the conversation gets, the more sticky notes the AI makes. Eventually, you run out of desk space! 🤯
What is actually going on
The size of the KV cache scales linearly with the context length, batch size, and the model's hidden dimension size. A large context window (e.g., 128k tokens) can require tens of gigabytes of RAM just for the cache alone.
Why it matters
Running Out of Memory (OOM) during long conversations is usually caused by the KV Cache outgrowing available VRAM, not the model weights.
Another way to think about it
A browser with 500 tabs open—eventually the computer runs out of memory. 💻
Check yourself
What causes the KV cache to grow in size?
- Running out of battery
- Generating more tokens in a conversation
- Downloading a new model
- Quantizing the model
Show answer
Generating more tokens in a conversation — Every new token generated adds new entries to the KV Cache.
The KV cache uses a fixed amount of memory that never changes.
- True
- False
Show answer
False — The KV cache grows linearly as the context length increases.
To reduce KV Cache size, modern architectures sometimes use:
- Grouped Query Attention (GQA)
- Watercooling
- More RGB lights
- Fewer training epochs
Show answer
Grouped Query Attention (GQA) — GQA shares key/value heads across multiple query heads, significantly shrinking the cache size.