Learn · Making Models Fit · Quantization Basics
What is Quantization?
In plain English
Quantization is like saving a huge 4K photo as a JPEG. It looks almost the same, but it takes up way less space on your phone! 📸
What is actually going on
Quantization maps high-precision floating point numbers (FP16/FP32) to lower precision formats (INT8, INT4). This drastically reduces the memory footprint and bandwidth required to run a model.
Why it matters
Without quantization, a 7B parameter model requires 14GB of VRAM (FP16). Quantized to 4-bit, it requires only 4GB, making it runnable on almost any laptop.
Another way to think about it
Using a smaller box to pack the exact same amount of stuff by folding the clothes tighter. 📦
Check yourself
What does quantization do to a model's weights?
- Increases their precision
- Reduces their precision to save memory
- Encrypts them
- Deletes random weights
Show answer
Reduces their precision to save memory — It converts weights from 16-bit to lower bits (like 4-bit) to reduce memory usage.
Quantization requires you to have more VRAM.
- True
- False
Show answer
False — The exact opposite! It drastically reduces VRAM requirements.
A model in FP16 (16-bit) uses roughly how much space per parameter?
- 1 byte
- 2 bytes
- 4 bytes
- 8 bytes
Show answer
2 bytes — 16 bits equals 2 bytes. So a 7B model needs ~14GB of memory.