Multi-Token Prediction (MTP)
Training models to predict several future tokens at once
Architecture
In plain English
Training a writer to plan 2 or 3 words ahead in their mind instead of just guessing the next single letter.
Technically
An architectural extension (Gloeckle et al., Meta 2024 / DeepSeek-V3) where independent output heads predict n future tokens simultaneously at each layer. Acts as a built-in speculative draft mechanism, boosting training sample efficiency and inference speed.