What is the purpose of MTP module in LLMs?
- User
What is the purpose of MTP module in LLMs?
- Assistant
MTP = Masked-Token-Prediction head
(You’ll also see it called “MLM head” for Masked-Language-Modeling.)
Why it exists
- During pre-training we randomly hide (mask) ~15 % of the input tokens.
- The transformer backbone produces a contextual embedding for every position.
- The MTP module is a small classifier that sits on top of those embeddings and tries to recover the original, hidden token.
• Technically it is just a linear projection W ∈ R(hidden_size × vocab_size) + bias, followed by soft-max and cross-entropy loss, evaluated only on the masked positions.
What this buys us
• Forces the model to build rich, bidirectional representations (it must look left and right to guess the missing word).
• Gives a self-supervised objective, so the model can learn from raw text without human labels.
• After this pre-training, the MTP head can be thrown away or reused; the backbone weights are what matter for fine-tuning or for initializing larger generative models.Where you’ll meet it
• Encoder-only LLMs (BERT, RoBERTa, DeBERTa, ALBERT, etc.).
• Multi-task setups that combine MTP with other pre-training objectives (e.g., Next-Sentence-Prediction, SOP, contrastive losses).
• Codebases often name the file/module “MTPHead”, “MLMHead”, or similar.So, the “MTP module” isn’t some exotic new block—it's simply the vocabulary-projection layer whose sole purpose is to compute the masked-token-prediction loss that teaches the large language model its basic knowledge of language.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.