does fine tuning a llm model expand its parameters size?
- User
does fine tuning a llm model expand its parameters size?
- Assistant
Short answer
• If you do a classic “full-parameter” fine-tune, the network architecture stays identical, so the parameter count does not grow—but you now need a second copy of those parameters on disk (the original + the tuned weights).
• If you use a parameter-efficient fine-tuning (PEFT) method—LoRA, adapters, prefix-tuning, p-tuning, etc.—you add a small set of extra weights; the base model is frozen. In that case the parameter count grows a little (often <1 % of the base model), yet it is still far smaller than a full checkpoint.Details
-
Full fine-tuning
Architecture: unchanged
Param count: unchanged
Storage: you need ~1× model size for the new checkpoint.
Runtime memory: unchanged at inference; training temporarily needs gradients & optimizer states (2–6× model size, but that’s transient). -
Parameter-efficient variants
• LoRA: inserts two low-rank matrices per targeted weight tensor. Extra params = 2 × rank × tensor width (typically 0.1–3 % of base).
• Adapter/IA³: small bottleneck MLP blocks; ~1–2 %.
• Prefix/p-tuning: learnable “virtual tokens”; tens of thousands of parameters (≪1 %).
• BitFit or bias-only: just biases; <0.1 %.You ship only these small deltas, so disk footprint = base model (public) + tiny addon (your weights).
-
Quantization + fine-tune
You can quantize the frozen base model (e.g., 4-bit) and fine-tune LoRA in 16-bit. Param count same story; memory savings come from lower precision. -
Training-time overhead
No matter the scheme, during training you still allocate:
• Gradients (same size as trainable parameters)
• Optimizer states (Adam ⇒ 2× gradients)
So RAM/VRAM can be 2-3× the trainable parameter subset.
Takeaway
• Number of parameters only “grows” if your fine-tuning technique explicitly inserts new layers/modules; classic full-tuning keeps the count the same.
• Disk usage always grows (you have to store the new weights), but with PEFT methods that growth is typically just a few megabytes instead of many gigabytes. -
- User
what about pre training on a existing model?
- Assistant
short version
– if you just “keep pre-training” the same model (sometimes called continued pre-training, domain-adaptive pre-training, etc.) you’re still updating the very same weight tensors, so the parameter count does not grow. you only end up with a new checkpoint file that is the same size as the old one.
– params only grow if you deliberately change the architecture: add new vocab tokens, extend absolute-position embeddings, tack on extra layers, switch to mixture-of-experts, drop in adapters/LoRA, etc.a bit more detail
-
vanilla continued pre-training
• architecture: unchanged
• #parameters: unchanged
• storage: need one more full checkpoint on disk (original + new)
• training memory: you still carry gradients + optimizer state (≈ 2–3× trainable params) -
common tweaks that DO increase param count (all optional)
• new tokenizer tokens
– every extra token adds hidden_size params to the input-embedding matrix
(e.g. 4096 hidden dim → one new token = 4 096 params ≈ 16 KB in fp32)
• longer context if you use absolute positional embeddings
– extending from 2 048 to 4 096 positions doubles that embedding table
– rotary/ALiBi styles don’t need new params, so no growth there
• adapters / LoRA during pre-training
– you freeze the base model and only train the small inserts → +0.1–2 % params
– saves memory and lets you release just the deltas
• adding brand-new layers (e.g. a bigger FFN, MoE experts)
– obviously increases size proportionally to what you add -
inference time
– if you kept the architecture intact, running the model after pre-training needs the same VRAM / RAM as before
– if you added tokens or longer position embeddings, it’s a tiny bump in RAM but linear in the new table size
takeaway
continuing pre-training is the same story as full fine-tuning: parameter count stays flat unless you explicitly bolt on extra pieces. the “cost” is just another copy of the weights on disk (or a diff file if you prefer). -
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.