Skip to content
The Chat Archive
o3-2025-04-16· OpenAI LMArena Human Preference 140K

does fine tuning a llm model expand its parameters size?

  1. User

    does fine tuning a llm model expand its parameters size?

  2. Assistant

    Short answer
    • If you do a classic “full-parameter” fine-tune, the network architecture stays identical, so the parameter count does not grow—but you now need a second copy of those parameters on disk (the original + the tuned weights).
    • If you use a parameter-efficient fine-tuning (PEFT) method—LoRA, adapters, prefix-tuning, p-tuning, etc.—you add a small set of extra weights; the base model is frozen. In that case the parameter count grows a little (often <1 % of the base model), yet it is still far smaller than a full checkpoint.

    Details

    1. Full fine-tuning
      Architecture: unchanged
      Param count: unchanged
      Storage: you need ~1× model size for the new checkpoint.
      Runtime memory: unchanged at inference; training temporarily needs gradients & optimizer states (2–6× model size, but that’s transient).

    2. Parameter-efficient variants
      • LoRA: inserts two low-rank matrices per targeted weight tensor. Extra params = 2 × rank × tensor width (typically 0.1–3 % of base).
      • Adapter/IA³: small bottleneck MLP blocks; ~1–2 %.
      • Prefix/p-tuning: learnable “virtual tokens”; tens of thousands of parameters (≪1 %).
      • BitFit or bias-only: just biases; <0.1 %.

      You ship only these small deltas, so disk footprint = base model (public) + tiny addon (your weights).

    3. Quantization + fine-tune
      You can quantize the frozen base model (e.g., 4-bit) and fine-tune LoRA in 16-bit. Param count same story; memory savings come from lower precision.

    4. Training-time overhead
      No matter the scheme, during training you still allocate:
      • Gradients (same size as trainable parameters)
      • Optimizer states (Adam ⇒ 2× gradients)
      So RAM/VRAM can be 2-3× the trainable parameter subset.

    Takeaway
    • Number of parameters only “grows” if your fine-tuning technique explicitly inserts new layers/modules; classic full-tuning keeps the count the same.
    • Disk usage always grows (you have to store the new weights), but with PEFT methods that growth is typically just a few megabytes instead of many gigabytes.

  3. User

    what about pre training on a existing model?

  4. Assistant

    short version
    – if you just “keep pre-training” the same model (sometimes called continued pre-training, domain-adaptive pre-training, etc.) you’re still updating the very same weight tensors, so the parameter count does not grow. you only end up with a new checkpoint file that is the same size as the old one.
    – params only grow if you deliberately change the architecture: add new vocab tokens, extend absolute-position embeddings, tack on extra layers, switch to mixture-of-experts, drop in adapters/LoRA, etc.

    a bit more detail

    1. vanilla continued pre-training
      • architecture: unchanged
      • #parameters: unchanged
      • storage: need one more full checkpoint on disk (original + new)
      • training memory: you still carry gradients + optimizer state (≈ 2–3× trainable params)

    2. common tweaks that DO increase param count (all optional)
      • new tokenizer tokens
      – every extra token adds hidden_size params to the input-embedding matrix
      (e.g. 4096 hidden dim → one new token = 4 096 params ≈ 16 KB in fp32)
      • longer context if you use absolute positional embeddings
      – extending from 2 048 to 4 096 positions doubles that embedding table
      – rotary/ALiBi styles don’t need new params, so no growth there
      • adapters / LoRA during pre-training
      – you freeze the base model and only train the small inserts → +0.1–2 % params
      – saves memory and lets you release just the deltas
      • adding brand-new layers (e.g. a bigger FFN, MoE experts)
      – obviously increases size proportionally to what you add

    3. inference time
      – if you kept the architecture intact, running the model after pre-training needs the same VRAM / RAM as before
      – if you added tokens or longer position embeddings, it’s a tiny bump in RAM but linear in the new table size

    takeaway
    continuing pre-training is the same story as full fine-tuning: parameter count stays flat unless you explicitly bolt on extra pieces. the “cost” is just another copy of the weights on disk (or a diff file if you prefer).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.