{ "batchsize": 18, "numepochs": 20, "lr": 5e-5, "seqlen": 256, "dmodel": 256, "N": 4,…
- User
{ "batch_size": 18, "num_epochs": 20, "lr": 5e-5, "seq_len": 256, "d_model": 256, "N": 4, "h": 4, "train": "dataset/openweb_fine.jsonl", "test": "dataset/openweb_fine.jsonl", "d_ff": 1024, "dropout": 0.1, "model_folder": "openweb2", "model_basename": "openweb2-", "preload": "05", "tokenizer_file": "openweb2.tokenizer.json", "experiment_name": "runs/openweb2", "dataset": "dataset/dataset_general.jsonl", "loss_file": "openweb2/losses.jsonl", "fine_dataset": "dataset/fine_tune.jsonl", "fine_epochs": 0 }
calculate the number of parameters in this model ? also suggest a config for 500M+ math solving llm
- Assistant
Let's break down your model config and estimate the number of parameters. Then, I'll suggest a config for a 500M+ parameter math-solving LLM.
1. Parameter Calculation for Your Model
Config:
d_model: 256 (embedding size)N: 4 (number of transformer layers)h: 4 (number of attention heads)d_ff: 1024 (feedforward hidden size)- Assume vocab size: Not specified, but let's assume 50,000 (typical for English LLMs)
- Ignore embedding tying and layer norm params for simplicity
A. Embedding Layer
- Token embedding:
vocab_size * d_model= 50,000 × 256 = 12,800,000
B. Transformer Block (per layer)
Each block has:
-
Multi-Head Attention:
- Q, K, V: Each is
d_model × d_model(for all heads together) - Output projection:
d_model × d_model - Total: 3 × (256 × 256) + (256 × 256) = 4 × 65,536 = 262,144
- Q, K, V: Each is
-
Feedforward:
- First linear:
d_model × d_ff= 256 × 1024 = 262,144 - Second linear:
d_ff × d_model= 1024 × 256 = 262,144 - Total: 262,144 + 262,144 = 524,288
- First linear:
-
LayerNorms: 2 per block, each with
d_modelparams: 2 × 256 = 512 -
Total per block:
262,144 (attn) + 524,288 (ffn) + 512 (ln) = 786,944 -
Total for N=4 layers:
4 × 786,944 = 3,147,776
C. Output Layer
- Output projection:
d_model × vocab_size= 256 × 50,000 = 12,800,000
D. Grand Total
- Embedding: 12,800,000
- Transformer blocks: 3,147,776
- Output: 12,800,000
Total ≈ 28,747,776 parameters (~28.7M)
2. Config for a 500M+ Parameter Math LLM
How to scale up:
- Increase
d_model,N,d_ff, and possibly vocab size. - Example: GPT-2 Medium (345M) uses d_model=1024, N=24, d_ff=4096, vocab=50,257.
- For 500M+, use d_model=1280, N=24, d_ff=5120, vocab=50,000.
Example Config:
{ "batch_size": 32, "num_epochs": 20, "lr": 5e-5, "seq_len": 1024, "d_model": 1280, "N": 24, "h": 20, "d_ff": 5120, "dropout": 0.1, "vocab_size": 50000, "model_folder": "mathllm500m", "model_basename": "mathllm500m-", "tokenizer_file": "mathllm.tokenizer.json", "experiment_name": "runs/mathllm500m" }Parameter Estimate for This Config:
- Embedding: 50,000 × 1280 = 64,000,000
- Each block:
- Attn: 4 × (1280 × 1280) = 4 × 1,638,400 = 6,553,600
- FFN: 1280 × 5120 + 5120 × 1280 = 2 × 6,553,600 = 13,107,200
- LayerNorm: 2 × 1280 = 2,560
- Total per block: 6,553,600 + 13,107,200 + 2,560 = 19,663,360
- 24 blocks: 24 × 19,663,360 = 471,920,640
- Output: 1280 × 50,000 = 64,000,000
Total ≈ 64M + 471.9M + 64M = 599.9M parameters
Summary Table
Model Size d_model N (layers) d_ff Heads Vocab Params (approx) Your Model 256 4 1024 4 50k 28.7M 500M LLM 1280 24 5120 20 50k 600M
Let me know if you want a config for a different size or more details on math-specific pretraining!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.