Untitled conversation
- User
{ "architectures": [ "DeepseekV3ForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "auto_map": { "AutoConfig": "configuration_deepseek.DeepseekV3Config", "AutoModel": "modeling_deepseek.DeepseekV3Model", "AutoModelForCausalLM": "modeling_deepseek.DeepseekV3ForCausalLM" }, "aux_loss_alpha": 0.001, "bos_token_id": 163584, "eos_token_id": 163585, "first_k_dense_replace": 1, "hidden_act": "silu", "hidden_size": 7168, "initializer_range": 0.02, "intermediate_size": 18432, "kv_lora_rank": 512, "max_position_embeddings": 131072, "model_type": "kimi_k2", "moe_intermediate_size": 2048, "moe_layer_freq": 1, "n_group": 1, "n_routed_experts": 384, "n_shared_experts": 1, "norm_topk_prob": true, "num_attention_heads": 64, "num_experts_per_tok": 8, "num_hidden_layers": 61, "num_key_value_heads": 64, "num_nextn_predict_layers": 0, "pretraining_tp": 1, "q_lora_rank": 1536, "qk_nope_head_dim": 128, "qk_rope_head_dim": 64, "quantization_config": { "activation_scheme": "dynamic", "fmt": "e4m3", "quant_method": "fp8", "weight_block_size": [ 128, 128 ] }, "rms_norm_eps": 1e-06, "rope_theta": 50000.0, "routed_scaling_factor": 2.827, "rope_scaling": { "beta_fast": 1.0, "beta_slow": 1.0, "factor": 32.0, "mscale": 1.0, "mscale_all_dim": 1.0, "original_max_position_embeddings": 4096, "type": "yarn" }, "scoring_func": "sigmoid", "seq_aux": true, "tie_word_embeddings": false, "topk_group": 1, "topk_method": "noaux_tc", "torch_dtype": "bfloat16", "transformers_version": "4.48.3", "use_cache": true, "v_head_dim": 128, "vocab_size": 163840 } 计算这个模型的总参数量
Assistant
1. 理解模型结构
从配置文件中,我们可以提取以下关键信息:
- 模型类型:
kimi_k2(类似DeepseekV3的MoE架构) - 隐藏层维度:
hidden_size = 7168 - 前馈网络(FFN)维度:
intermediate_size = 18432(用于非MoE层) - MoE层:
n_routed_experts = 384(路由专家数量)moe_intermediate_size = 2048(每个专家的FFN维度)num_experts_per_tok = 8(每个token激活的专家数量)n_shared_experts = 1(共享专家数量)
- 层数:
num_hidden_layers = 61 - 注意力头数:
num_attention_heads = 64 - 词汇表大小:
vocab_size = 163840 - 其他:
tie_word_embeddings = false(输入和输出嵌入不共享)
2. 参数量计算
模型的参数量主要由以下几部分组成:
- 嵌入层(Embedding)
- Transformer层(包括注意力层和FFN/MoE层)
- 输出层(LM Head)
2.1 嵌入层
- 输入嵌入矩阵:
vocab_size * hidden_size = 163840 * 7168 = 1,173,637,120
2.2 Transformer层
共有61层,其中:
-
注意力层(每层):
- Q/K/V投影:
hidden_size * (num_attention_heads * (qk_nope_head_dim + qk_rope_head_dim)) * 3qk_nope_head_dim = 128,qk_rope_head_dim = 64- 每个头的维度:
128 + 64 = 192 - 总维度:
64 * 192 = 12288 - Q/K/V投影:
7168 * 12288 * 3 = 264,241,152
- 输出投影:
hidden_size * hidden_size = 7168 * 7168 = 51,380,224 - 注意力层总参数量:
264,241,152 + 51,380,224 = 315,621,376
- Q/K/V投影:
-
FFN/MoE层:
- 非MoE层(
first_k_dense_replace = 1,即第1层是dense FFN):- 第1层是dense FFN,其余60层是MoE。
- Dense FFN参数量:
- 门控:
hidden_size * intermediate_size = 7168 * 18432 = 132,120,576 - 输出:
intermediate_size * hidden_size = 18432 * 7168 = 132,120,576 - 总参数量:
132,120,576 * 2 = 264,241,152
- 门控:
- MoE层(60层):
- 每个MoE层:
- 路由专家:
- 每个专家的FFN:
hidden_size * moe_intermediate_size + moe_intermediate_size * hidden_size = 7168 * 2048 * 2 = 29,360,128 - 384个专家的总参数量:
384 * 29,360,128 = 11,274,289,152 - 但每个token只激活8个专家,因此实际计算的是总参数量(存储所有专家的参数)。
- 每个专家的FFN:
- 共享专家:
- 共享专家的FFN:
hidden_size * moe_intermediate_size + moe_intermediate_size * hidden_size = 7168 * 2048 * 2 = 29,360,128
- 共享专家的FFN:
- 门控网络:
- 路由权重:
hidden_size * n_routed_experts = 7168 * 384 = 2,752,512
- 路由权重:
- 每个MoE层总参数量:
11,274,289,152 + 29,360,128 + 2,752,512 = 11,306,401,792
- 路由专家:
- 60层MoE总参数量:
60 * 11,306,401,792 = 678,384,107,520
- 每个MoE层:
- 非MoE层(
-
LayerNorm:
- 每层有2个LayerNorm(注意力后和FFN后):
- 每个LayerNorm:
hidden_size = 7168 - 总参数量:
61 * 2 * 7168 = 874,496
- 每个LayerNorm:
- 每层有2个LayerNorm(注意力后和FFN后):
2.3 输出层
- 输出投影(LM Head):
hidden_size * vocab_size = 7168 * 163840 = 1,173,637,120
3. 总参数量汇总
- 嵌入层:
1,173,637,120 - Transformer层:
- 注意力层:
61 * 315,621,376 = 19,252,903,936 - FFN/MoE层:
- Dense FFN(第1层):
264,241,152 - MoE层(60层):
678,384,107,520
- Dense FFN(第1层):
- LayerNorm:
874,496
- 注意力层:
- 输出层:
1,173,637,120
总参数量:
= 嵌入层 + 注意力层 + Dense FFN + MoE层 + LayerNorm + 输出层 = 1,173,637,120 + 19,252,903,936 + 264,241,152 + 678,384,107,520 + 874,496 + 1,173,637,120 = 700,249,401,3444. 验证
- 这是一个MoE模型,大部分参数来自MoE层的专家(384个专家,每个专家参数量较小)。
- 非MoE部分(嵌入、注意力、输出)的参数量约为
1.17B + 19.25B + 1.17B = 21.59B。 - MoE部分(60层)的参数量约为
678.38B。 - 总参数量约为
700.25B(7002.5亿参数)。
5. 最终答案
该模型的总参数量约为 7002.5亿(700.25B)。
- 模型类型:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.