Skip to content
TheChatArchiveThe Chat Archive
Kimikimi-k2-0711-preview· Moonshot AIHugging Face LMArena Human Preference 140K

Untitled conversation

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.0-flash-001’s answer.
  1. User

    { "architectures": [ "DeepseekV3ForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "auto_map": { "AutoConfig": "configuration_deepseek.DeepseekV3Config", "AutoModel": "modeling_deepseek.DeepseekV3Model", "AutoModelForCausalLM": "modeling_deepseek.DeepseekV3ForCausalLM" }, "aux_loss_alpha": 0.001, "bos_token_id": 163584, "eos_token_id": 163585, "first_k_dense_replace": 1, "hidden_act": "silu", "hidden_size": 7168, "initializer_range": 0.02, "intermediate_size": 18432, "kv_lora_rank": 512, "max_position_embeddings": 131072, "model_type": "kimi_k2", "moe_intermediate_size": 2048, "moe_layer_freq": 1, "n_group": 1, "n_routed_experts": 384, "n_shared_experts": 1, "norm_topk_prob": true, "num_attention_heads": 64, "num_experts_per_tok": 8, "num_hidden_layers": 61, "num_key_value_heads": 64, "num_nextn_predict_layers": 0, "pretraining_tp": 1, "q_lora_rank": 1536, "qk_nope_head_dim": 128, "qk_rope_head_dim": 64, "quantization_config": { "activation_scheme": "dynamic", "fmt": "e4m3", "quant_method": "fp8", "weight_block_size": [ 128, 128 ] }, "rms_norm_eps": 1e-06, "rope_theta": 50000.0, "routed_scaling_factor": 2.827, "rope_scaling": { "beta_fast": 1.0, "beta_slow": 1.0, "factor": 32.0, "mscale": 1.0, "mscale_all_dim": 1.0, "original_max_position_embeddings": 4096, "type": "yarn" }, "scoring_func": "sigmoid", "seq_aux": true, "tie_word_embeddings": false, "topk_group": 1, "topk_method": "noaux_tc", "torch_dtype": "bfloat16", "transformers_version": "4.48.3", "use_cache": true, "v_head_dim": 128, "vocab_size": 163840 } 计算这个模型的总参数量

  2. Kimi Assistant

    1. 理解模型结构

    从配置文件中,我们可以提取以下关键信息:

    • 模型类型: kimi_k2(类似DeepseekV3的MoE架构)
    • 隐藏层维度: hidden_size = 7168
    • 前馈网络(FFN)维度: intermediate_size = 18432(用于非MoE层)
    • MoE层:
      • n_routed_experts = 384(路由专家数量)
      • moe_intermediate_size = 2048(每个专家的FFN维度)
      • num_experts_per_tok = 8(每个token激活的专家数量)
      • n_shared_experts = 1(共享专家数量)
    • 层数: num_hidden_layers = 61
    • 注意力头数: num_attention_heads = 64
    • 词汇表大小: vocab_size = 163840
    • 其他: tie_word_embeddings = false(输入和输出嵌入不共享)

    2. 参数量计算

    模型的参数量主要由以下几部分组成:

    1. 嵌入层(Embedding)
    2. Transformer层(包括注意力层和FFN/MoE层)
    3. 输出层(LM Head)

    2.1 嵌入层

    • 输入嵌入矩阵: vocab_size * hidden_size = 163840 * 7168 = 1,173,637,120

    2.2 Transformer层

    共有61层,其中:

    • 注意力层(每层):

      • Q/K/V投影: hidden_size * (num_attention_heads * (qk_nope_head_dim + qk_rope_head_dim)) * 3
        • qk_nope_head_dim = 128, qk_rope_head_dim = 64
        • 每个头的维度: 128 + 64 = 192
        • 总维度: 64 * 192 = 12288
        • Q/K/V投影: 7168 * 12288 * 3 = 264,241,152
      • 输出投影: hidden_size * hidden_size = 7168 * 7168 = 51,380,224
      • 注意力层总参数量: 264,241,152 + 51,380,224 = 315,621,376
    • FFN/MoE层:

      • 非MoE层(first_k_dense_replace = 1,即第1层是dense FFN):
        • 第1层是dense FFN,其余60层是MoE。
        • Dense FFN参数量:
          • 门控: hidden_size * intermediate_size = 7168 * 18432 = 132,120,576
          • 输出: intermediate_size * hidden_size = 18432 * 7168 = 132,120,576
          • 总参数量: 132,120,576 * 2 = 264,241,152
      • MoE层(60层):
        • 每个MoE层:
          • 路由专家:
            • 每个专家的FFN: hidden_size * moe_intermediate_size + moe_intermediate_size * hidden_size = 7168 * 2048 * 2 = 29,360,128
            • 384个专家的总参数量: 384 * 29,360,128 = 11,274,289,152
            • 但每个token只激活8个专家,因此实际计算的是总参数量(存储所有专家的参数)。
          • 共享专家:
            • 共享专家的FFN: hidden_size * moe_intermediate_size + moe_intermediate_size * hidden_size = 7168 * 2048 * 2 = 29,360,128
          • 门控网络:
            • 路由权重: hidden_size * n_routed_experts = 7168 * 384 = 2,752,512
          • 每个MoE层总参数量: 11,274,289,152 + 29,360,128 + 2,752,512 = 11,306,401,792
        • 60层MoE总参数量: 60 * 11,306,401,792 = 678,384,107,520
    • LayerNorm:

      • 每层有2个LayerNorm(注意力后和FFN后):
        • 每个LayerNorm: hidden_size = 7168
        • 总参数量: 61 * 2 * 7168 = 874,496

    2.3 输出层

    • 输出投影(LM Head): hidden_size * vocab_size = 7168 * 163840 = 1,173,637,120

    3. 总参数量汇总

    • 嵌入层: 1,173,637,120
    • Transformer层:
      • 注意力层: 61 * 315,621,376 = 19,252,903,936
      • FFN/MoE层:
        • Dense FFN(第1层): 264,241,152
        • MoE层(60层): 678,384,107,520
      • LayerNorm: 874,496
    • 输出层: 1,173,637,120

    总参数量:

    = 嵌入层 + 注意力层 + Dense FFN + MoE层 + LayerNorm + 输出层
    = 1,173,637,120 + 19,252,903,936 + 264,241,152 + 678,384,107,520 + 874,496 + 1,173,637,120
    = 700,249,401,344
    

    4. 验证

    • 这是一个MoE模型,大部分参数来自MoE层的专家(384个专家,每个专家参数量较小)。
    • 非MoE部分(嵌入、注意力、输出)的参数量约为 1.17B + 19.25B + 1.17B = 21.59B。
    • MoE部分(60层)的参数量约为 678.38B。
    • 总参数量约为 700.25B(7002.5亿参数)。

    5. 最终答案

    该模型的总参数量约为 7002.5亿(700.25B)。

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.