Skip to content
TheChatArchiveThe Chat Archive
Qwenqwq-32b· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

Untitled conversation

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-pro-preview-05-06’s answer.
  1. User
    {
      "architectures": [
        "Qwen3ForCausalLM"
      ],
      "attention_bias": false,
      "attention_dropout": 0.0,
      "bos_token_id": 151643,
      "eos_token_id": 151645,
      "head_dim": 128,
      "hidden_act": "silu",
      "hidden_size": 4096,
      "initializer_range": 0.02,
      "intermediate_size": 12288,
      "max_position_embeddings": 40960,
      "max_window_layers": 36,
      "model_type": "qwen3",
      "num_attention_heads": 32,
      "num_hidden_layers": 36,
      "num_key_value_heads": 8,
      "rms_norm_eps": 1e-06,
      "rope_scaling": null,
      "rope_theta": 1000000,
      "sliding_window": null,
      "tie_word_embeddings": false,
      "torch_dtype": "bfloat16",
      "transformers_version": "4.51.0",
      "use_cache": true,
      "use_sliding_window": false,
      "vocab_size": 151936
    }
    

    You are an expert in machine learning. Please estimate the parameter count for the model.

  2. Qwen Assistant

    The estimated parameter count for the Qwen3ForCausalLM model is approximately 7.3 billion parameters. Here's the breakdown of the calculation:

    Key Components and Parameters:

    1. Transformer Layers (36 layers):

      • Attention Parameters (per layer):
        • Query (Q), Key (K), Value (V), and Output matrices:
          ( 4 \times (4096 \times 4096) = 4 \times 16,777,216 = 67,108,864 )
        • Biases for Q, K, V, and Output:
          ( 4 \times 4096 = 16,384 )
        • Total Attention per layer: ( 67,108,864 + 16,384 = 67,125,248 )
      • Feed-Forward Network (FFN) Parameters (per layer):
        • First layer (hidden → intermediate):
          ( 4096 \times 12,288 = 50,331,648 )
        • Second layer (intermediate → hidden):
          ( 12,288 \times 4096 = 50,331,648 )
        • Biases for FFN:
          ( 12,288 + 4096 = 16,384 )
        • Total FFN per layer: ( 100,663,296 + 16,384 = 100,679,680 )
      • Layer Normalization (per layer):
        ( 2 \times 4096 = 8,192 ) (RMSNorm scales only, no biases).
      • Total Parameters per Layer:
        ( 67,125,248 + 100,679,680 + 8,192 = 167,813,120 )
      • Total for All Layers:
        ( 167,813,120 \times 36 \approx 6,041,272,320 )
    2. Embedding Layer:

      • Vocabulary size (( 151,936 )) × Hidden size (( 4096 )):
        ( 151,936 \times 4096 \approx 622,329,856 )
    3. Output Projection Layer:

      • Hidden size (( 4096 )) × Vocabulary size (( 151,936 )):
        ( 4096 \times 151,936 \approx 622,329,856 )
      • Bias for output layer:
        ( 151,936 )
    4. Final Layer Normalization:

      • ( 4096 ) (RMSNorm scale).

    Total Parameters:

    [ \begin{align*} \text{Total} &= \text{Layers} + \text{Embedding} + \text{Output} + \text{Final Norm} \ &= 6,041,272,320 + 622,329,856 + (622,329,856 + 151,936) + 4,096 \ &\approx 7.286 \text{ billion parameters}. \end{align*} ]

    Notes:

    • The model uses multi-query attention (32 query heads, 8 key/value heads), reducing parameters in the key/value projections.
    • The calculation assumes biases for all linear layers (Q, K, V, output, FFN).
    • RMSNorm parameters (only scales, no biases) are included in the layer norms.

    This estimate aligns with typical configurations for large LLMs like Qwen3, which are designed to balance efficiency and performance.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.