Skip to content
The Chat Archive
claude-3-5-sonnet-20241022· Anthropic LMArena Human Preference 140K

And 1.2T model?

  1. User

    And 1.2T model?

  2. Assistant

    I'll calculate the sizes for a 1.2 trillion (1.2T) parameter model using the same quantization levels.

    Calculations for a 1.2T Parameter Model:

    1. FP32 (32-bit float):

      • Size per parameter = 4 bytes
      • Total size = 1.2T parameters × 4 bytes = 4,800,000,000,000 bytes
      • Size in GB = 4,800,000,000,000 ÷ 1,073,741,824 ≈ 4,470 GB (≈ 4.47 TB)
    2. FP16 or BF16 (16-bit float):

      • Size per parameter = 2 bytes
      • Total size = 1.2T parameters × 2 bytes = 2,400,000,000,000 bytes
      • Size in GB = 2,400,000,000,000 ÷ 1,073,741,824 ≈ 2,235 GB (≈ 2.24 TB)
    3. INT8 (8-bit integer):

      • Size per parameter = 1 byte
      • Total size = 1.2T parameters × 1 byte = 1,200,000,000,000 bytes
      • Size in GB = 1,200,000,000,000 ÷ 1,073,741,824 ≈ 1,118 GB (≈ 1.12 TB)
    4. INT4 (4-bit integer):

      • Size per parameter = 0.5 bytes
      • Total size = 1.2T parameters × 0.5 bytes = 600,000,000,000 bytes
      • Size in GB = 600,000,000,000 ÷ 1,073,741,824 ≈ 559 GB (≈ 0.56 TB)

    Summary Table:

    QuantizationBits per ParameterSize per Parameter (Bytes)Total Size (GB)Total Size (TB)
    FP32324~4,470~4.47
    FP16/BF16162~2,235~2.24
    INT881~1,118~1.12
    INT440.5~559~0.56

    Note that these sizes are exactly double those of the 600B model, as 1.2T is twice 600B. The same considerations about real-world usage, training vs. inference, and compression techniques apply here as well.

  3. User

    Assume other bit quants are possible. What is the biggest quant that I can load in 700GB RAM(context excluded)

  4. Assistant

    Let me calculate the maximum parameters possible for different quantizations with 700GB RAM available.

    700GB = 700 * 1,073,741,824 bytes ≈ 751,619,277,800 bytes

    Formula: Max Parameters = Available Memory / Bytes per Parameter

    For different quantizations:

    • 2-bit: 751,619,277,800 / 0.25 ≈ 3.0T parameters
    • 3-bit: 751,619,277,800 / 0.375 ≈ 2.0T parameters
    • 4-bit: 751,619,277,800 / 0.5 ≈ 1.5T parameters
    • 5-bit: 751,619,277,800 / 0.625 ≈ 1.2T parameters
    • 6-bit: 751,619,277,800 / 0.75 ≈ 1.0T parameters
    • 8-bit: 751,619,277,800 / 1 ≈ 751B parameters
    • 16-bit: 751,619,277,800 / 2 ≈ 375B parameters
    • 32-bit: 751,619,277,800 / 4 ≈ 188B parameters

    So with 700GB RAM, you could theoretically load:

    • A 1.2T parameter model in 5-bit quantization
    • A 1.5T parameter model in 4-bit quantization
    • A 3.0T parameter model in 2-bit quantization

    Note: This is a theoretical calculation. In practice, you need:

    1. Additional memory for context/activations
    2. Some overhead for the system
    3. Consider stability/quality tradeoffs with lower bits
    4. Some quantization methods might require additional memory for quantization tables

    The most practical choice would likely be a 1.2T model in 5-bit quantization or a 1.5T model in 4-bit quantization, considering real-world constraints.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.