And 1.2T model?
- User
And 1.2T model?
- Assistant
I'll calculate the sizes for a 1.2 trillion (1.2T) parameter model using the same quantization levels.
Calculations for a 1.2T Parameter Model:
-
FP32 (32-bit float):
- Size per parameter = 4 bytes
- Total size = 1.2T parameters × 4 bytes = 4,800,000,000,000 bytes
- Size in GB = 4,800,000,000,000 ÷ 1,073,741,824 ≈ 4,470 GB (≈ 4.47 TB)
-
FP16 or BF16 (16-bit float):
- Size per parameter = 2 bytes
- Total size = 1.2T parameters × 2 bytes = 2,400,000,000,000 bytes
- Size in GB = 2,400,000,000,000 ÷ 1,073,741,824 ≈ 2,235 GB (≈ 2.24 TB)
-
INT8 (8-bit integer):
- Size per parameter = 1 byte
- Total size = 1.2T parameters × 1 byte = 1,200,000,000,000 bytes
- Size in GB = 1,200,000,000,000 ÷ 1,073,741,824 ≈ 1,118 GB (≈ 1.12 TB)
-
INT4 (4-bit integer):
- Size per parameter = 0.5 bytes
- Total size = 1.2T parameters × 0.5 bytes = 600,000,000,000 bytes
- Size in GB = 600,000,000,000 ÷ 1,073,741,824 ≈ 559 GB (≈ 0.56 TB)
Summary Table:
Quantization Bits per Parameter Size per Parameter (Bytes) Total Size (GB) Total Size (TB) FP32 32 4 ~4,470 ~4.47 FP16/BF16 16 2 ~2,235 ~2.24 INT8 8 1 ~1,118 ~1.12 INT4 4 0.5 ~559 ~0.56 Note that these sizes are exactly double those of the 600B model, as 1.2T is twice 600B. The same considerations about real-world usage, training vs. inference, and compression techniques apply here as well.
-
- User
Assume other bit quants are possible. What is the biggest quant that I can load in 700GB RAM(context excluded)
- Assistant
Let me calculate the maximum parameters possible for different quantizations with 700GB RAM available.
700GB = 700 * 1,073,741,824 bytes ≈ 751,619,277,800 bytes
Formula: Max Parameters = Available Memory / Bytes per Parameter
For different quantizations:
- 2-bit: 751,619,277,800 / 0.25 ≈ 3.0T parameters
- 3-bit: 751,619,277,800 / 0.375 ≈ 2.0T parameters
- 4-bit: 751,619,277,800 / 0.5 ≈ 1.5T parameters
- 5-bit: 751,619,277,800 / 0.625 ≈ 1.2T parameters
- 6-bit: 751,619,277,800 / 0.75 ≈ 1.0T parameters
- 8-bit: 751,619,277,800 / 1 ≈ 751B parameters
- 16-bit: 751,619,277,800 / 2 ≈ 375B parameters
- 32-bit: 751,619,277,800 / 4 ≈ 188B parameters
So with 700GB RAM, you could theoretically load:
- A 1.2T parameter model in 5-bit quantization
- A 1.5T parameter model in 4-bit quantization
- A 3.0T parameter model in 2-bit quantization
Note: This is a theoretical calculation. In practice, you need:
- Additional memory for context/activations
- Some overhead for the system
- Consider stability/quality tradeoffs with lower bits
- Some quantization methods might require additional memory for quantization tables
The most practical choice would likely be a 1.2T model in 5-bit quantization or a 1.5T model in 4-bit quantization, considering real-world constraints.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.