"5M context and 83x72b paramaters, output modalities: VRChat Full-Body-Tracking, audio…
- User
"5M context and 83x72b paramaters, output modalities: VRChat Full-Body-Tracking, audio (stereo), text, images , VRChat facial tracking data, object and scene interaction data (like it can press buttons) input modalities: vision (in realtime allows it to see the other participants and respond), audio (stereo), text, images (videos included), discord custom emoji & stickers, tactile feel, context thoughts, taste & scents"
Can this model be considered absurdly large?
- Assistant
Yes, that model would be considered absurdly large for the following reasons:
- 83x72b Parameters: The "83 x 72b" (83 multiplied by 72 billion) parameters is the most significant factor. This means the model would have roughly 6 trillion parameters. State-of-the-art models like GPT-4 are rumored to have around 1.8 trillion parameters. A model with 6 trillion parameters would be significantly larger, potentially leading to massive computational costs for training and inference.
- Multimodal Complexity: Handling so many different input and output modalities simultaneously adds significant complexity to the model architecture and training process. It needs to learn intricate relationships between vision, audio, text, tactile information, and even taste/scents. Creating a model that effectively integrates and translates across all of these modalities is extremely challenging.
- Real-time Requirements: The ability to process visual information in real-time to respond to other participants further increases the computational demands and architectural complexity. Real-time processing with such a large model would require powerful hardware infrastructure.
- VRChat Integration: The integration with VRChat full-body tracking, facial tracking data, and object interaction adds another layer of complexity. It requires understanding the nuances of the VRChat environment, human avatar movements, and object interactions.
While AI model sizes continue to grow, a model with the specifications you provided is still far beyond current capabilities, making it an absurdly large and ambitious undertaking.
- User
Estimate the amount of VRAM needed to run it at 5M context.
- Assistant
Okay, let's break down the VRAM estimation for such a massive model with a 5M context length. This will be an approximation, but it gives you a sense of the scale:
1. Model Parameters:
- The model has 83 * 72 billion parameters = ~6 trillion parameters.
2. Precision:
- Let's assume the model uses FP16 (half-precision floating point) for the parameters. This is a common optimization to reduce VRAM usage. Each FP16 parameter takes 2 bytes.
3. Parameter VRAM:
- 6 trillion parameters * 2 bytes/parameter = 12 trillion bytes = 12 TB (Terabytes)
4. Context Length and Activation Memory:
- A 5M context length is very, very long. We need to estimate the VRAM required to store activations (intermediate calculations) during inference. This is where things get complicated, as the activation memory depends heavily on the model architecture.
- Let's assume, optimistically, that each token in the context requires 1 KB (kilobyte) of VRAM for activations. This is a simplification, but it will give us a lower bound. In reality, it's likely to be much more.
- 5 million tokens * 1 KB/token = 5 billion KB = 5 TB
5. Gradient Calculation (If Training):
- If you're training (not just inferencing), you'll need to store gradients for backpropagation. This typically requires the same amount of memory as the parameters themselves.
- Gradient VRAM: 12 TB (same as parameters)
6. Optimizer States (If Training):
- Optimizers like Adam often require storing additional state information for each parameter (e.g., momentum, variance). Let's assume the optimizer state requires 4 bytes per parameter.
- Optimizer State VRAM: 6 trillion parameters * 4 bytes/parameter = 24 trillion bytes = 24 TB
7. Total VRAM (Approximate Lower Bound):
- Inference (FP16): 12 TB (Parameters) + 5 TB (Activations) = 17 TB
- Training (FP16): 12 TB (Parameters) + 5 TB (Activations) + 12 TB (Gradients) + 24 TB (Optimizer State) = 53 TB
Important Considerations:
- Estimates are Lower Bounds: These are extremely rough estimates. The actual VRAM usage could be significantly higher depending on the specific model architecture, batch size, optimization techniques, and other factors.
- Hardware: There is currently no single GPU available with 17 TB or 53 TB of VRAM. You would need to use techniques like model parallelism and data parallelism to distribute the model across multiple GPUs or even multiple machines.
- Quantization: More aggressive quantization techniques (e.g., INT8, INT4) could reduce VRAM usage, but they also might impact model performance.
- Offloading: It might be possible to offload some layers or activations to CPU RAM or NVMe storage, but this would significantly slow down inference/training.
- Model Architecture: This calculation makes the large assumption about the architecture of the model. More modern transformers use techniques to reduce VRAM but still the calculation would remain in the same order of magnitude
- 5M context size: This is huge. Normally models do not go above a few hundreds of thousands context sizes.
Conclusion:
Running this model, especially for training, would require a massive and very expensive hardware infrastructure. It's currently beyond the capabilities of most organizations. Even inference would be extremely challenging without substantial optimization and distributed computing techniques. Even if such model can be fit, processing a prompt of 5 Million context sizes would take considerable amount of time.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.