Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-instant-1· AnthropicHugging Face Arena Human Preference 55K

How many V100 32GB would be needed to train a 70B model with the qlora method in 4-bit…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with vicuna-33b’s answer.
  1. User

    How many V100 32GB would be needed to train a 70B model with the qlora method in 4-bit NormalFloat (nf4) with 100K context window?

  2. Claude Assistant

    I do not have enough information to give an exact number of V100 GPUs needed. The resource requirements for training large models depend on many factors, including:

    • Size and architecture of the model: A larger model will generally require more compute.

    • Training batch size: Larger batch sizes can utilize more parallelism but may not fit in memory.

    • Data type and precision: Lower precision like 4-bit floats can reduce memory usage but may impact training speed.

    • Context window size: A larger window size means more input tokens to process per batch.

    • Algorithm and implementation: Different algorithms and framework optimizations can affect efficiency.

    As a rough estimate, training a 70B parameter model with a context window of 100K tokens using 4-bit floats (which reduces memory usage significantly) would likely require dozens or even over 100 V100 GPUs to train efficiently in a reasonable time frame (weeks rather than months). The exact number could vary a lot based on the implementation and hyperparameter tuning. Multi-node distributed training across many servers would almost certainly be needed.

    Without more details about the specific model architecture, implementation, and target timeframe, it's difficult to give an exact GPU count. The number of GPUs needed generally scales exponentially with model size for these very large pretrained models.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.