Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.0-flash-grounding· GoogleHugging Face LMArena Search Arena 24K

What is the best opensource, one I can use on my server, coding large language model at…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-reasoning’s answer.
  1. User

    What is the best opensource, one I can use on my server, coding large language model at this time that can fit on an NVIDIA H100 GPU with 93 GB of VRAM. Also Include quantized models.

  2. Gemini Assistant

    It appears you're looking for the best open-source large language model (LLM) that can run on your NVIDIA H100 GPU with 93 GB of VRAM, including quantized versions. Here's a breakdown of factors to consider and some potential models:

    1[1]. Understanding VRAM Requirements

    • Rule of Thumb: A general guideline is that you need approximately 2GB of GPU memory for every 1 billion parameters in the model when using 16-bit precision (half precision)[1].

    • Quantization: Quantization reduces the memory footprint by using lower precision for the model's weights[1][2]. Common quantization levels include 8-bit and 4-bit[1].

    • Formula: To calculate the memory needed for a quantized model: M = (P x (Q/8)) x 1.2[1]

      • M = GPU memory (VRAM) in GB[3][4][5]
      • P = Number of parameters in billions[2]
      • Q = Number of bits used for quantization (e.g., 16, 8, or 4)[1]
      • 1.2 = 20% overhead for key-value caching and other tasks[1]

    2. Potential LLMs and Considerations

    • Models to Consider: Based on your VRAM, you should be able to run models with up to ~46B parameters at 16-bit precision. However, using quantization, you can fit much larger models.
      • Llama 3: Meta's Llama 3 comes in 8B and 70B parameter sizes[6]. The 8B model would easily fit, and a quantized 70B version might be possible.
      • Gemma 3: Gemma 3 models, including the 27B parameter version, are designed for efficient inference and can run on NVIDIA H100 GPUs. Quantized versions (int4) drastically reduce VRAM requirements[3].
      • DeepSeek: The DeepSeek models are also worth investigating. DeepSeek-R1 requires a multi-GPU setup, but distilled versions like DeepSeek-R1-Distill-Qwen-32B might be an option with quantization.

    3[7]. Quantization Formats and Tools

    • Common Formats: Quantized models often come in GGUF (CPU/macOS), AWQ, and GPTQ (GPU) formats[8].
    • Hugging Face: Tom Jobbins ("TheBloke" on Hugging Face) is a well-known provider of quantized model versions[8].
    • Inference Engines: Tools like TensorRT-LLM and vLLM are crucial for optimizing LLM inference on NVIDIA GPUs, offering features like quantization, in-flight batching, and paged attention.

    4. Benchmarking and Performance

    • NVIDIA H100 Performance: The H100 significantly outperforms the A100 for LLM inference due to its higher FLOPS and advanced Tensor Cores[9]. It's optimized for developing, training, and deploying LLMs[7].
    • TensorRT-LLM: Using TensorRT-LLM on the H100 can result in significant speedups compared to the A100.
    • vLLM: This open-source inference engine is designed for high throughput and low latency[10].

    5. Key Takeaways

    • Quantization is Essential: To fit larger models within your 93 GB VRAM, you'll almost certainly need to use quantization.
    • Experimentation is Key: The best model for you will depend on your specific needs and performance requirements. Experiment with different models and quantization levels to find the optimal balance.
    • Utilize Inference Engines: Employ tools like TensorRT-LLM or vLLM to maximize the performance of your chosen model on the H100.
    • Check VRAM Usage: Carefully monitor VRAM usage to ensure the model fits within your GPU's memory.

    By considering these factors and experimenting with different models and quantization techniques, you should be able to find an open-source LLM that performs well on your NVIDIA H100 GPU.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.