Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-reasoning· PerplexityHugging Face LMArena Search Arena 24K

What is the best opensource, one I can use on my server, coding large language model at…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.0-flash-grounding’s answer.
  1. User

    What is the best opensource, one I can use on my server, coding large language model at this time that can fit on an NVIDIA H100 GPU with 93 GB of VRAM. Also Include quantized models.

  2. Perplexity Assistant

    The best open-source LLMs that can run on an NVIDIA H100 GPU (80–93GB VRAM) while supporting quantization include Llama 3 70B, Mixtral 8x22B, and Qwen2 72B. These models balance performance and efficiency when optimized with 4-bit quantization, reducing VRAM usage by ~75% compared to full-precision models[6][8].

    Top Models and Quantization Performance

    ModelSize (Params)4-bit VRAM UsageKey Features
    Llama 3 70B70B35GBState-of-the-art reasoning, multilingual support[3][6]
    Mixtral 8x22B141B (active)45GBSparse MoE architecture, cost-efficient inference[3][5]
    Qwen2 72B72B36GBStrong coding/math capabilities, Apache 2.0 licensed[3][8]

    Recommended Deployment Frameworks

    1. OpenLLM

      • Serves quantized models via NVIDIA TensorRT-LLM optimizations
      • Achieves 300+ tokens/sec on H100 with 4-bit Llama 3[6][8]
      • Deploy with one command:
        openllm start llama3-70b --quantize int4
        
    2. vLLM

      • Implements PagedAttention for 2.3x faster throughput than baseline
      • Supports AWQ quantization for minimal accuracy loss[1][5]
    3. Ollama

      • Local-focused with simple CLI for rapid prototyping:
        ollama run llama3:70b-q4_0
        
      • Limited scalability (best for <50 req/sec)[5][8]

    VRAM Optimization Techniques

    • 4-bit GPTQ: Reduces 70B models to 35GB without performance loss[6]
    • FlashAttention-2: Cuts attention VRAM by 40% for 32k-token contexts[6]
    • Dynamic Batching: Enables 50+ concurrent sessions on single H100[6]

    For enterprise deployments, combine OpenLLM with Kubernetes via BentoML/Yatai for auto-scaling across GPU clusters[8]. The H100's tensor parallelism handles 70B models at 150 tokens/sec while serving multiple users[6][7].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.