What is the best opensource, one I can use on my server, coding large language model at…
- User
What is the best opensource, one I can use on my server, coding large language model at this time that can fit on an NVIDIA H100 GPU with 93 GB of VRAM. Also Include quantized models.
Assistant
The best open-source LLMs that can run on an NVIDIA H100 GPU (80–93GB VRAM) while supporting quantization include Llama 3 70B, Mixtral 8x22B, and Qwen2 72B. These models balance performance and efficiency when optimized with 4-bit quantization, reducing VRAM usage by ~75% compared to full-precision models[6][8].
Top Models and Quantization Performance
Model Size (Params) 4-bit VRAM Usage Key Features Llama 3 70B 70B 35GB State-of-the-art reasoning, multilingual support[3][6] Mixtral 8x22B 141B (active) 45GB Sparse MoE architecture, cost-efficient inference[3][5] Qwen2 72B 72B 36GB Strong coding/math capabilities, Apache 2.0 licensed[3][8] Recommended Deployment Frameworks
-
OpenLLM
- Serves quantized models via NVIDIA TensorRT-LLM optimizations
- Achieves 300+ tokens/sec on H100 with 4-bit Llama 3[6][8]
- Deploy with one command:
openllm start llama3-70b --quantize int4
-
vLLM
- Implements PagedAttention for 2.3x faster throughput than baseline
- Supports AWQ quantization for minimal accuracy loss[1][5]
-
Ollama
- Local-focused with simple CLI for rapid prototyping:
ollama run llama3:70b-q4_0 - Limited scalability (best for <50 req/sec)[5][8]
- Local-focused with simple CLI for rapid prototyping:
VRAM Optimization Techniques
- 4-bit GPTQ: Reduces 70B models to 35GB without performance loss[6]
- FlashAttention-2: Cuts attention VRAM by 40% for 32k-token contexts[6]
- Dynamic Batching: Enables 50+ concurrent sessions on single H100[6]
For enterprise deployments, combine OpenLLM with Kubernetes via BentoML/Yatai for auto-scaling across GPU clusters[8]. The H100's tensor parallelism handles 70B models at 150 tokens/sec while serving multiple users[6][7].
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.