Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-opus-4-20250514· AnthropicHugging Face LMArena Human Preference 140K

what is the best open source model to fine tune locally on chats about coding

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with MiniMaxminimax-m1’s answer.
  1. User

    what is the best open source model to fine tune locally on chats about coding

  2. Claude Assistant

    For fine-tuning locally on coding conversations, here are the top open source options:

    Llama 3.1 (8B or 70B) - Meta's latest models have shown strong coding performance and are popular for fine-tuning. The 8B version is manageable on consumer GPUs (24GB VRAM), while 70B requires more substantial hardware but offers better base performance.

    DeepSeek Coder V2 - Specifically designed for code, available in sizes from 1.3B to 33B parameters. The 6.7B and 16B versions offer a good balance of performance and hardware requirements. It's trained on a large corpus of code and has shown excellent results when fine-tuned on domain-specific coding tasks.

    CodeLlama - Meta's code-specific variant of Llama 2, available in 7B, 13B, and 34B sizes. While older than Llama 3.1, it's still effective and has extensive community support and fine-tuning guides.

    Mistral 7B - A strong general-purpose model that handles code well and is very efficient for its size. Easy to fine-tune on consumer hardware and has good instruction-following capabilities.

    For your specific use case of coding chats, I'd recommend starting with either Llama 3.1 8B or DeepSeek Coder 6.7B. Both can be fine-tuned using tools like:

    • Axolotl or Unsloth for efficient fine-tuning
    • LoRA/QLoRA to reduce memory requirements
    • Datasets formatted in a conversational structure (like ShareGPT format)

    The choice depends on your hardware constraints and whether you want a model specialized for code (DeepSeek) or a more general model that's also strong at coding (Llama 3.1).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.