Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro· GoogleHugging Face LMArena Human Preference 140K

I didn't find any entry for SMOLLM or SmolVLM why aren't these model on the leaderboard?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Mistralmistral-medium-2505’s answer.
  1. User

    I didn't find any entry for SMOLLM or SmolVLM why aren't these model on the leaderboard?

  2. Gemini Assistant

    That's an excellent question, and it gets to the heart of what leaderboards are for versus what some projects are for.

    You won't find SMOLLM or SmolVLM on leaderboards like the Hugging Face Open LLM Leaderboard or Chatbot Arena for a simple reason: they are not formal, competitive models. They are educational projects.

    Let's break down what they actually are:

    SMOLLM (Small Language Model)

    The term "SMOLLM" is most famously associated with the work of Andrej Karpathy, a renowned AI researcher and educator (formerly at Tesla and OpenAI).

    • It's a Codebase, Not a Model: SMOLLM isn't a specific, pre-trained model you can download and use like Llama 3 or Phi-3. Instead, it refers to Karpathy's projects like llm.c and llama2.c.
    • The Goal is Education: The purpose of these projects is to teach people how LLMs work from the ground up. The code is intentionally simple, written in pure C with minimal dependencies, so that a developer can read, understand, and compile it in minutes. It's designed to demystify the "magic" of LLMs.
    • Not Trained for Performance: The models trained with this code are tiny and trained on small datasets (like Shakespeare's works). They are not designed to be good at general-purpose tasks or to compete on benchmarks. Their value is in their clarity and simplicity.

    SmolVLM (Small Vision-Language Model)

    Similarly, "SmolVLM" is the name Andrej Karpathy gave to his educational project on building a Vision-Language Model from scratch.

    • A Learning Recipe: This project shows you how to combine a pre-existing vision encoder (like CLIP) with a language model (like GPT-2) to create a model that can "see" an image and describe it in text.
    • Focus on Fundamentals: The goal is to teach the architectural principles behind models like GPT-4V or LLaVA. It's about understanding how the "vision" and "language" parts connect and are trained together.
    • Not a Competitor: Again, the resulting model is not optimized for state-of-the-art performance and isn't intended to be benchmarked against production-grade VLMs.

    Why This Means No Leaderboard Presence

    Leaderboards exist to rank and compare models based on their performance on standardized benchmarks. For a model to be on a leaderboard, it generally needs to be:

    1. A Specific, Released Artifact: A downloadable set of weights that everyone can test (e.g., meta-llama/Llama-3-8B-Instruct).
    2. Trained for Performance: The model was created with the intention of being powerful, useful, and competitive.
    3. Submitted for Evaluation: The creators or the community formally submit the model to be evaluated on the leaderboard's benchmarks.

    SMOLLM and SmolVLM are educational recipes, not competitive dishes. Putting them on a leaderboard would be like entering a "Learn to Cook" class project in a Michelin-star competition—it completely misses the point of the project.

    What to Look for Instead

    If you are interested in small, efficient models that are on the leaderboards, you should look for names like:

    • Phi-3: Microsoft's family of powerful small models (e.g., Phi-3-mini).
    • Gemma: Google's open models (e.g., Gemma 2B and 7B).
    • Llama 3 8B: Meta's smallest model in the Llama 3 family.
    • Mistral 7B: A highly capable model from Mistral AI that set a new standard for its size.

    These are the models that are actually designed for high performance in a compact size and are actively competing on the leaderboards.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.