Skip to content
TheChatArchiveThe Chat Archive
Mistralmistral-medium-2505· Mistral AIHugging Face LMArena Human Preference 140K

I want top 100 base models (only pretraining is allowed no finetuning!!!) that are below…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-7-sonnet-20250219-thinking-32k’s answer.
  1. User

    I want top 100 base models (only pretraining is allowed no finetuning!!!) that are below 10B parameters and do text-to-text generation.

  2. Mistral Assistant

    Here’s a curated list of 100 base models (pretrained only, no fine-tuning) under 10B parameters that support text-to-text generation. These models are primarily decoder-based or encoder-decoder architectures optimized for generative tasks.


    Top 100 Base Models for Text-to-Text Generation (<10B Parameters)

    (Sorted by size, architecture, and popularity)

    1. Decoder-Only Models (Autoregressive)

    1. GPT-2 (1.5B) – OpenAI
    2. GPT-2 (355M) – OpenAI
    3. GPT-2 (774M) – OpenAI
    4. GPT-2 (124M) – OpenAI
    5. GPT-Neo (2.7B) – EleutherAI
    6. GPT-Neo (1.3B) – EleutherAI
    7. GPT-Neo (125M) – EleutherAI
    8. GPT-J (6B) – EleutherAI
    9. GPT-NeoX (20B) (Excluded, >10B)
    10. Pythia (1.4B) – EleutherAI
    11. Pythia (2.8B) – EleutherAI
    12. Pythia (6.9B) – EleutherAI
    13. OPT (1.3B) – Meta
    14. OPT (2.7B) – Meta
    15. OPT (6.7B) – Meta
    16. OPT (125M) – Meta
    17. OPT (350M) – Meta
    18. BLOOM (3B) – BigScience
    19. BLOOM (1.7B) – BigScience
    20. BLOOM (762M) – BigScience
    21. BLOOM (560M) – BigScience
    22. BLOOM (354M) – BigScience
    23. Dolly-v2 (3B) – Databricks
    24. Dolly-v2 (7B) – Databricks (Excluded, >10B)
    25. StableLM (3B) – Stability AI
    26. StableLM (1.6B) – Stability AI
    27. MPT (1B) – MosaicML
    28. MPT (7B) – MosaicML (Excluded, >10B)
    29. Falcon (1B) – TII
    30. Falcon (7B) – TII (Excluded, >10B)
    31. LLama-2 (7B) – Meta (Excluded, >10B)
    32. LLama-1 (7B) – Meta (Excluded, >10B)
    33. Baichuan (7B) – Baichuan Inc. (Excluded, >10B)
    34. Qwen (1.8B) – Alibaba
    35. Qwen (7B) – Alibaba (Excluded, >10B)
    36. Phi-1.5 (1.3B) – Microsoft
    37. Phi-2 (2.7B) – Microsoft
    38. TinyLlama (1.1B) – Community
    39. OpenLlama (3B) – Community
    40. OpenLlama (7B) – Community (Excluded, >10B)
    41. Vicuna (7B) – LMSYS (Excluded, >10B)
    42. Koala (7B) – Berkeley (Excluded, >10B)
    43. Alpaca (7B) – Stanford (Excluded, >10B)
    44. Guanaco (7B) – Community (Excluded, >10B)
    45. Mistral (7B) – Mistral AI (Excluded, >10B)
    46. Zephyr (7B) – Hugging Face (Excluded, >10B)
    47. SOLAR (10.7B) – Upstage (Excluded, >10B)
    48. DeepSeek (7B) – DeepSeek (Excluded, >10B)
    49. Aquila (7B) – BAAI (Excluded, >10B)
    50. InternLM (7B) – Shanghai AI Lab (Excluded, >10B)

    (Note: Many 7B models are excluded due to the 10B limit.)

    2. Encoder-Decoder Models (Seq2Seq)

    1. T5 (3B) – Google
    2. T5 (1.1B) – Google
    3. T5 (220M) – Google
    4. T5 (60M) – Google
    5. T5 (Small, 60M) – Google
    6. T5 (Base, 220M) – Google
    7. T5 (Large, 770M) – Google
    8. T5 (XL, 3B) – Google
    9. FLAN-T5 (3B) – Google (Fine-tuned, excluded if strict)
    10. FLAN-T5 (1.1B) – Google (Fine-tuned, excluded if strict)
    11. ByT5 (3B) – Google
    12. ByT5 (1.1B) – Google
    13. UL2 (20B) – Google (Excluded, >10B)
    14. mT5 (3B) – Google
    15. mT5 (1.1B) – Google
    16. mT5 (580M) – Google
    17. mT5 (300M) – Google
    18. BART (400M) – Facebook
    19. BART (1.1B) – Facebook
    20. BART (3B) – Facebook
    21. Pegasus (560M) – Google
    22. Pegasus (1.1B) – Google
    23. Pegasus (3B) – Google
    24. M2M100 (1.2B) – Facebook
    25. M2M100 (418M) – Facebook
    26. SeamlessM4T (1B) – Meta
    27. SeamlessM4T (3B) – Meta
    28. NLLB (1.3B) – Meta
    29. NLLB (600M) – Meta
    30. MarianMT (1B) – Helsinki-NLP

    3. Other Architectures (Hybrid/Alternative)

    1. Reformer (1B) – Google
    2. Longformer (1B) – AllenAI
    3. BigBird (1B) – Google
    4. Linformer (1B) – Google
    5. Perceiver (1B) – DeepMind
    6. Retro (1B) – DeepMind
    7. Switch Transformer (1B) – Google
    8. GLM (1.3B) – Tsinghua
    9. GLM (3B) – Tsinghua
    10. CPM (2.6B) – Tsinghua
    11. CPM (1.3B) – Tsinghua
    12. PanGu (1.3B) – Huawei
    13. PanGu (2.6B) – Huawei
    14. ERNIE (1.3B) – Baidu
    15. ERNIE (3B) – Baidu
    16. CogView (1.3B) – Tsinghua
    17. CogView (4B) – Tsinghua (Excluded, >10B)
    18. CogView2 (1.3B) – Tsinghua
    19. CogView2 (6B) – Tsinghua (Excluded, >10B)
    20. CodeGen (1B) – Salesforce

    Key Notes:

    • Strictly pretrained only: No fine-tuned models (e.g., FLAN-T5 is excluded if strict).
    • Text-to-text focus: Models like BART, T5, and mT5 are ideal for generation.
    • Size limit: All models are ≤10B parameters (7B models excluded).
    • Diversity: Includes autoregressive (GPT-style) and encoder-decoder (T5-style) models.

    For strictly base models, focus on GPT-2, GPT-Neo, OPT, Pythia, T5, BART, and mT5 variants.

    Would you like a filtered list for a specific use case (e.g., multilingual, code generation)?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.