I want top 100 base models (only pretraining is allowed no finetuning!!!) that are below…
- User
I want top 100 base models (only pretraining is allowed no finetuning!!!) that are below 10B parameters and do text-to-text generation.
Assistant
Here’s a curated list of 100 base models (pretrained only, no fine-tuning) under 10B parameters that support text-to-text generation. These models are primarily decoder-based or encoder-decoder architectures optimized for generative tasks.
Top 100 Base Models for Text-to-Text Generation (<10B Parameters)
(Sorted by size, architecture, and popularity)
1. Decoder-Only Models (Autoregressive)
- GPT-2 (1.5B) – OpenAI
- GPT-2 (355M) – OpenAI
- GPT-2 (774M) – OpenAI
- GPT-2 (124M) – OpenAI
- GPT-Neo (2.7B) – EleutherAI
- GPT-Neo (1.3B) – EleutherAI
- GPT-Neo (125M) – EleutherAI
- GPT-J (6B) – EleutherAI
- GPT-NeoX (20B) (Excluded, >10B)
- Pythia (1.4B) – EleutherAI
- Pythia (2.8B) – EleutherAI
- Pythia (6.9B) – EleutherAI
- OPT (1.3B) – Meta
- OPT (2.7B) – Meta
- OPT (6.7B) – Meta
- OPT (125M) – Meta
- OPT (350M) – Meta
- BLOOM (3B) – BigScience
- BLOOM (1.7B) – BigScience
- BLOOM (762M) – BigScience
- BLOOM (560M) – BigScience
- BLOOM (354M) – BigScience
- Dolly-v2 (3B) – Databricks
- Dolly-v2 (7B) – Databricks (Excluded, >10B)
- StableLM (3B) – Stability AI
- StableLM (1.6B) – Stability AI
- MPT (1B) – MosaicML
- MPT (7B) – MosaicML (Excluded, >10B)
- Falcon (1B) – TII
- Falcon (7B) – TII (Excluded, >10B)
- LLama-2 (7B) – Meta (Excluded, >10B)
- LLama-1 (7B) – Meta (Excluded, >10B)
- Baichuan (7B) – Baichuan Inc. (Excluded, >10B)
- Qwen (1.8B) – Alibaba
- Qwen (7B) – Alibaba (Excluded, >10B)
- Phi-1.5 (1.3B) – Microsoft
- Phi-2 (2.7B) – Microsoft
- TinyLlama (1.1B) – Community
- OpenLlama (3B) – Community
- OpenLlama (7B) – Community (Excluded, >10B)
- Vicuna (7B) – LMSYS (Excluded, >10B)
- Koala (7B) – Berkeley (Excluded, >10B)
- Alpaca (7B) – Stanford (Excluded, >10B)
- Guanaco (7B) – Community (Excluded, >10B)
- Mistral (7B) – Mistral AI (Excluded, >10B)
- Zephyr (7B) – Hugging Face (Excluded, >10B)
- SOLAR (10.7B) – Upstage (Excluded, >10B)
- DeepSeek (7B) – DeepSeek (Excluded, >10B)
- Aquila (7B) – BAAI (Excluded, >10B)
- InternLM (7B) – Shanghai AI Lab (Excluded, >10B)
(Note: Many 7B models are excluded due to the 10B limit.)
2. Encoder-Decoder Models (Seq2Seq)
- T5 (3B) – Google
- T5 (1.1B) – Google
- T5 (220M) – Google
- T5 (60M) – Google
- T5 (Small, 60M) – Google
- T5 (Base, 220M) – Google
- T5 (Large, 770M) – Google
- T5 (XL, 3B) – Google
- FLAN-T5 (3B) – Google (Fine-tuned, excluded if strict)
- FLAN-T5 (1.1B) – Google (Fine-tuned, excluded if strict)
- ByT5 (3B) – Google
- ByT5 (1.1B) – Google
- UL2 (20B) – Google (Excluded, >10B)
- mT5 (3B) – Google
- mT5 (1.1B) – Google
- mT5 (580M) – Google
- mT5 (300M) – Google
- BART (400M) – Facebook
- BART (1.1B) – Facebook
- BART (3B) – Facebook
- Pegasus (560M) – Google
- Pegasus (1.1B) – Google
- Pegasus (3B) – Google
- M2M100 (1.2B) – Facebook
- M2M100 (418M) – Facebook
- SeamlessM4T (1B) – Meta
- SeamlessM4T (3B) – Meta
- NLLB (1.3B) – Meta
- NLLB (600M) – Meta
- MarianMT (1B) – Helsinki-NLP
3. Other Architectures (Hybrid/Alternative)
- Reformer (1B) – Google
- Longformer (1B) – AllenAI
- BigBird (1B) – Google
- Linformer (1B) – Google
- Perceiver (1B) – DeepMind
- Retro (1B) – DeepMind
- Switch Transformer (1B) – Google
- GLM (1.3B) – Tsinghua
- GLM (3B) – Tsinghua
- CPM (2.6B) – Tsinghua
- CPM (1.3B) – Tsinghua
- PanGu (1.3B) – Huawei
- PanGu (2.6B) – Huawei
- ERNIE (1.3B) – Baidu
- ERNIE (3B) – Baidu
- CogView (1.3B) – Tsinghua
- CogView (4B) – Tsinghua (Excluded, >10B)
- CogView2 (1.3B) – Tsinghua
- CogView2 (6B) – Tsinghua (Excluded, >10B)
- CodeGen (1B) – Salesforce
Key Notes:
- Strictly pretrained only: No fine-tuned models (e.g., FLAN-T5 is excluded if strict).
- Text-to-text focus: Models like BART, T5, and mT5 are ideal for generation.
- Size limit: All models are ≤10B parameters (7B models excluded).
- Diversity: Includes autoregressive (GPT-style) and encoder-decoder (T5-style) models.
For strictly base models, focus on GPT-2, GPT-Neo, OPT, Pythia, T5, BART, and mT5 variants.
Would you like a filtered list for a specific use case (e.g., multilingual, code generation)?
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.