Skip to content
TheChatArchiveThe Chat Archive
Human-written demonstration (OpenAssistant volunteers)· OpenAssistant (human-written)Hugging Face OpenAssistant Conversations (oasst2)

How long of an embedding vector is common for a vision transformer?

In OpenAssistant data, “assistant” turns were written by human volunteers as demonstrations. They are not model outputs.
  1. User

    How long of an embedding vector is common for a vision transformer?

  2. Assistant

    Vision Transformers commonly use embedding vectors of size:

    768 (e.g. ViT-Base) 1024 (e.g. ViT-Large) 2048 (e.g. ViT-Huge)

    Andrei-Cristian Rad explains in his article on Medium that the ViT architecture uses a trainable embedding tensor of shape (p²*c, d), which learns to linearly project each flat patch to dimension d. This dimension d is constant in the architecture and is used in most of the components.

    So anywhere from 768 to 2048 dimensions is common and has been explored in research. In general, larger embedding sizes allow the model to capture more fine-grained relationships and representations, but also increase parameter count and risk of overfitting. So it's a tradeoff, and the exact size used will depend on the specific use case and data available.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.