Skip to content
TheChatArchiveThe Chat Archive
Databricksmpt-7b-chat· DatabricksHugging Face Arena Human Preference 55K

Which choices for positional embeddings in transformers networks exist?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-1’s answer.
  1. User

    Which choices for positional embeddings in transformers networks exist?

  2. Databricks Assistant

    There are two main choices for positional embeddings in transformers networks:

    1. Linear: In this case, the positional embeddings are learned as linear functions of the integers, typically with a fixed length.
    2. Learning: In this case, the positional embeddings are learned as part of the model parameters, typically using a neural network to estimate the embeddings.

    Both choices have their own advantages and disadvantages, and the choice often depends on the specific problem and dataset. Linear embeddings are easier to compute and often more interpretable, while learning embeddings can capture more complex relationships between positions and may be more effective in certain scenarios.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt