Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0613· OpenAIHugging Face Arena Human Preference 55K

While pre-training an LLM on some corpus, what is the basic component for attention…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Ai2tulu-2-dpo-70b’s answer.
  1. User

    While pre-training an LLM on some corpus, what is the basic component for attention layers learning? Sequences which are indicated by EOS and BOS tokens, or fixed length chunks indicated by max tokens length (e.g 512) ?

  2. Assistant

    While pre-training a Language Model (LLM) like BERT or GPT, the basic component for attention layers learning is neither entirely sequences indicated by End Of Sentence (EOS) and Beginning Of Sentence (BOS) tokens, nor just fixed length chunks indicated by maximum token length (e.g., 512 tokens for BERT).

    The models operate on fixed length chunks of tokens. For instance, BERT operates on chunks of 512 tokens due to memory constraints. But, the chunks are not necessarily aligned with the actual sentences in the text. The model doesn't know where the sentences start or end unless explicitly marked with special tokens such as BOS or EOS.

    However, these models do not directly learn from the BOS or EOS tokens. The attention mechanism instead learns from the patterns in the data - the context of words within these fixed length chunks. The BOS and EOS tokens just provide additional context that helps the model understand the start and end of sentences.

    So, in summary, the basic component for attention layers learning is the fixed length chunks of tokens, and the model learns from the context of words within these chunks. Tokens like BOS or EOS can provide additional context but are not the primary learning component.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.