While pre-training an LLM on some corpus, what is the basic component for attention…
- User
While pre-training an LLM on some corpus, what is the basic component for attention layers learning? Sequences which are indicated by EOS and BOS tokens, or fixed length chunks indicated by max tokens length (e.g 512) ?
Assistant
When pre-training a large language model (LLM) on a corpus, the basic component for attention layers learning is the input sequence. The sequence is typically delimited by special tokens such as Beginning of Sequence (BOS) and End of Sequence (EOS) tokens, which indicate the beginning and end of the input sequence, respectively.
The attention mechanism in Transformer-based models is designed to learn contextual relationships between the tokens in the input sequence. The model learns to weigh the importance of each token in the sequence relative to the other tokens, depending on the specific task or objective.
The fixed length chunks you mention, often referred to as the maximum token length, generally refer to the maximum number of tokens considered in a single input sequence. This is usually set to a fixed number, such as 512, to limit the computational resources required for processing and to ensure that the model can handle input sequences of varying lengths.
In summary, the basic component for attention layers learning in LLM pre-training is the input sequence delimited by BOS and EOS tokens, while the maximum token length determines the fixed chunk size for processing those sequences.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.