1. Introduction Teaching computers to understand language is an active field in computer…
- User
1. Introduction
Teaching computers to understand language is an active field in computer science that began long before the development of what we know today as large language models (LLMs) and ChatGPT. Practically all current state-of-the-art LLMs and chatbots run on an architecture called transformers. Transformers were first described as an architecture in 2017 in the paper "Attention is all you need" by researchers at Google. This architecture revolutionised the field, not by being the first to show how one can teach computers to capture the meaning of text, but by enabling the processing of large amounts of text data efficiently and in parallel using graphic processing units (GPUs). A key property in the transformer giving the paper it’s catchy name is the attention mechanism. Attention allows the system to capture the relationship words have to their neighbours even over long distances. Previous architectures like Long Short-Term Memory (LSTM) networks see Fig.1 already showed promise in translating text and where able to remember the dependencies of words on each other but not over such long distances as transformers because they used a sequential processing scheme where one block only got informations from the previous block. Because their inability to be run in a highly parallelized way they could not keep up with the superior transformer architecture, which nowadays are so well optimised that they are said to be trained on all text data available online.
Figure 1: History of different transformer architectures. Source: Google [https://youtu.be/zbdong_h-x4?si=z5YE38miqIhc-kK4] Research into enhancing architectures like LSTMs (e.g., xLSTMs) continues to this day with researchers incorporating lessons learned from transformers and its many variants, though transformers remain the dominant architecture.
1.1 The Impact of Transformers on the field of conversational AI
With transformers being relatively new the field of conversational AI is just beginning to explore all the possibilities transformers trained on ever growing datasets bring to the field. Before the use of deep learning methods, NLP relied heavily on rule-based approaches and Semantic parsing which converts a sentence into a machine-understandable format.[Stanford Presentation https://youtu.be/fKMB5UlVY1E] While these approaches allow for a much more transparent analysis they struggled to capture the complexity, ambiguity and nuances of natural languages.
In fields like image processing and protein folding it became apparent that using large amounts of unstructured but labelled data and training a neural network on them lead to significant improvements compared to hand crafted rule-based systems starting with the successes of image net and its ability to label images better than state of the art models at that time.
Something similar can be observed in the field of language understanding where transformers performance improves by scaling the amount of high quality training data without the need of rule-based additions, even though the incorporation of rule based methods can be used to improve the results and explainability. The current transformer architecture acts much like a black box when it comes to interpreting the process that leads to an output prediction because of the high number of parameters and dimensions it is hard to interpret the reasoning. [Stanford https://youtu.be/fKMB5UlVY1E] Explainable AI is its own branch becoming increasingly important as transformers get used in more and more areas.
- Overview of the Transformer architecture
In this chapter all relevant parts of the transformer architecture will be listed. Some of them play an especially important role for the topic of conversational AI and we will be looking at them in some more detail.
Figure 2: The transformer architecture. Source: Attention is all you need. 2017
Inputs: The input text is transformed into tokens often 1 token is smaller than a word. Input Embedding: Tokens are converted into vectors containing the meaning of the words in the input text. Positional Encoding: Since transformers don’t have a built-in representation of order as found in the sequence to sequence models like RNNs. Positional encodings are baked into the embedding using a clever combination of sine and cosine waves. Information about the position of each token in the sequence is stored in a positional encoding matrix. This allows the model to capture the information about the order the tokens appear in the input text. Encoder: The decoder is a part of the transformer that is able to capture the meaning and relationship of words given to it in the input sequence. The idea of encoding words into vectors is not unique to transformers and can be implemented in different architectures like RNNs and LSTMs. [https://youtu.be/zbdong_h-x4?si=z5YE38miqIhc-kK4]. The encoder consists of multiple layers and parts: Multi-Head Self-Attention Mechanism: Allows the model to focus on different parts of the input sequence simultaneously, capturing various relationships and dependencies. Feed-Forward Neural Network (FFN): Transformation performed on the vector for each word after each attention layer. Helps capture additional meanings beside the relationship information captured by the attention mechanism. In this step the words don't use the neighbouring words directly and only use the single vector each word was assigned to. A ReLU activation function is used to put negative values to 0 and keep only the positive values. [3blue1brown https://youtu.be/9-Jl0dxWQs8?si=sIiviKMvrmId1S6n] Add & Norm (Addition and Normalization): Helps stabilise and speed up training. Decoder: The decoder uses many parts in common with the encoder. It uses some of the outputs of the encoder together with a start token and returns a probability prediction for the next possible token. Some of its parts differing from the encoder are: Encoder-Decoder Attention / Cross Attention: Allows the decoder to also focus on relevant parts of the input sequence given only to the encoder, by using the encodings generated by the encoder. Masked-Multi-Head-Attention Mechanism: Masking hides words that appear after the word that the model needs to predict. Doing this helps the model not to use these words in calculating the probabilities for the next token. Only words before the token that should be predicted are used by the decoder for making a prediction. Output Embedding and Softmax Layer: Converts the decoder’s output into a probability for each word in the vocabulary, allowing the model to generate the most likely next token in the sequence.
3 .Word Representations and Embeddings
To represent words, we use a technique called word embeddings. The brilliant idea here is that words are not simply represented by symbols but rather as vectors. By using a high-dimensional vector space, the meaning and relation of words towards each other can be represented and computation can be performed on these representations.
The higher the dimension of the vector the higher is the number of representations and nuanced meanings the word can take on. It turns the discrete representation of a word into a continuous one via the vector. [Stanford https://youtu.be/fKMB5UlVY1E]
Words are first given a unique number and then they are transformed into a vector (see Figure 3). This vector can be given any size and dimension and set random at start. It then points to a point in the vector space with dimensions of the vector size. Larger dimensions give each word more degrees of freedom in the vector space whereby each dimension could encode some meaning or relationship to other words. During training the position in the vector space will become meaningful.
Figure 3: Source: Umar Jamil hkproj/transformer-from-scratch-notes: Notes about "Attention is all you need" video (https://www.youtube.com/watch?v=bCz4OMemCcA) (github.com)
A famous example to explain how this representation as vectors can be used to capture the meaning and relationships with other words is the following: The words king, man, queen, and woman get represented as vectors. Then the following vector operation can be performed: king - man + woman ≈ queen. The match is very close but not exact since the word queen also finds use in different contexts that are not exactly the same as female ruler (see Figure 4.)
Figure 4: Source: 3Blue1Brown - But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning - Youtube Using these vectors, we can measure their relatedness towards one another using the dot product. The dot product returns a negative number when they point in opposite directions and a positive when they point in the same direction. In other words not just the meaning of words is captured by the vectors but it also becomes possible to make vector calculations on these representations to generate meaningful insights in the relationship between the words.. These static embeddings allow machines to discern contextual relationships. However, transformers go beyond this by using contextual embeddings, which adjust a word's vector based on its context in the sentence. The transformer takes dot products of whole matrices instead of single vectors but matrices are just collections of vectors so it is basically the same process.
3.1 Tokenization
Transformers tokenize words into smaller chunks called tokens, providing a uniform size for easier processing. The embeddings, along with positional information about where each token appears in a sentence, are used to understand the meaning behind words. In the following words and tokens will be used without distinguishing between them.
But not just words can be tokenized, all kinds of data like images or speech can also be represented as tokens and next token prediction can be performed on them by the transformer. LLMs are special transformers that process and predict only word tokens. When image or sound tokens are added in addition to text tokens, the term multimodal LLMs is used.
Since transformers work on sequences of data, tokenizing 2D objects is already introducing inefficiencies which grow when working with spatial data like objects in 3D, which hints at possible limits of the transformer architecture to be used on spatial data [“The Future of AI is Here” — Fei-Fei Li Unveils the Next Frontier of AI https://youtu.be/vIXfYFB7aBI?si=Vu7RnHxdbCVW9yh1]. However, for images, systems already exist that accept text and images with the idea that the input of an image triggers similar activations as words describing the images, allowing the model to understand the combined meaning of images and text. 4. The Transformer Architecture The original architecture is divided into two main components: the encoder and the decoder, both use the same self-attention mechanism and have multiple layers in common. (See figure). In the following we take a closer look at some of these parts making up the transformer
Figure 5: Source: https://youtu.be/4Bdc55j80l8?si=LxEEXiVFZwyyMtg2 4.1 Positional Encoding When given an input sequence each word in the sequence can be given a unique position label. In the original transformer paper a technique using cosine and sinus was used to generate such an unique label. When the labes is generated it can be added to the original embedding vector simply by addition to end up with the Embedding that is passed on to the transformer (see Fig. 6)
Figure 6. Generation of the encoder input matrics using the addition of word ambedding and positional embedding. Source: Umar Jamil hkproj/transformer-from-scratch-notes: Notes about "Attention is all you need" video (https://www.youtube.com/watch?v=bCz4OMemCcA) (github.com)
To make the positional encoding using cosine and sine clear, consider a simple sentence with just 3 words and 1 dimensions. Imagine the sentence is "I love cats" with "I" on position 1, "love" on position 2, and "cats" on position 3. If we had just 1 dimension, the vector would look like this [1, 2, 3], and we could put one sine and cosine wave in this 1 dimension (See Figure 7).
Figure. 7: One dimensional positional encoding using sine and cosine waves of the same frequency. For each position we get 2 new numbers which we use as a description to the position. In Fig. 7 at position 1 we have the values 5.1 and 8.0. These are now simply added to the vector representing the word I in the sentence. [1 + (5.1 + 8.0) , 2, 3]. This is also done for the other two words baking the positional information into the vectors that describes each word.
By allowing for a second dimension [ [1, _], [2, _], [3, _] ] which is filled with a random number in this example we get a embedding for the words that looks like this: [ [1, 22], [2, 41], [3, 58] ]. We again add the values the sine and cosine waves provide at this location (See. Fig. 8). For position 1 it would look like this: [ [1 + (5.1 + 8.0), 22 + (1.2 + (-1) ) ], [2, 41], [3, 58] ]. Same is done for position 2 and 3.
Figure. 8: Two dimensional positional encoding using sine and cosine waves of different frequency for the different dimensions. Because sine and cosine repeat, it is possible that the same value pair is observed multiple times. Therefore we allow each dimension to use a different frequency for the sine and cosine waves making the combinations unique when enough frequencies are used. The different frequencies also allow the model to capture positional information of words close together by using high frequency and words far apart by using low frequencies resulting in stretched out sine and cosine waves (see Fig. 9).
Figure 9: Different frequencies allow the model to capture positional information for words close together or far apart. [Source:https://youtu.be/T3OT8kqoqjc?si=07XhaPyIfJ6tfGS_] Using these cosine and sine waves has the following advantages: Uniqueness of position encoding. More frequencies lead to very unique combinations that can describe a position in a unique way. Capturing different patterns in the sentence structure by using low and high frequencies Works with very long sequences since sine and cosine are infinite Allows to estimate the distance of two token positions when they are located on one period of a wave Deterministic property. The position is defined by the sine and cosine waves which are both clearly defined mathematically Unambiguous. Each position has only one value (combination of values) [Source:https://youtu.be/T3OT8kqoqjc?si=07XhaPyIfJ6tfGS_]
4.2 Self-Attention Mechanism Self-attention allows the model to understand how each word in a sentence relates to every other word. It allows the model to focus its attention on one aspect in the input text. Using multiple self attention blocks allows the model to pay attention to not just one but multiple aspects of the input text [Stanford https://youtu.be/fKMB5UlVY1E]. In the training step weights are computed that allow the model to learn how much attention is given to each word in the input at each position. Attention weights are computed using dot products between vectors represented in the Query (Q), Key (K), and Value (V) matrices [Stanford https://youtu.be/fKMB5UlVY1E] and a final softmax to end up with values that can be interpreted as probabilities.
There are geometrical ways to interpret Query, Key and Value but here the following higher level interpretation is used: Q stores the question “what is the information I am looking for” each word can ask other words or to itself. For example, “Am I a subject? Do I appear beside the word ‘the’ or am I often found at the beginning of a sentence?” [Stanford https://youtu.be/fKMB5UlVY1E] K can be expressed as the question “What are the things that I have?” For example a book has a summery on its back and the key/summery helps find the information quickly [Stanford https://youtu.be/fKMB5UlVY1E] V represents “What is it that I communicate?”. When accessing the book the detailed information that is return is the value [Stanford https://youtu.be/fKMB5UlVY1E]The attention for all the words is then calculated using the following formula:
Q, K and V are matrices which contain all vectors given to each word and their positions. They get updated during training to capture the relationships and meanings of the words towards each other. When plugged into the formula, they return the attention matrix, which contains the information of which words are most important to each other. The size of Q, K, and V can be set freely and it determines how much information the model can capture from the training data. (d_k) is the size or dimension of each word vector which was defined in the very beginning of the word embedding.
The softmax function results in values between 0 and 1, so we avoid very large or negative numbers and can calculate probabilities from it.
Softmax then helps to choose from all the retrieved information which one is the most relevant by increasing the values for good matches and pushing values for less ideal matches down to zero. The outputs can also be interpreted as probabilities which ius useful when the question is which next word has the highest probability to be chosen as the next word in the sequence. Since each word is also related to itself, the attention matrix has most entries on its diagonal since most words in a sentence are not related to all the others and might only be related to themself (see Fig. 10). Figure 10: Attention matrix with highest relatedness between words themself on the diagonal. Source: Umar Jamil hkproj/transformer-from-scratch-notes: Notes about "Attention is all you need" video (https://www.youtube.com/watch?v=bCz4OMemCcA) (github.com) Other than previous architectures the distance between the words plays no role for a transformer if it gets a reasonable amount of compute power unlike LSTMs and RNNs which were limited by their sequential nature. Attention in transformers allows the capture of long-range dependencies and between all words (see Fig. 11). Furthermore self-attention is able to run in parallel and process all words simultaneously, this allows transformers to run highly parallelized and with less training steps speeding up training [Stanford https://youtu.be/fKMB5UlVY1E].
Figure 11. Attention can focus on words everywhere in the sequence. Source: https://youtu.be/4Bdc55j80l8?si=LxEEXiVFZwyyMtg2
However, even in transformers allowing for longer-range dependencies comes at a cost. The more relations with other words are allowed and the more positions exist, the more entries are required in the different embedding matrices. When the vocabulary grows, more relations between words are possible and need to be stored and when the context length grows, more possibilities of how words are positioned towards each other can be generated and also need to be stored. Changes in vocabulary and context length therefore result in a non-linear increase of compute complexity limiting the size of the context window even in the latest LLMs.
4.3 Multi-Head Attention The attention mechanism determines the importance of other words towards a given word and can be applied multiple times independent of each other to focus on different parts of the input sequence [ How Decoder-Only Transformers (like GPT) Work https://youtu.be/baykhvFS_e4?si=Qipm02OQVgnwMR8q] . The different runs can then be combined again via a concatenation operation (see Fig. 12). This process is called multi-head attention mechanism and allows for the capture of more diverse patterns in the input. The context for a word can look very different, and allowing for multiple heads also allows for multiple diverse contexts.
Figure 12: Multi Head Attention with their outputs getting concatenated. Source: Attention is all you need. 2017
In the Fig. 13, each head is labelled by a different colour and when looking at only one word and asking each of the heads which words are most related to it, different colours indicate that each head focuses on different words and marks them as most important. Figure 13: Multiple heads and their prediction for the most relevant words related to “making”. For each head a different colour is used. Source: Umar Jamil hkproj/transformer-from-scratch-notes: Notes about "Attention is all you need" video (https://www.youtube.com/watch?v=bCz4OMemCcA) (github.com)
4.3.1 Cross Attention
In the decoder encoder architecture the encoder and decoder have different inputs and for these different inputs both calculate distinct Q,V and K matrices.
With both encoder and decoder using similar but different Q, K, and V matrices, it becomes possible to recombine the matrices from encoder and decoder which is known as cross attention because information crosses from the encoder into the decoder helping the decoder make a prediction using the learned weights from the encoder (see Fig. 14).
Cross attention allows the encoder to send information to the decoder. K and V come from the encoder and Q is used from the decoder.
Figure 15: Source: Showing the flow of information required for a translation task from Encoder (left) to Decoder (right) [Stanford https://youtu.be/fKMB5UlVY1E] Via the connection to the encoder, the decoder knows what needs to be translated in this example Fig. 15. Otherwise, it would need to make a prediction based on the knowledge stored in its own Q, V, and K matrices without knowing exactly what to translate. The Decoder without the Encoder would still be able to make a prediction for the most likely next word but that would not be a translation anymore but a basic next token prediction. 4.4 Feed-Forward Neural Networks (Multi-Layer Perceptron Blocks) After the attention mechanism, each word goes through its own feed-forward neural network. This transformation is performed separately for each word vector, after each attention step. This step helps capture additional information not captured by the attention mechanism alone. In the feed forward step, the words don’t use the neighbouring words directly and only work with the single vector which describes each word and its position and relationships. An intuitive way to understand what information are learned in this feed forward network is doing is not so easy but one can think of it as if each token vector is asked questions about his role in a sentence like “is the word related to an animal” and if it is this weight is strengthen giving the network the ability to learn what different token vectors represent and stand for (see Fig. 16). [ 3Brown1Blue https://youtu.be/wjZofJX0v4M ]
Figure 16: Given one token vector the feed forward network learns what meaning this particular token vector carries. Source: [ 3Brown1Blue https://youtu.be/wjZofJX0v4M ] 4.4 Encoder-Decoder Structure Both the encoder and decoder share several common components and mechanisms, such as input embeddings that convert tokens into vectors, the addition of positional encodings to these embeddings, performing self-attention to capture relationships between words within the input sequence, applying feed-forward transformations to further process the embeddings, and utilizing normalization after certain layers. 4.4.1 Encoder The encoder processes the input sentence, using self-attention to capture the relationships between words. Its main task is to encode the input data into a set of vectors that represent the relationships and meanings of the words. These encoded representations are then made available to the decoder for generating predictions. 4.4.2 Decoder The decoder is responsible for generating the next token in the output sequence. It receives as input a start token and predicts the most likely next token. For each prediction, the decoder uses parts of the encoder's output (specifically the Value (V) and Key (K) matrices) and the tokens generated so far. This process is known as inference and involves generating one word at a time in a loop until an end-of-sequence token is produced.
4.5 Masked self-attention
Is used by the decoder to prevent it from looking at future tokens during the generation process. This is also called an autoregressive property meaning each token is generated one at a time.
The decoder uses this method to only look back at the tokens already generated and not to future tokens. This guarantees that the prediction is auto regressive meaning only one token is generated at a time and not all in parallel. After each new prediction the newly predicted token as well as all previous tokens are considered before performing the next token prediction. This can be useful when the decoder's own prediction should also influence its next prediction.
Fig. 17 shows how masking can be used to hide all upcoming words.
Figure 17. Masking used by the decoder to prevent itself from looking ahead during the prediction step, ensuring that the prediction is auto regressive one token after the other. Source: 3Blue1Brown - But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning - Youtube
In combination with cross-attention the decoder is allowed to focus on relevant parts of the encoder's output despite masking preventing the decoder to look ahead in the initial step. This connection to the encoder is needed to perform translation tasks.
- Training Transformers are trained to predict the next word in a sentence or translate text sequences. The training process enables the model to learn relationships between words, storing this information within its network weights.
The following learning process can be observed: Attention Mechanism: The model learns to focus on relevant parts in the input to make a prediction for the most likely next word. Feed-Forward Blocks: Help the model learn what the token vectors represent Positional Encodings: The network learns the position patterns using sine and cosine.
What these mechanisms learn was already covered in the corresponding chapters. The attention mechanism learns connection between words, positional embeddings are learned only once and then applied in the inference phase.Feed forward networks learn what each token vectors represents.
- Inference Inference is the phase where the trained transformer model generates the next token based on a never seen before input sequence. At first the input is processed. The text is tokenized, then an embedding created for the token input, and positional encodings added to the vectors. The embeddings and learned parameters are used to generate the Q, V and K matrices for the given input sequence. Q,V and K are used by the attention mechanism to predict the scores for each token to be the next one and the token with the highest likelihood can be chosen to be appended to the sequence the decoder started with.
Figure 18 shows how this inference step would look like for the prediction of the next token in a translation task.It starts with the encoder calculating its Q,K and V matrices and offering them to the decoder to use. The decoder on the other hand starts with a single ”Start of Sequence” token. Via cross attention the information what to translate is combined with the information provided by the start token to come up with the likely next word which is Ti as the italian word of I and first word in the translation.
Figure 18. Encoder generates the Q,K and V matrices and provides K and V to the decoder which uses them to make a prediction of the first word in the translation. Source: Umar Jamil hkproj/transformer-from-scratch-notes: Notes about "Attention is all you need" video (https://www.youtube.com/watch?v=bCz4OMemCcA) (github.com)
In the following steps the Q, K and V matrices of the encoder do not need to be recalculated and can be reused (see Fig. 19).
Figure 19: Prediction of the second word in the translation by the decoder reusing the already calculated matrices from the encoder. Source: Umar Jamil hkproj/transformer-from-scratch-notes: Notes about "Attention is all you need" video (https://www.youtube.com/watch?v=bCz4OMemCcA) (github.com)
- Model Zoo
7.1 Decoder only models
The most famous transformer model might not be the encode-decoder model presented in this Expose but instead its Decoder-Only successors the GPT (Generative Pre-trained Transformer).
The Decoder only model is very similar to the original transformer simple removing the encoder and the parts required for the cross attention in the original transformer (See Fig.20)
Instead of having booth an encoder and decoder with the decoder having access to the encoder weights, a decoder only architecture only has to train the decoder reducing its training time and reducing the required compute resources.
Figure 20: Source: [Andrej Karpathy - Stanford https://youtu.be/XfpMkf4rD6E]
The training of the decoder is done on data created by masking as was shown in Fig. 17 training data can be generated by masking all upcoming words of a sentence and training the model to predict the next tokens. Since the next tokens are known and only synthetically occluded the true labels are known and the loos function can be calculated and used to do backpropagation and weights tuning. [Robin van Hoorn - StackOverflow - deep learning - How does the (decoder-only) transformer architecture work? - Artificial Intelligence Stack Exchange]
7.2 Encoder Only Models
As the name suggests here only the encoder is trained and used. A prominend example of a Encoder only Model is BERT (Bidirectional Encoder Representations from Transformers). Since the encoder has access to a complete input text during training, it understands the structure of text very well. It therefore is used in tasks like sentiment analysis where the whole input text can be searched for clues about the emotional tone of the sentence. An encoder only model can also be used to spot irregular patterns in the input sequence like reversed sentence order in a paragraph. [Sebastian Raschka, PhD- Understanding Encoder And Decoder LLMs- Understanding Encoder And Decoder LLMs (sebastianraschka.com)] Named Entity Recognition (NER) is another common problem in the NLP-field where the model needs to find important information/entities in a text like names, nouns, verbs, etc.
7.3 Multi Model Transformers
In previous chapters it was mentioned that text is only one of the modalities one can give to a transformer. Transformers which are trained on multiple modalities can be used for very different tasks like required in visual question answering (VQA). This could be that an image is given to the transformer model in combination with a Question prompt to describe the image and the model is tasked to return an answer requiring a deeper understanding of what is shown on the image and being able to describe it in written language.. [Tezan Sahu - Visual question answering with multimodal transformers- Visual question answering with multimodal transformers | by Tezan Sahu | Data Science at Microsoft | Medium] Another multi model transformer especially relevant for conversational AI are transformers that are able to accept text and audio data and are able to not only transcribe text to speech but also to capture the meaning of tone, pitch and emotional tone in a speech example and show the ability to describe it using text.
- Conclusion.
In this expose the workings of the original transformer model were put in the centre of the focus. The goal was to gain a deeper understanding of the underlying technology. Since this technology is still powering so many innovations happening in the field of AI in general and conversational AI in particular having a deeper understanding helps to understand the current trends and possible future directions this field can move in.
Many topics like fine tuning, prompt engineering or quantization as well as all the diverse use cases transformer based models can be deployed for where not covered. But the amount of subjects one could cover underlines the importance and relevance of the ideas covered here for the field of conversational AI.
fix all grammatical sentence structure errors and post the result, do not change the text to much only help remove grammar errors and mybe uncommon words or structures should be fixed. if you fix somethin make it bold so i see what changes imediatly. my carrear depends on this text so please make it as good as you can
- Assistant
1. Introduction
Teaching computers to understand language is an active field in computer science that began long before the development of what we know today as large language models (LLMs) and ChatGPT. Practically all current state-of-the-art LLMs and chatbots run on an architecture called transformers.
Transformers were first described as an architecture in 2017 in the paper "Attention is All You Need" by researchers at Google. This architecture revolutionized the field, not by being the first to show how one can teach computers to capture the meaning of text, but by enabling the processing of large amounts of text data efficiently and in parallel using graphics processing units (GPUs).
A key property in the transformer giving the paper its catchy name is the attention mechanism. Attention allows the system to capture the relationships words have to their neighbors even over long distances. Previous architectures like Long Short-Term Memory (LSTM) networks, see Fig.1, already showed promise in translating text and were able to remember the dependencies of words on each other, but not over such long distances as transformers because they used a sequential processing scheme where one block only received information from the previous block. Because of their inability to be run in a highly parallelized way, they could not keep up with the superior transformer architecture, which nowadays is so well optimized that it is said to be trained on all text data available online.
[image: Figure 1: History of different transformer architectures. Source: Google] open
Research into enhancing architectures like LSTMs (e.g., xLSTMs) continues to this day with researchers incorporating lessons learned from transformers and their many variants, though transformers remain the dominant architecture.
1.1 The Impact of Transformers on the Field of Conversational AI
With transformers being relatively new, the field of conversational AI is just beginning to explore all the possibilities that transformers trained on ever-growing datasets bring to the field. Before the use of deep learning methods, NLP relied heavily on rule-based approaches and semantic parsing, which converts a sentence into a machine-understandable format.Stanford Presentation While these approaches allow for a much more transparent analysis, they struggled to capture the complexity, ambiguity, and nuances of natural languages.
In fields like image processing and protein folding, it became apparent that using large amounts of unstructured but labeled data and training a neural network on them led to significant improvements compared to hand-crafted rule-based systems, starting with the successes of ImageNet and its ability to label images better than state-of-the-art models at that time.
Something similar can be observed in the field of language understanding, where transformers' performance improves by scaling the amount of high-quality training data without the need for rule-based additions, even though the incorporation of rule-based methods can be used to improve the results and explainability. The current transformer architecture acts much like a black box when it comes to interpreting the process that leads to an output prediction because of the high number of parameters and dimensions it is hard to interpret the reasoning.Stanford Explainable AI is its own branch becoming increasingly important as transformers get used in more and more areas.
2. Overview of the Transformer Architecture
In this chapter, all relevant parts of the transformer architecture will be listed. Some of them play an especially important role for the topic of conversational AI, and we will be looking at them in some more detail.
[image: Figure 2: The transformer architecture. Source: Attention is All You Need. 2017]
- Inputs: The input text is transformed into tokens, often where 1 token is smaller than a word.
- Input Embedding: Tokens are converted into vectors containing the meaning of the words in the input text.
- Positional Encoding: Since transformers don’t have a built-in representation of order as found in the sequence-to-sequence models like RNNs, positional encodings are baked into the embedding using a clever combination of sine and cosine waves.
- Information about the position of each token in the sequence is stored in a positional encoding matrix. This allows the model to capture the information about the order in which the tokens appear in the input text.
- Encoder: The encoder is a part of the transformer that is able to capture the meaning and relationship of words given to it in the input sequence. The idea of encoding words into vectors is not unique to transformers and can be implemented in different architectures like RNNs and LSTMs.YouTube
- The encoder consists of multiple layers and parts:
- Multi-Head Self-Attention Mechanism: Allows the model to focus on different parts of the input sequence simultaneously, capturing various relationships and dependencies.
- Feed-Forward Neural Network (FFN): Transformation performed on the vector for each word after each attention layer. Helps capture additional meanings beside the relationship information captured by the attention mechanism. In this step, the words don't use the neighboring words directly and only use the single vector each word was assigned to. A ReLU activation function is used to set negative values to 0 and keep only the positive values.3Blue1Brown
- Add & Norm (Addition and Normalization): Helps stabilize and speed up training.
- The encoder consists of multiple layers and parts:
- Decoder: The decoder uses many parts in common with the encoder. It uses some of the outputs of the encoder together with a start token and returns a probability prediction for the next possible token. Some of its parts differ from the encoder:
- Encoder-Decoder Attention / Cross Attention: Allows the decoder to also focus on relevant parts of the input sequence given only to the encoder, by using the encodings generated by the encoder.
- Masked Multi-Head Attention Mechanism: Masking hides words that appear after the word that the model needs to predict. Doing this helps the model not use these words in calculating the probabilities for the next token. Only words before the token that should be predicted are used by the decoder for making a prediction.
- Output Embedding and Softmax Layer: Converts the decoder’s output into a probability for each word in the vocabulary, allowing the model to generate the most likely next token in the sequence.
3. Word Representations and Embeddings
To represent words, we use a technique called word embeddings. The brilliant idea here is that words are not simply represented by symbols but rather as vectors. By using a high-dimensional vector space, the meaning and relations of words towards each other can be represented and computation can be performed on these representations.
The higher the dimension of the vector, the higher is the number of representations and nuanced meanings the word can take on. It turns the discrete representation of a word into a continuous one via the vector.Stanford
Words are first given a unique number and then they are transformed into a vector (see Figure 3). This vector can be of any size and dimension and set randomly at the start. It then points to a point in the vector space with dimensions of the vector size. Larger dimensions give each word more degrees of freedom in the vector space, whereby each dimension could encode some meaning or relationship to other words. During training, the position in the vector space becomes meaningful.
[image: Figure 3: Source: Umar Jamil hkproj/transformer-from-scratch-notes: Notes about "Attention is All You Need" video] open
A famous example to explain how this representation as vectors can be used to capture the meaning and relationships with other words is the following: The words king, man, queen, and woman get represented as vectors. Then the following vector operation can be performed: king - man + woman ≈ queen. The match is very close but not exact since the word queen also finds use in different contexts that are not exactly the same as female ruler (see Figure 4).
[image: Figure 4: Source: 3Blue1Brown - But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning - YouTube]
Using these vectors, we can measure their relatedness towards one another using the dot product. The dot product returns a negative number when they point in opposite directions and a positive number when they point in the same direction. In other words, not just the meaning of words is captured by the vectors, but it also becomes possible to make vector calculations on these representations to generate meaningful insights into the relationships between the words.
These static embeddings allow machines to discern contextual relationships. However, transformers go beyond this by using contextual embeddings, which adjust a word's vector based on its context in the sentence. The transformer takes dot products of whole matrices instead of single vectors, but matrices are just collections of vectors, so it is basically the same process.
3.1 Tokenization
Transformers tokenize words into smaller chunks called tokens, providing a uniform size for easier processing. The embeddings, along with positional information about where each token appears in a sentence, are used to understand the meaning behind words. In the following, words and tokens will be used without distinguishing between them.
But not just words can be tokenized; all kinds of data like images or speech can also be represented as tokens, and next token prediction can be performed on them by the transformer. LLMs are special transformers that process and predict only word tokens. When image or sound tokens are added in addition to text tokens, the term multimodal LLMs is used.
Since transformers work on sequences of data, tokenizing 2D objects is already introducing inefficiencies, which grow when working with spatial data like objects in 3D, which hints at possible limits of the transformer architecture to be used on spatial data “The Future of AI is Here” — Fei-Fei Li Unveils the Next Frontier of AI. However, for images, systems already exist that accept text and images with the idea that the input of an image triggers similar activations as words describing the images, allowing the model to understand the combined meaning of images and text.
4. The Transformer Architecture
The original architecture is divided into two main components: the encoder and the decoder, both use the same self-attention mechanism and have multiple layers in common. (See figure). In the following, we take a closer look at some of these parts making up the transformer.
[image: Figure 5: Source: YouTube] open
4.1 Positional Encoding
When given an input sequence, each word in the sequence can be given a unique position label. In the original transformer paper, a technique using cosine and sine was used to generate such a unique label. When the label is generated, it can be added to the original embedding vector simply by addition to end up with the embedding that is passed on to the transformer (see Fig. 6).
[image: Figure 6: Generation of the encoder input matrix using the addition of word embedding and positional embedding. Source: Umar Jamil hkproj/transformer-from-scratch-notes] open
To make the positional encoding using cosine and sine clear, consider a simple sentence with just 3 words and 1 dimension. Imagine the sentence is "I love cats" with "I" in position 1, "love" in position 2, and "cats" in position 3. If we had just 1 dimension, the vector would look like this [1, 2, 3], and we could put one sine and cosine wave in this 1 dimension (See Figure 7).
[image: Figure 7: One-dimensional positional encoding using sine and cosine waves of the same frequency.]
For each position, we get 2 new numbers which we use as a description of the position. In Fig. 7 at position 1, we have the values 5.1 and 8.0. These are now simply added to the vector representing the word "I" in the sentence. [1 + (5.1 + 8.0), 2, 3]. This is also done for the other two words, baking the positional information into the vectors that describe each word in the sentence.
By allowing for a second dimension [[1, _], [2, _], [3, _]] which is filled with a random number in this example, we get an embedding for the words that looks like this: [[1, 22], [2, 41], [3, 58]]. We again add the values the sine and cosine waves provide at this location (See Fig. 8). For position 1, it would look like this: [[1 + (5.1 + 8.0), 22 + (1.2 + (-1))], [2, 41], [3, 58]]. The same is done for positions 2 and 3.
[image: Figure 8: Two-dimensional positional encoding using sine and cosine waves of different frequencies for the different dimensions.]
Because sine and cosine repeat, it is possible that the same value pair is observed multiple times. Therefore, we allow each dimension to use a different frequency for the sine and cosine waves, making the combinations unique when enough frequencies are used. The different frequencies also allow the model to capture positional information of words close together by using high frequencies and words far apart by using low frequencies, resulting in stretched-out sine and cosine waves (see Fig. 9).
[image: Figure 9: Different frequencies allow the model to capture positional information for words close together or far apart. Source: YouTube] open
Using these cosine and sine waves has the following advantages:
- Uniqueness of position encoding. More frequencies lead to very unique combinations that can describe a position in a unique way.
- Capturing different patterns in the sentence structure by using low and high frequencies.
- Works with very long sequences since sine and cosine are infinite.
- Allows to estimate the distance of two token positions when they are located on one period of a wave.
- Deterministic property. The position is defined by the sine and cosine waves, which are both clearly defined mathematically.
- Unambiguous. Each position has only one value (combination of values).
4.2 Self-Attention Mechanism
Self-attention allows the model to understand how each word in a sentence relates to every other word. It allows the model to focus its attention on one aspect in the input text. Using multiple self-attention blocks allows the model to pay attention to not just one but multiple aspects of the input text.Stanford
In the training step, weights are computed that allow the model to learn how much attention is given to each word in the input at each position. Attention weights are computed using dot products between vectors represented in the Query (Q), Key (K), and Value (V) matrices.Stanford and a final softmax to end up with values that can be interpreted as probabilities.
There are geometrical ways to interpret Query, Key, and Value, but here the following higher-level interpretation is used:
- Q stores the question “What is the information I am looking for?” Each word can ask other words or itself. For example, “Am I a subject? Do I appear beside the word ‘the’? Or am I often found at the beginning of a sentence?” Stanford
- K can be expressed as the question “What are the things that I have?” For example, a book has a summary on its back, and the key/summary helps find the information quickly.Stanford
- V represents “What is it that I communicate?” When accessing the book, the detailed information that is returned is the value.Stanford
The attention for all the words is then calculated using the following formula:
[ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V ]
Q, K, and V are matrices that contain all vectors given to each word and their positions. They get updated during training to capture the relationships and meanings of the words towards each other. When plugged into the formula, they return the attention matrix, which contains the information of which words are most important to each other. The size of Q, K, and V can be set freely, and it determines how much information the model can capture from the training data. ( d_k ) is the size or dimension of each word vector, which was defined at the very beginning of the word embedding.
The softmax function results in values between 0 and 1, so we avoid very large or negative numbers and can calculate probabilities from it.
Softmax then helps to choose from all the retrieved information which one is the most relevant by increasing the values for good matches and pushing values for less ideal matches down to zero. The outputs can also be interpreted as probabilities, which is useful when the question is which next word has the highest probability to be chosen as the next word in the sequence.
Since each word is also related to itself, the attention matrix has most entries on its diagonal since most words in a sentence are not related to all the others and might only be related to themselves (see Fig. 10).
[image: Figure 10: Attention matrix with highest relatedness between words themselves on the diagonal. Source: Umar Jamil hkproj/transformer-from-scratch-notes] open
Other than previous architectures, the distance between the words plays no role for a transformer if it gets a reasonable amount of compute power, unlike LSTMs and RNNs which were limited by their sequential nature. Attention in transformers allows the capture of long-range dependencies between all words (see Fig. 11). Furthermore, self-attention is able to run in parallel and process all words simultaneously. This allows transformers to run highly parallelized and with fewer training steps, speeding up training.Stanford
[image: Figure 11: Attention can focus on words everywhere in the sequence. Source: YouTube] open
However, even in transformers, allowing for longer-range dependencies comes at a cost. The more relations with other words are allowed and the more positions exist, the more entries are required in the different embedding matrices. When the vocabulary grows, more relations between words are possible and need to be stored, and when the context length grows, more possibilities of how words are positioned relative to each other can be generated and also need to be stored. Changes in vocabulary and context length, therefore, result in a non-linear increase of compute complexity, limiting the size of the context window even in the latest LLMs.
4.3 Multi-Head Attention
The attention mechanism determines the importance of other words towards a given word and can be applied multiple times independently of each other to focus on different parts of the input sequence.How Decoder-Only Transformers (like GPT) Work The different runs can then be combined again via a concatenation operation (see Fig. 12).
This process is called the multi-head attention mechanism and allows for the capture of more diverse patterns in the input. The context for a word can look very different, and allowing for multiple heads also allows for multiple diverse contexts.
[image: Figure 12: Multi-Head Attention with their outputs getting concatenated. Source: Attention is All You Need. 2017]
In Fig. 13, each head is labelled by a different color, and when looking at only one word and asking each of the heads which words are most related to it, different colors indicate that each head focuses on different words and marks them as most important.
[image: Figure 13: Multiple heads and their prediction for the most relevant words related to “making”. For each head, a different color is used. Source: Umar Jamil hkproj/transformer-from-scratch-notes] open
4.3.1 Cross Attention
In the encoder-decoder architecture, the encoder and decoder have different inputs, and for these different inputs, both calculate distinct Q, V, and K matrices.
With both encoder and decoder using similar but different Q, K, and V matrices, it becomes possible to recombine the matrices from the encoder and decoder, which is known as cross attention because information crosses from the encoder into the decoder, helping the decoder make a prediction using the learned weights from the encoder (see Fig. 14).
Cross attention allows the encoder to send information to the decoder. K and V come from the encoder, and Q is used from the decoder.
[image: Figure 15: Source: Showing the flow of information required for a translation task from Encoder (left) to Decoder (right)] open
Via the connection to the encoder, the decoder knows what needs to be translated in this example (Fig. 15). Otherwise, it would need to make a prediction based on the knowledge stored in its own Q, V, and K matrices without knowing exactly what to translate. The decoder without the encoder would still be able to make a prediction for the most likely next word, but that would not be a translation anymore but a basic next token prediction.
4.4 Feed-Forward Neural Networks (Multi-Layer Perceptron Blocks)
After the attention mechanism, each word goes through its own feed-forward neural network. This transformation is performed separately for each word vector after each attention step. This step helps capture additional information not captured by the attention mechanism alone.
In the feed-forward step, the words don’t use the neighboring words directly and only work with the single vector which describes each word and its position and relationships. An intuitive way to understand what information is learned in this feed-forward network is not so easy, but one can think of it as if each token vector is asked questions about its role in a sentence like “Is the word related to an animal?” and if it is, this weight is strengthened, giving the network the ability to learn what different token vectors represent and stand for (see Fig. 16).3Blue1Brown
[image: Figure 16: Given one token vector, the feed-forward network learns what meaning this particular token vector carries. Source: 3Blue1Brown]
4.4 Encoder-Decoder Structure
Both the encoder and decoder share several common components and mechanisms, such as input embeddings that convert tokens into vectors, the addition of positional encodings to these embeddings, performing self-attention to capture relationships between words within the input sequence, applying feed-forward transformations to further process the embeddings, and utilizing normalization after certain layers.
4.4.1 Encoder
The encoder processes the input sentence, using self-attention to capture the relationships between words. Its main task is to encode the input data into a set of vectors that represent the relationships and meanings of the words. These encoded representations are then made available to the decoder for generating predictions.
4.4.2 Decoder
The decoder is responsible for generating the next token in the output sequence. It receives as input a start token and predicts the most likely next token. For each prediction, the decoder uses parts of the encoder's output (specifically the Value (V) and Key (K) matrices) and the tokens generated so far. This process is known as inference and involves generating one word at a time in a loop until an end-of-sequence token is produced.
4.5 Masked Self-Attention
Masked self-attention is used by the decoder to prevent it from looking at future tokens during the generation process. This is also called an autoregressive property, meaning each token is generated one at a time.
The decoder uses this method to only look back at the tokens already generated and not at future tokens. This guarantees that the prediction is autoregressive, meaning only one token is generated at a time and not all in parallel. After each new prediction, the newly predicted token as well as all previous tokens are considered before performing the next token prediction.
Fig. 17 shows how masking can be used to hide all upcoming words.
[image: Figure 17: Masking used by the decoder to prevent itself from looking ahead during the prediction step, ensuring that the prediction is autoregressive, one token after the other. Source: 3Blue1Brown] open
In combination with cross-attention, the decoder is allowed to focus on relevant parts of the encoder's output despite masking preventing the decoder from looking ahead in the initial step. This connection to the encoder is needed to perform translation tasks.
5. Training
Transformers are trained to predict the next word in a sentence or translate text sequences. The training process enables the model to learn relationships between words, storing this information within its network weights.
The following learning processes can be observed:
- Attention Mechanism: The model learns to focus on relevant parts in the input to make a prediction for the most likely next word.
- Feed-Forward Blocks: Help the model learn what the token vectors represent.
- Positional Encodings: The network learns the position patterns using sine and cosine.
What these mechanisms learn was already covered in the corresponding chapters. The attention mechanism learns connections between words, positional embeddings are learned only once and then applied in the inference phase, and feed-forward networks learn what each token vector represents.
6. Inference
Inference is the phase where the trained transformer model generates the next token based on a never-before-seen input sequence.
At first, the input is processed. The text is tokenized, then an embedding is created for the token input, and positional encodings are added to the vectors. The embeddings and learned parameters are used to generate the Q, V, and K matrices for the given input sequence. Q, V, and K are used by the attention mechanism to predict the scores for each token to be the next one, and the token with the highest likelihood can be chosen to be appended to the sequence the decoder started with.
Figure 18 shows how this inference step would look like for the prediction of the next token in a translation task. It starts with the encoder calculating its Q, K, and V matrices and offering them to the decoder to use. The decoder, on the other hand, starts with a single “Start of Sequence” token. Via cross-attention, the information about what to translate is combined with the information provided by the start token to come up with the likely next word, which is Ti as the Italian word for "I" and the first word in the translation.
[image: Figure 18: Encoder generates the Q, K, and V matrices and provides K and V to the decoder, which uses them to make a prediction of the first word in the translation. Source: Umar Jamil hkproj/transformer-from-scratch-notes] open
In the following steps, the Q, K, and V matrices of the encoder do not need to be recalculated and can be reused (see Fig. 19).
[image: Figure 19: Prediction of the second word in the translation by the decoder reusing the already calculated matrices from the encoder. Source: Umar Jamil hkproj/transformer-from-scratch-notes] open
7. Model Zoo
7.1 Decoder-Only Models
The most famous transformer model might not be the encoder-decoder model presented in this exposé, but instead its Decoder-Only successors, the GPT (Generative Pre-trained Transformer).
The Decoder-only model is very similar to the original transformer, simply removing the encoder and the parts required for the cross-attention in the original transformer (See Fig.20).
Instead of having both an encoder and decoder with the decoder having access to the encoder weights, a decoder-only architecture only has to train the decoder, reducing its training time and the required compute resources.
[image: Figure 20: Source: Andrej Karpathy - Stanford] open
The training of the decoder is done on data created by masking, as was shown in Fig. 17. Training data can be generated by masking all upcoming words of a sentence and training the model to predict the next tokens. Since the next tokens are known and only synthetically occluded, the true labels are known, and the loss function can be calculated and used to do backpropagation and weight tuning.Robin van Hoorn - StackOverflow
7.2 Encoder-Only Models
As the name suggests, here only the encoder is trained and used. A prominent example of an Encoder-only model is BERT (Bidirectional Encoder Representations from Transformers).
Since the encoder has access to the complete input text during training, it understands the structure of text very well. It is therefore used in tasks like sentiment analysis, where the whole input text can be searched for clues about the emotional tone of the sentence. An encoder-only model can also be used to spot irregular patterns in the input sequence, like reversed sentence order in a paragraph.Sebastian Raschka, PhD Named Entity Recognition (NER) is another common problem in the NLP field where the model needs to find important information/entities in a text like names, nouns, verbs, etc.
7.3 Multi-Modal Transformers
In previous chapters, it was mentioned that text is only one of the modalities one can give to a transformer. Transformers which are trained on multiple modalities can be used for very different tasks, like those required in visual question answering (VQA). This could be that an image is given to the transformer model in combination with a question prompt to describe the image, and the model is tasked to return an answer requiring a deeper understanding of what is shown in the image and the ability to describe it using written language.Tezan Sahu - Visual question answering with multimodal transformers
Another multi-modal transformer especially relevant for conversational AI are transformers that can accept text and audio data and are able to not only transcribe text to speech but also to capture the meaning of tone, pitch, and emotional tone in a speech example and show the ability to describe it using text.
8. Conclusion
In this exposé, the workings of the original transformer model were put at the center of focus. The goal was to gain a deeper understanding of the underlying technology. Since this technology is still powering so many innovations happening in the field of AI in general and conversational AI in particular, having a deeper understanding helps to understand the current trends and possible future directions this field can move in.
Many topics like fine-tuning, prompt engineering, or quantization, as well as all the diverse use cases transformer-based models can be deployed for, were not covered. But the amount of subjects one could cover underlines the importance and relevance of the ideas covered here for the field of conversational AI.