Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0125-preview· OpenAIAi2 WildChat-4.8M (AI2)

please explain the following like i am a layperson: "" Search Write Sign up Sign in How…

  1. User

    please explain the following like i am a layperson: "" Search Write Sign up

    Sign in

    How to Fit Large Language Models in Small Memory: Quantization This article is part of the Langchain 101 Course Ivan Reznikov Towards AI Ivan Reznikov

    · Follow

    Published in Towards AI

    · 6 min read · Sep 26, 2023 58

    This article will discuss the theory and practice behind quantization and fitting a LLM on your PC. All the code is provided on Github

    Read more about LLMs in my Langchain 101 series:

    LangChain 101: Part 2ab. All You Need to Know About (Large Language) Models This article is part of the Langchain 101 Course pub.towardsai.net

    Large Language Models Large Language Models can be used for text generation, translation, question-answering tasks, etc. However, LLMs are also very large (obviously, Large language models) and require a lot of memory. This can make them challenging for small devices like phones and tablets.

    Multiply the parameters by the chosen precision size to determine the model size in bytes. Let’s say the precision we’ve chosen is float16 (16 bits = 2 bytes). Let’s say we want to use the BLOOM-176B model. We need 176 billion parameters * 2 bytes = 352GB to load the model!

    Large Language Models In other words, to load all parameter weights, we require 12(!) 32GB machines! This is too much if we ever want to make LLMs portable. Techniques for reducing the memory footprint of LLMs were developed to overcome such a struggle. The most popular techniques are:

    Quantization involves converting the LLM’s weights into a lower-precision format, reducing the memory required to store them. Knowledge distillation involves training a smaller LLM to mimic the behavior of a larger LLM. This can be done by transferring the knowledge from the larger LLM to the smaller LLM. These techniques have made it possible to fit LLMs in small memory. This has opened up new possibilities for using LLMs on various devices. Today, we’ll talk about quantization (stay tuned for knowledge distillation).

    Quantization Let’s start with a simple example. We’ll need to transform 2023 to binary:

    Converting int to binary As you can see, the process is relatively straightforward. In order to store the number 2023, we’ll need 12+ bits (1 bit for the + or — sign). For the number, we might use the int16 type.

    There is a big difference between storing int as binary and float as such. Let’s try converting 20.23 to binary:

    Converting float to binary As one can see, the floating part (mantissa) is calculated as the combination of 1/2^n, and cannot be calculated very precisely, even with 10 bits dedicated to the floating part. The whole number part (exponent) is set to 5 bits, covering all numbers up to 32. In total, we’re using 16 bits (FP16) for storing the closest we can to 20.23, but is it the most effective way to keep floats? What if the whole part number is much larger, say 202.3?

    If we look at the standard float types, we’ll notice that to store 202.3, we’ll need to use FP32, which, from a computational perspective, is far from reasonable. Instead, we can use a bfloat16 to save the range (exponent) as 8 bits and 7 bits for precision (mantissa). This allows us to widen the scope of possible decimals without losing much precision.

    FP32, FP16, BFloat16 data types To be clear, while training occurs, we need all the precision we can take. But prioritizing speed and size over the 6th decimal point makes sense for inference.

    Can we decrease the memory usage from bfloat16 to, say, int8?

    Zero-point and abs-max quantization In fact, we can, and there are several approaches for such quantization:

    Zero-point quantization saves half the memory by converting a fixed range (-1, 1) to int8 (-127, 127), followed by converting int8 back to bfloat16.

    Zero-point quantization Abs-max quantization is similar to zero-point, but instead of setting a custom range (-1,1), we set it as (-abs(max), abs(max)).

    Abs-max quantization Let’s take a look at how these practices are used in an example of matrix multiplication:

    Exact matrix multiplication Zero-point quantization:

    Zero-point quantization for matrix multiplication Abs-max quantization:

    Abs-max quantization for matrix multiplication As one can notice, the score for large values [-1579, -1780] is quite low ([-1579, -1752] for zero-point and [-1565,-1786] for abs-max). In order to overcome such issues, we can separate outlier multiplication:

    Separating outlier multiplication As you can see, the results are much closer to true values.

    But is there a way we can use even less space without losing much quality?

    To my very surprise, there is a way! What if we, instead of independently converting each number to a lower-type, would account for the error and use it for adjustments? This tecnique is called GPTQ.

    Like previous quantization, we find the closest match for the decimals we can, keeping the total conversion error as close to zero as possible.

    GPTQ approximation. Step 1 We fill the matrix row-wise in such a manner.

    GPTQ approximation. Step 2 The result, in combination with anomaly-separate calculations, provides quite okay results:

    GPTQ matrix multiplication with filtered outliers We can now compare all the methods:

    Result comparison LLM.int8() methods perform pretty well! The GPTQ approach loses quality but allows the use of twice as much GPU memory as the int8 method.

    In the code, you might find something similar to the following:

    from transformers import BitsAndBytesConfig

    Configure BitsAndBytesConfig for 4-bit quantization

    bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, )

    Loading model in pre-set configuration

    pretrained_model = AutoModelForCausalLM.from_pretrained( model_id, quantization_config=bnb_config, ) The load_in_4bit flag specifies that the model should be loaded in 4-bit precision. The bnb_4bit_use_double_quant flag specifies that double quantization should be used. The bnb_4bit_quant_type flag specifies the quantization type. The bnb_4bit_compute_dtype flag specifies the compute dtype.

    To sum up, we’ve learned how decimals are stored in memory, how to reduce memory footprint with some precision loss, and how to run selected models with 4-bit quantization.

    Afterword Check out how LLMs are used in langchain:

    LangChain 101: Part 2ab. All You Need to Know About (Large Language) Models This is part 2ab of the LangChain 101 course. It is strongly recommended to check the first part to understand the… pub.towardsai.net

    LangChain 101: Part 2c. Fine-tuning LLMs with PEFT, LORA, and RL All you need to know about fine-tuning llms, PEFT, LORA and training large language models pub.towardsai.ne

    I would appreciate your support if you’ve enjoyed the illustrations and the article content. Until next time!""

  2. Assistant

    Alright, let's explain this in a simple way!

    Imagine Large Language Models (LLMs) like enormous digital brains that help computers understand and use human language. They can do things like write stories, translate languages, and answer questions. However, these digital brains are so big that fitting them into smaller devices like your laptop or smartphone is like trying to fit an elephant into a fridge. This is where "quantization" steps in to help.

    Quantization is like a magic trick that makes the digital brain smaller without making it forget too much. It does this by simplifying the brain's thoughts (technically called "weights") into a less detailed format. Think of it as taking a high-definition photo and making it a bit blurrier so it takes up less space on your phone. The photo is still good enough, and you save memory.

    The article also mentions some specific ways to do this magic trick, like "Zero-point" and "Abs-max" quantization. Imagine you have a bag of marbles with numbers on them. Instead of using big numbers (which take up more space), you swap them out for smaller numbers but in a way that you can still understand what the big numbers were. This swapping trick allows the digital brain to take up less room.

    GPTQ is another method mentioned, which is like a more advanced magic trick. It not only swaps out big numbers for smaller ones but also does some smart adjustments to make sure the replacement is as accurate as possible, almost like fixing minor errors in the swapping process to ensure the bag of marbles (or the digital brain) still makes sense.

    The article wraps up by showing how, after all these tricks, we can make these enormous digital brains small enough to fit on smaller devices, like turning an elephant into a mini-elephant that can fit comfortably in your fridge (or in this case, allowing the LLMs to run smoothly on your laptop or phone).

    In a nutshell, quantization helps shrink these digital brains, making it possible to use them on devices with limited memory, which opens up new and exciting ways for us to interact with technology using natural language.

  3. User

    how accurate is the quantized model compared to the unquantized model? i.e. how much information (as a percentage) does the model lose?

  4. Assistant

    The accuracy of a quantized model compared to an unquantized (full-precision) model can vary greatly depending on several factors, including the specific quantization technique used, the nature of the model, and the task it is performing. However, I'll give you a general idea to help understand the potential impact of quantization on model performance.

    Quantization reduces the precision of the model's parameters (weights and activations) by representing them with fewer bits. For example, reducing from 32-bit floating-point numbers (float32) to 8-bit integers (int8) significantly decreases the model's memory footprint and can speed up computation, especially on hardware optimized for lower-precision arithmetic. But, this also means that the model might not capture nuances as finely as it did before quantization.

    In practice, the accuracy loss due to quantization can sometimes be surprisingly small, often within a few percentage points of the original model. For many tasks, especially large models and those not requiring extreme precision, a 1-3% loss in accuracy might be acceptable in exchange for the drastic reduction in memory usage and computational efficiency. For example, after quantization, a model's accuracy might drop from 95% to 93.5%, which is quite minimal considering the benefits.

    The key here is the trade-off: You're balancing between efficiency (in terms of memory and speed) and accuracy. Some advanced quantization techniques, such as quantization-aware training or adaptive quantization, can minimize the loss of accuracy. These methods fine-tune the model with quantization in mind or adjust the quantization parameters to better suit the model's distribution of values, respectively.

    In summary, while it's hard to give a one-size-fits-all percentage for how much information a model loses when quantized, most well-applied quantization techniques result in relatively minor losses in accuracy that are often deemed acceptable for the benefits of reduced size and increased speed. The exact impact can only be determined by testing the specific model on its intended tasks both before and after quantization.