Hey, can you explain the following code to me? It's inference code for "GPTQ-for-LLaMa",…
- User
Hey, can you explain the following code to me? It's inference code for "GPTQ-for-LLaMa", a branch of GPTQ for Meta's LLM so it can be quantized to 4 bit precision. I am not really interested in quantizing myself, but I'd like to know how I can run inference myself using PyTorch/HF Transformers on such a 4-bit quantized LLaMA model. Thanks!
import time
import torch import torch.nn as nn
from gptq import * from modelutils import * import quant
from transformers import AutoTokenizer
DEV = torch.device('cuda:0')
def get_llama(model): import torch def skip(*args, **kwargs): pass torch.nn.init.kaiming_uniform_ = skip torch.nn.init.uniform_ = skip torch.nn.init.normal_ = skip from transformers import LlamaForCausalLM model = LlamaForCausalLM.from_pretrained(model, torch_dtype='auto') model.seqlen = 2048 return model
def load_quant(model, checkpoint, wbits, groupsize = -1, fused_mlp = True, eval=True, warmup_autotune = True): from transformers import LlamaConfig, LlamaForCausalLM config = LlamaConfig.from_pretrained(model) def noop(*args, **kwargs): pass torch.nn.init.kaiming_uniform_ = noop torch.nn.init.uniform_ = noop torch.nn.init.normal_ = noop
torch.set_default_dtype(torch.half) transformers.modeling_utils._init_weights = False torch.set_default_dtype(torch.half) model = LlamaForCausalLM(config) torch.set_default_dtype(torch.float) if eval: model = model.eval() layers = find_layers(model) for name in ['lm_head']: if name in layers: del layers[name] quant.make_quant_linear(model, layers, wbits, groupsize) del layers print('Loading model ...') if checkpoint.endswith('.safetensors'): from safetensors.torch import load_file as safe_load model.load_state_dict(safe_load(checkpoint), strict = False) else: model.load_state_dict(torch.load(checkpoint), strict = False) quant.make_quant_attn(model) if eval and fused_mlp: quant.make_fused_mlp(model) if warmup_autotune: quant.autotune_warmup_linear(model,transpose=not(eval)) if eval and fused_mlp: quant.autotune_warmup_fused(model) model.seqlen = 2048 print('Done.') return modelif name == 'main': import argparse from datautils import *
parser = argparse.ArgumentParser() parser.add_argument( 'model', type=str, help='llama model to load' ) parser.add_argument( '--wbits', type=int, default=16, choices=[2, 3, 4, 8, 16], help='#bits to use for quantization; use 16 for evaluating base model.' ) parser.add_argument( '--groupsize', type=int, default=-1, help='Groupsize to use for quantization; default uses full row.' ) parser.add_argument( '--load', type=str, default='', help='Load quantized model.' ) parser.add_argument( '--text', type=str, help='input text' ) parser.add_argument( '--min_length', type=int, default=10, help='The minimum length of the sequence to be generated.' ) parser.add_argument( '--max_length', type=int, default=50, help='The maximum length of the sequence to be generated.' ) parser.add_argument( '--top_p', type=float , default=0.95, help='If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation.' ) parser.add_argument( '--temperature', type=float, default=0.8, help='The value used to module the next token probabilities.' ) parser.add_argument( '--device', type=int, default=-1, help='The device used to load the model when using safetensors. Default device is "cpu" or specify, 0,1,2,3,... for GPU device.' ) args = parser.parse_args() if type(args.load) is not str: args.load = args.load.as_posix() if args.load: model = load_quant(args.model, args.load, args.wbits, args.groupsize, args.device) else: model = get_llama(args.model) model.eval() model.to(DEV) tokenizer = AutoTokenizer.from_pretrained(args.model, use_fast=False) input_ids = tokenizer.encode(args.text, return_tensors="pt").to(DEV) with torch.no_grad(): generated_ids = model.generate( input_ids, do_sample=True, min_length=args.min_length, max_length=args.max_length, top_p=args.top_p, temperature=args.temperature, ) print(tokenizer.decode([el.item() for el in generated_ids[0]])) - Assistant
This code is for loading and running inference using a GPTQ-for-LLaMa model quantized to 4-bit precision. Let me break down the main sections of the code and their purpose.
-
Importing necessary libraries and setting the device: The code imports libraries such as PyTorch, transformers, etc., and sets the device for running the model (in this case, a GPU is used).
-
get_llamafunction: This function is used to load the LLaMa model using Hugging Face Transformers. -
load_quantfunction: This function is responsible for loading a quantized version of the LLaMa model given its checkpoint. It performs several operations such as modifying the layers for quantization, loading the checkpoint, applying quantization on attention and linear layers, and so on. -
Main execution block (
if __name__ == '__main__':): This block deals with parsing command-line arguments, loading the LLaMa model either as a base model or quantized version based on the provided arguments, loading the tokenizer, and performing text generation using the model.
Given your interest in running inference with a 4-bit quantized LLaMa model, you'll run this script with appropriate command-line arguments such as the model name, bit-width, checkpoint path, input text, and device.
For example, if you save this code in a file named
inference.py, you can use the following command to run inference with your desired model:python inference.py <model_name> --wbits 4 --load <path_to_checkpoint> --text "Your input text" --device 0Replace
<model_name>with the name of your GPTQ-for-LLaMa model,<path_to_checkpoint>with the path of the quantized model checkpoint, and adjust other input arguments as needed. -
- User
Can you explain DEV to me, and the "control flow"? I'm thinking about maybe I only need the last section of the code (+ imports of course).
- Assistant
Sure, let me explain the
DEVvariable and the control flow in the last section of the code.DEVis a variable that represents the device on which you want to run the model. In this script,DEVis set totorch.device('cuda:0'), which means the model will be run on the first available GPU. You can also set it to other GPU indices (e.g.,cuda:1) if you have more than one GPU, or totorch.device('cpu')if you want to run the model on the CPU.As for the control flow in the last section of the code (main execution block):
-
Command-line arguments are parsed using
argparse. These include model name, bit-width for quantization, path to the quantized model checkpoint, input text, and the device on which to run the model. -
Based on the provided arguments, either the base LLaMa model or the quantized version is loaded. If the
--loadargument is provided, theload_quantfunction is called to load the quantized model. Otherwise, the base model is loaded using theget_llamafunction. -
The model is moved to the target device specified by
DEVusingmodel.to(DEV). -
The tokenizer for the LLaMa model is loaded using Hugging Face Transformers.
-
The input text provided as a command-line argument is tokenized and converted to input IDs using the tokenizer. These input IDs are then moved to the target device.
-
The
model.generatefunction is called with the input IDs and generation settings (such as min_length, max_length, top_p, and temperature) to generate the output text. -
The generated output text is decoded back into human-readable text using the tokenizer and finally printed.
If you are only interested in running inference with a pre-quantized model, you will indeed require only the imports, the last section of the code for running inference (main execution block), and the
load_quantandget_llamafunctions to load the model. You can discard some parts of the code that are not relevant to your use-case, such as the argparse arguments and argument loading details. Adjust the parameters and input text as needed in the code directly instead of providing them as command-line arguments. -
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.