from unsloth import FastLanguageModel from transformers import AutoModelForCausalLM from…
- User
from unsloth import FastLanguageModel from transformers import AutoModelForCausalLM from peft import LoftQConfig, LoraConfig, get_peft_model import torch max_seq_length = 8192 # Choose any! We auto support RoPE Scaling internally! dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+ load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.
4bit pre quantized models we support for 4x faster downloading + no OOMs.
fourbit_models = [ "unsloth/Meta-Llama-3.1-8B-bnb-4bit", # Llama-3.1 15 trillion tokens model 2x faster! "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit", "unsloth/Meta-Llama-3.1-70B-bnb-4bit", "unsloth/Meta-Llama-3.1-405B-bnb-4bit", # We also uploaded 4bit for 405b! "unsloth/Mistral-Nemo-Base-2407-bnb-4bit", # New Mistral 12b 2x faster! "unsloth/Mistral-Nemo-Instruct-2407-bnb-4bit", "unsloth/mistral-7b-v0.3-bnb-4bit", # Mistral v3 2x faster! "unsloth/mistral-7b-instruct-v0.3-bnb-4bit", "unsloth/Phi-3.5-mini-instruct", # Phi-3.5 2x faster! "unsloth/Phi-3-medium-4k-instruct", "unsloth/gemma-2-9b-bnb-4bit", "unsloth/gemma-2-27b-bnb-4bit", # Gemma 2x faster! ] # More models at https://huggingface.co/unsloth
model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Mistral-Nemo-Base-2407-bnb-4bit", max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf )
How to set float16?
- Assistant
To explicitly set the
dtypetofloat16in your code, you need to specifytorch.float16as thedtypeparameter when callingFastLanguageModel.from_pretrained. This ensures that the model uses 16-bit floating-point precision, which can reduce memory usage and improve performance on compatible hardware (e.g., GPUs like Tesla T4 or V100).Here’s how you can modify your code to set
float16:from unsloth import FastLanguageModel from transformers import AutoModelForCausalLM from peft import LoftQConfig, LoraConfig, get_peft_model import torch max_seq_length = 8192 # Choose any! We auto support RoPE Scaling internally! dtype = torch.float16 # Explicitly set to float16 load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False. # 4bit pre quantized models we support for 4x faster downloading + no OOMs. fourbit_models = [ "unsloth/Meta-Llama-3.1-8B-bnb-4bit", # Llama-3.1 15 trillion tokens model 2x faster! "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit", "unsloth/Meta-Llama-3.1-70B-bnb-4bit", "unsloth/Meta-Llama-3.1-405B-bnb-4bit", # We also uploaded 4bit for 405b! "unsloth/Mistral-Nemo-Base-2407-bnb-4bit", # New Mistral 12b 2x faster! "unsloth/Mistral-Nemo-Instruct-2407-bnb-4bit", "unsloth/mistral-7b-v0.3-bnb-4bit", # Mistral v3 2x faster! "unsloth/mistral-7b-instruct-v0.3-bnb-4bit", "unsloth/Phi-3.5-mini-instruct", # Phi-3.5 2x faster! "unsloth/Phi-3-medium-4k-instruct", "unsloth/gemma-2-9b-bnb-4bit", "unsloth/gemma-2-27b-bnb-4bit", # Gemma 2x faster! ] # More models at https://huggingface.co/unsloth model, tokenizer = FastLanguageModel.from_pretrained( model_name="unsloth/Mistral-Nemo-Base-2407-bnb-4bit", max_seq_length=max_seq_length, dtype=dtype, # This will now use float16 load_in_4bit=load_in_4bit, # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf )Key Notes:
-
Hardware Compatibility: Ensure your hardware supports
float16(e.g., NVIDIA GPUs with Tensor Cores like V100, T4, or newer). If you're using a CPU or unsupported hardware, you may encounter errors or performance degradation. -
Automatic Detection: When
dtype=None, the library automatically selects the best precision based on your hardware (e.g.,float16for Tesla T4/V100,bfloat16for Ampere GPUs). Explicitly settingdtype=torch.float16overrides this behavior. -
4-bit Quantization: You're already using
load_in_4bit=True, which applies 4-bit quantization to further reduce memory usage. This is compatible withfloat16, but note that quantization and precision settings serve different purposes:load_in_4bit=Truereduces model weights to 4-bit integers.dtype=torch.float16controls the precision of computations and activations.
-
Alternative Precision: If you want to use
bfloat16(better for training stability on Ampere GPUs), you can setdtype=torch.bfloat16instead.
By setting
dtype=torch.float16, your model will use 16-bit floating-point precision, which is ideal for memory efficiency and performance on supported hardware. -
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.