Help i keep getting this error after running ONEAPIDEVICESELECTOR="levelzero:0"…
- User
Help i keep getting this error after running ONEAPI_DEVICE_SELECTOR="level_zero:0" ZES_ENABLE_SYSMAN=1 accelerate launch g7.py
File "/home/sdp/.local/lib/python3.11/site-packages/torch/nn/functional.py", line 2264, in embedding return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and xpu:0! (when checking argument for argument index in method wrapper_XPU__index_select) ^CKeyboard interruption in main thread... closing server. Killing tunnel 127.0.0.1:7860 <> https://2919925b734be25df0.gradio.live Traceback (most recent call last): File "/home/sdp/.local/bin/accelerate", line 8, in sys.exit(main()) ^^^^^^
tell me how can i make my script run well across all 8 XPU accelerator intel 1550 gpus on this machine
import gradio as gr import torch from transformers import AutoTokenizer, AutoModelForCausalLM
Initialize the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")
Load the model from safetensors format
model = AutoModelForCausalLM.from_pretrained( "/scratch-1/q14", # Update this path to your actual model path torch_dtype=torch.float16, device_map="auto", # This will distribute across all available GPUs automatically trust_remote_code=True ).eval()
Function to generate text using the model
def generate_text(prompt, max_tokens=50, temperature=0.7, top_p=0.9): # Prepare the input prompt messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} ]
# Tokenize the input prompt input_text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) inputs = tokenizer(input_text, return_tensors="pt").to("xpu") # Move input to XPU # Generate output using model with torch.no_grad(): outputs = model.generate( inputs["input_ids"], max_length=max_tokens + len(inputs["input_ids"][0]), temperature=temperature, top_p=top_p, repetition_penalty=1.05, pad_token_id=tokenizer.eos_token_id ) # Decode the generated tokens generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True) return generated_textCreate the Gradio interface
iface = gr.Interface( fn=generate_text, inputs=[ gr.Textbox(lines=2, placeholder="Enter your prompt here...", label="Prompt"), gr.Slider(minimum=1, maximum=512, value=50, label="Max Tokens"), gr.Slider(minimum=0.1, maximum=1.0, value=0.7, label="Temperature"), gr.Slider(minimum=0.1, maximum=1.0, value=0.9, label="Top P") ], outputs=gr.Textbox(label="Generated Text"), title="Qwen 2.5 14B Instruct Text Generator", description="Generate text using the Qwen 2.5 14B Instruct model with transformers.", )
Launch the interface
iface.launch(share=True)
- Assistant
The error you're encountering is due to a device mismatch: some tensors are on the CPU while others are on the XPU (
xpu:0). To run your script across all 8 Intel XPU accelerators (Intel 1550 GPUs) on your machine, you'll need to ensure that:- All relevant tensors are consistently placed on the XPU devices.
- You're correctly configuring
accelerateto utilize multiple XPU devices.
Here's a step-by-step guide to resolve the issue and optimize your script for multi-device execution:
1. Update Environment Variables to Select All XPUs
Currently, your
ONEAPI_DEVICE_SELECTORis set to"level_zero:0", which only selects the first XPU device. To utilize all 8 XPU devices, you should specify each device in the selector. Here's how you can modify it:export ONEAPI_DEVICE_SELECTOR="level_zero:0,level_zero:1,level_zero:2,level_zero:3,level_zero:4,level_zero:5,level_zero:6,level_zero:7" export ZES_ENABLE_SYSMAN=1Explanation:
ONEAPI_DEVICE_SELECTOR: This environment variable specifies which XPU devices to use. By listing all devices (level_zero:0throughlevel_zero:7), you're enablingaccelerateto recognize and utilize all 8 devices.ZES_ENABLE_SYSMAN=1: This enables Level Zero system management, which is necessary for monitoring and managing the XPU devices.
You can set these variables directly in your shell before running your script or include them in your launch command.
2. Configure
acceleratefor Multi-Device SetupBefore proceeding, ensure that
accelerateis properly configured to handle multiple devices. If you haven't already initialized anaccelerateconfiguration, do so now:accelerate configDuring the configuration:
- Compute Environment: Choose
Multi-Processingor the appropriate option that suits your setup. - Distributed Type: Select
Multi-GPUorMulti-Acceleratorbased on what's available. - Number of Processes: Set this to
8to match your 8 XPU devices. - Processes Per Host: Typically
1unless you have a specific multi-host setup. - Main Device Port: Leave as default unless there's a port conflict.
- Use FP16?: Yes, since you're using
torch.float16.
This sets up
accelerateto manage multiple devices effectively.3. Modify Your Python Script to Leverage
accelerateTo ensure all tensors are placed on the correct devices and to utilize multiple accelerators, integrate
accelerate'sAcceleratorclass into your script. Here's how you can modify your script:import gradio as gr import torch from transformers import AutoTokenizer, AutoModelForCausalLM from accelerate import Accelerator # Initialize the accelerator accelerator = Accelerator() # Initialize the tokenizer and model tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct") # Load the model from safetensors format model = AutoModelForCausalLM.from_pretrained( "/scratch-1/q14", # Update this path to your actual model path torch_dtype=torch.float16, device_map="auto", # This will distribute across all available XPU devices trust_remote_code=True ).eval() # Prepare the model with accelerator model = accelerator.prepare(model) # Function to generate text using the model def generate_text(prompt, max_tokens=50, temperature=0.7, top_p=0.9): # Prepare the input prompt messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} ] # Tokenize the input prompt input_text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) # Tokenize and prepare inputs inputs = tokenizer( input_text, return_tensors="pt" ) # Move inputs to the appropriate device(s) inputs = {k: v.to(accelerator.device) for k, v in inputs.items()} # Generate output using model with torch.no_grad(): outputs = model.generate( inputs["input_ids"], max_length=max_tokens + inputs["input_ids"].shape[1], temperature=temperature, top_p=top_p, repetition_penalty=1.05, pad_token_id=tokenizer.eos_token_id ) # Decode the generated tokens generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True) return generated_text # Create the Gradio interface iface = gr.Interface( fn=generate_text, inputs=[ gr.Textbox(lines=2, placeholder="Enter your prompt here...", label="Prompt"), gr.Slider(minimum=1, maximum=512, value=50, label="Max Tokens"), gr.Slider(minimum=0.1, maximum=1.0, value=0.7, label="Temperature"), gr.Slider(minimum=0.1, maximum=1.0, value=0.9, label="Top P") ], outputs=gr.Textbox(label="Generated Text"), title="Qwen 2.5 14B Instruct Text Generator", description="Generate text using the Qwen 2.5 14B Instruct model with transformers.", ) # Launch the interface using accelerator's launch method if necessary if __name__ == "__main__": iface.launch(share=True)Key Modifications:
-
Initialize the Accelerator:
from accelerate import Accelerator accelerator = Accelerator()This sets up the
Acceleratorobject to manage device placement and distributed training. -
Prepare the Model with Accelerator:
model = accelerator.prepare(model)This ensures that the model is correctly distributed across all available XPU devices as per the
accelerateconfiguration. -
Move Inputs to the Correct Device: Instead of manually moving inputs to
"xpu", use the accelerator's device:inputs = {k: v.to(accelerator.device) for k, v in inputs.items()}This ensures that all input tensors are placed on the same device(s) as the model.
-
Launch with
accelerate: Ensure you're launching the script usingaccelerate launchto leverage the multi-device setup:accelerate launch g7.pySince you set the environment variables earlier and configured
accelerate, this command will handle the multi-device distribution.
4. Run the Script
With the environment variables set and the script modified, run your script using the
acceleratelauncher:ONEAPI_DEVICE_SELECTOR="level_zero:0,level_zero:1,level_zero:2,level_zero:3,level_zero:4,level_zero:5,level_zero:6,level_zero:7" ZES_ENABLE_SYSMAN=1 accelerate launch g7.pyAlternative: If you've exported the environment variables in your shell, you can simply use:
accelerate launch g7.py5. Additional Tips
-
Verify Device Allocation: Ensure that all XPUs are being utilized. You can add logging to check where different parts of the model are placed.
-
Monitor Resource Usage: Use Intel's profiling tools or other system monitors to verify that all 8 XPUs are actively being used during model inference.
-
Batch Processing: For optimal performance across multiple devices, consider implementing batch processing of prompts if applicable.
-
Error Handling: If you encounter further errors related to device placement, consider using
accelerator.print(...)for debugging or consult theacceleratedocumentation for advanced configurations.
By following these steps, you should be able to resolve the device mismatch error and efficiently run your script across all 8 Intel XPU accelerators on your machine.