Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

Help i keep getting this error after running ONEAPIDEVICESELECTOR="levelzero:0"…

  1. User

    Help i keep getting this error after running ONEAPI_DEVICE_SELECTOR="level_zero:0" ZES_ENABLE_SYSMAN=1 accelerate launch g7.py

    File "/home/sdp/.local/lib/python3.11/site-packages/torch/nn/functional.py", line 2264, in embedding return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and xpu:0! (when checking argument for argument index in method wrapper_XPU__index_select) ^CKeyboard interruption in main thread... closing server. Killing tunnel 127.0.0.1:7860 <> https://2919925b734be25df0.gradio.live Traceback (most recent call last): File "/home/sdp/.local/bin/accelerate", line 8, in sys.exit(main()) ^^^^^^

    tell me how can i make my script run well across all 8 XPU accelerator intel 1550 gpus on this machine

    import gradio as gr import torch from transformers import AutoTokenizer, AutoModelForCausalLM

    Initialize the tokenizer and model

    tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")

    Load the model from safetensors format

    model = AutoModelForCausalLM.from_pretrained( "/scratch-1/q14", # Update this path to your actual model path torch_dtype=torch.float16, device_map="auto", # This will distribute across all available GPUs automatically trust_remote_code=True ).eval()

    Function to generate text using the model

    def generate_text(prompt, max_tokens=50, temperature=0.7, top_p=0.9): # Prepare the input prompt messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} ]

    # Tokenize the input prompt
    input_text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )
    
    inputs = tokenizer(input_text, return_tensors="pt").to("xpu")  # Move input to XPU
    
    # Generate output using model
    with torch.no_grad():
        outputs = model.generate(
            inputs["input_ids"],
            max_length=max_tokens + len(inputs["input_ids"][0]),
            temperature=temperature,
            top_p=top_p,
            repetition_penalty=1.05,
            pad_token_id=tokenizer.eos_token_id
        )
    
    # Decode the generated tokens
    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return generated_text
    

    Create the Gradio interface

    iface = gr.Interface( fn=generate_text, inputs=[ gr.Textbox(lines=2, placeholder="Enter your prompt here...", label="Prompt"), gr.Slider(minimum=1, maximum=512, value=50, label="Max Tokens"), gr.Slider(minimum=0.1, maximum=1.0, value=0.7, label="Temperature"), gr.Slider(minimum=0.1, maximum=1.0, value=0.9, label="Top P") ], outputs=gr.Textbox(label="Generated Text"), title="Qwen 2.5 14B Instruct Text Generator", description="Generate text using the Qwen 2.5 14B Instruct model with transformers.", )

    Launch the interface

    iface.launch(share=True)

  2. Assistant

    The error you're encountering is due to a device mismatch: some tensors are on the CPU while others are on the XPU (xpu:0). To run your script across all 8 Intel XPU accelerators (Intel 1550 GPUs) on your machine, you'll need to ensure that:

    1. All relevant tensors are consistently placed on the XPU devices.
    2. You're correctly configuring accelerate to utilize multiple XPU devices.

    Here's a step-by-step guide to resolve the issue and optimize your script for multi-device execution:

    1. Update Environment Variables to Select All XPUs

    Currently, your ONEAPI_DEVICE_SELECTOR is set to "level_zero:0", which only selects the first XPU device. To utilize all 8 XPU devices, you should specify each device in the selector. Here's how you can modify it:

    export ONEAPI_DEVICE_SELECTOR="level_zero:0,level_zero:1,level_zero:2,level_zero:3,level_zero:4,level_zero:5,level_zero:6,level_zero:7"
    export ZES_ENABLE_SYSMAN=1
    

    Explanation:

    • ONEAPI_DEVICE_SELECTOR: This environment variable specifies which XPU devices to use. By listing all devices (level_zero:0 through level_zero:7), you're enabling accelerate to recognize and utilize all 8 devices.
    • ZES_ENABLE_SYSMAN=1: This enables Level Zero system management, which is necessary for monitoring and managing the XPU devices.

    You can set these variables directly in your shell before running your script or include them in your launch command.

    2. Configure accelerate for Multi-Device Setup

    Before proceeding, ensure that accelerate is properly configured to handle multiple devices. If you haven't already initialized an accelerate configuration, do so now:

    accelerate config
    

    During the configuration:

    • Compute Environment: Choose Multi-Processing or the appropriate option that suits your setup.
    • Distributed Type: Select Multi-GPU or Multi-Accelerator based on what's available.
    • Number of Processes: Set this to 8 to match your 8 XPU devices.
    • Processes Per Host: Typically 1 unless you have a specific multi-host setup.
    • Main Device Port: Leave as default unless there's a port conflict.
    • Use FP16?: Yes, since you're using torch.float16.

    This sets up accelerate to manage multiple devices effectively.

    3. Modify Your Python Script to Leverage accelerate

    To ensure all tensors are placed on the correct devices and to utilize multiple accelerators, integrate accelerate's Accelerator class into your script. Here's how you can modify your script:

    import gradio as gr
    import torch
    from transformers import AutoTokenizer, AutoModelForCausalLM
    from accelerate import Accelerator
    
    # Initialize the accelerator
    accelerator = Accelerator()
    
    # Initialize the tokenizer and model
    tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")
    
    # Load the model from safetensors format
    model = AutoModelForCausalLM.from_pretrained(
        "/scratch-1/q14",  # Update this path to your actual model path
        torch_dtype=torch.float16,
        device_map="auto",  # This will distribute across all available XPU devices
        trust_remote_code=True
    ).eval()
    
    # Prepare the model with accelerator
    model = accelerator.prepare(model)
    
    # Function to generate text using the model
    def generate_text(prompt, max_tokens=50, temperature=0.7, top_p=0.9):
        # Prepare the input prompt
        messages = [
            {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
            {"role": "user", "content": prompt}
        ]
    
        # Tokenize the input prompt
        input_text = tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True
        )
    
        # Tokenize and prepare inputs
        inputs = tokenizer(
            input_text,
            return_tensors="pt"
        )
    
        # Move inputs to the appropriate device(s)
        inputs = {k: v.to(accelerator.device) for k, v in inputs.items()}
    
        # Generate output using model
        with torch.no_grad():
            outputs = model.generate(
                inputs["input_ids"],
                max_length=max_tokens + inputs["input_ids"].shape[1],
                temperature=temperature,
                top_p=top_p,
                repetition_penalty=1.05,
                pad_token_id=tokenizer.eos_token_id
            )
    
        # Decode the generated tokens
        generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
        return generated_text
    
    # Create the Gradio interface
    iface = gr.Interface(
        fn=generate_text,
        inputs=[
            gr.Textbox(lines=2, placeholder="Enter your prompt here...", label="Prompt"),
            gr.Slider(minimum=1, maximum=512, value=50, label="Max Tokens"),
            gr.Slider(minimum=0.1, maximum=1.0, value=0.7, label="Temperature"),
            gr.Slider(minimum=0.1, maximum=1.0, value=0.9, label="Top P")
        ],
        outputs=gr.Textbox(label="Generated Text"),
        title="Qwen 2.5 14B Instruct Text Generator",
        description="Generate text using the Qwen 2.5 14B Instruct model with transformers.",
    )
    
    # Launch the interface using accelerator's launch method if necessary
    if __name__ == "__main__":
        iface.launch(share=True)
    

    Key Modifications:

    1. Initialize the Accelerator:

      from accelerate import Accelerator
      accelerator = Accelerator()
      

      This sets up the Accelerator object to manage device placement and distributed training.

    2. Prepare the Model with Accelerator:

      model = accelerator.prepare(model)
      

      This ensures that the model is correctly distributed across all available XPU devices as per the accelerate configuration.

    3. Move Inputs to the Correct Device: Instead of manually moving inputs to "xpu", use the accelerator's device:

      inputs = {k: v.to(accelerator.device) for k, v in inputs.items()}
      

      This ensures that all input tensors are placed on the same device(s) as the model.

    4. Launch with accelerate: Ensure you're launching the script using accelerate launch to leverage the multi-device setup:

      accelerate launch g7.py
      

      Since you set the environment variables earlier and configured accelerate, this command will handle the multi-device distribution.

    4. Run the Script

    With the environment variables set and the script modified, run your script using the accelerate launcher:

    ONEAPI_DEVICE_SELECTOR="level_zero:0,level_zero:1,level_zero:2,level_zero:3,level_zero:4,level_zero:5,level_zero:6,level_zero:7" ZES_ENABLE_SYSMAN=1 accelerate launch g7.py
    

    Alternative: If you've exported the environment variables in your shell, you can simply use:

    accelerate launch g7.py
    

    5. Additional Tips

    • Verify Device Allocation: Ensure that all XPUs are being utilized. You can add logging to check where different parts of the model are placed.

    • Monitor Resource Usage: Use Intel's profiling tools or other system monitors to verify that all 8 XPUs are actively being used during model inference.

    • Batch Processing: For optimal performance across multiple devices, consider implementing batch processing of prompts if applicable.

    • Error Handling: If you encounter further errors related to device placement, consider using accelerator.print(...) for debugging or consult the accelerate documentation for advanced configurations.

    By following these steps, you should be able to resolve the device mismatch error and efficiently run your script across all 8 Intel XPU accelerators on your machine.