Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0125-preview· openaiHugging Face Suzhen/CodeChat

CUDAVISIBLEDEVICES=0,1,2,3 python -m torch.distributed.launch imagetrainh5.py…

  1. User

    CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch image_train_h5.py /home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/launch.py:183: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use-env is set by default in torchrun. If your script expects --local-rank argument to be set, please change it to read from os.environ['LOCAL_RANK'] instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions

    warnings.warn( usage: image_train_h5.py [-h] [--workers WORKERS] [--train_test_h5_path TRAIN_TEST_H5_PATH] [--val_test_h5_path VAL_TEST_H5_PATH] [--model_outpath MODEL_OUTPATH] [--lr LR] [--model MODEL] [--seed SEED] [--batch_size BATCH_SIZE] [--epochs EPOCHS] [--decay DECAY] [--milestone MILESTONE] [--local_rank LOCAL_RANK] [--nproc_per_node NPROC_PER_NODE] image_train_h5.py: error: unrecognized arguments: --local-rank=0 [2024-03-27 21:38:22,700] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 2) local_rank: 0 (pid: 443088) of binary: /home/amax/anaconda3/bin/python Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/launch.py", line 198, in main() File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/launch.py", line 194, in main launch(args) File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/launch.py", line 179, in launch run(args) File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 135, in call return launch_agent(self._config, self._entrypoint, list(args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

    image_train_h5.py FAILED

    Failures: <NO_OTHER_FAILURES>

    Root Cause (first observed failure): [0]: time : 2024-03-27_21:38:22 host : amax rank : 0 (local_rank: 0) exitcode : 2 (pid: 443088) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

    这是什么错误,应该怎么解决

  2. Assistant

    此错误提示来自于尝试在使用PyTorch分布式训练时遇到的几个问题。这里有一个解决方案的步骤:

    1. 弃用警告: torch.distributed.launch已被弃用,并将在未来版本中删除。PyTorch官方建议改用torchrun。因此,您应该避免使用python -m torch.distributed.launch来启动分布式训练,而应使用torchrun命令。

    2. --local_rank的使用: 旧的torch.distributed.launch工具需要使用--local_rank参数来手动传递每个进程的本地排名。但在torchrun中,默认情况下使用环境变量LOCAL_RANK来替代。因此,您应该确保您的代码内部是从os.environ['LOCAL_RANK']读取本地排名的,而不是期望从命令行参数获取。

    结合您的错误提示,您可以按照以下步骤尝试解决这个问题:

    调整您的代码

    • 确保代码中使用os.environ['LOCAL_RANK']获取本地排名,例如:

      import os
      
      local_rank = int(os.environ['LOCAL_RANK'])
      

    使用torchrun代替torch.distributed.launch

    • 根据您之前的命令,使用torchrun的等效形式可能如下:

      CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 image_train_h5.py [其他参数]
      

      这里--nproc_per_node=4表示您希望在每个节点上启动4个进程。请确保将[其他参数]替换为您想要传给image_train_h5.py的其他任何命令行参数(如果有的话)。

    • 注意,CUDA_VISIBLE_DEVICES=0,1,2,3这部分是对的,这限制了可用的GPU设备。

    最后的注意

    • 在执行新命令前,确保您已就绪并且符合任何版本要求。torchrun从PyTorch 1.9开始引入,确保您的PyTorch版本至少为1.9或更高。

    • 如果您的脚本还是不工作,检查脚本是否适配于分布式训练,例如使用合适的初始化方法、正确的数据并行处理等。

    通过这些步骤,您应该能够解决上述错误并使用PyTorch的新分布式训练工具torchrun成功运行您的训练脚本。

  3. User

    Traceback (most recent call last): File "/home/amax/PHD/Jiahao_Wang16/CT_MAR/Res_unet_image_train/image_train_h5.py", line 132, in torch.cuda.set_device(local_rank) File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/cuda/init.py", line 408, in set_device torch._C._cuda_setDevice(device) RuntimeError: CUDA error: invalid device ordinal CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

    Traceback (most recent call last): File "/home/amax/PHD/Jiahao_Wang16/CT_MAR/Res_unet_image_train/image_train_h5.py", line 132, in torch.cuda.set_device(local_rank) File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/cuda/init.py", line 408, in set_device torch._C._cuda_setDevice(device) RuntimeError: CUDA error: invalid device ordinal CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

    [2024-03-27 21:46:06,669] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 447402 closing signal SIGTERM [2024-03-27 21:46:06,884] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 447403) of binary: /home/amax/anaconda3/bin/python Traceback (most recent call last): File "/home/amax/anaconda3/bin/torchrun", line 8, in sys.exit(main()) ^^^^^^ File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 347, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 812, in main run(args) File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 803, in run elastic_launch( File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 135, in call return launch_agent(self._config, self._entrypoint, list(args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/amax/anaconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 268, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

    image_train_h5.py FAILED

    Failures: [1]: time : 2024-03-27_21:46:06 host : amax rank : 2 (local_rank: 2) exitcode : 1 (pid: 447404) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2024-03-27_21:46:06 host : amax rank : 3 (local_rank: 3) exitcode : 1 (pid: 447405) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

    Root Cause (first observed failure): [0]: time : 2024-03-27_21:46:06 host : amax rank : 1 (local_rank: 1) exitcode : 1 (pid: 447403) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

  4. Assistant

    这个错误信息指出的问题是关于无效的设备序号(Invalid device ordinal),这通常意味着你的代码试图访问一个不存在的CUDA设备。这种情况通常发生于当你尝试在一个有限数量的GPU上运行分布式程序,但程序却试图访问超出这个范围的设备时。

    例如,如果你的系统中只有2个GPU(设备序号为0和1),但是你试图设置设备为2或更高的设备序号,你就会遇到这个错误。

    要解决这个问题,你需要确保几个方面:

    1. 确认GPU数量:首先,验证你机器上可用的GPU数量。你可以通过运行nvidia-smi命令来查看。确保你没有试图使用超过这个数量的GPU。

    2. 适当设置CUDA_VISIBLE_DEVICES:如果你使用的是CUDA_VISIBLE_DEVICES环境变量来限制可用的GPU,确保你没有设置超过实际存在的GPU序号。例如,如果你有2个GPU,你应当只设置CUDA_VISIBLE_DEVICES=0,1。

    3. 检查代码:如果你在代码中显式地设置了设备(例如,torch.cuda.set_device(local_rank)),确保local_rank没有超出可用设备的范围。你可能需要根据CUDA_VISIBLE_DEVICES环境变量或其他逻辑来动态地确定local_rank的值,以确保它不会超出实际可用的GPU数量。

    4. torchrun的使用:当使用torchrun时,通过--nproc_per_node指定每个节点的进程数应该匹配你希望使用的GPU数量。例如,如果你有2个GPU并希望每个GPU上运行一个进程,你应当设置--nproc_per_node=2。

    基于给出的错误和情况,你可以尝试检查并修改代码,例如:

    CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 image_train_h5.py [其他参数]
    

    上面的命令仅当你有两个GPU时有效,且以每个节点上两个进程来执行你的训练脚本,每个进程对应一个GPU。

    确保你根据实际情况调整CUDA_VISIBLE_DEVICES和--nproc_per_node的值。

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.