这是什么错误masteraddr is only used for static rdzvbackend and when rdzv_endpoint is not…
- User
这是什么错误master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.
- Assistant
这是错误提示,表示在使用动态rdzv_backend且未指定rdzv_endpoint时,master_addr只能用于静态rdzv_backend。
rdzv_backend是torch.distributed包中的分布式同步策略的一部分,用于在分布式训练中进行进程间的通信和同步。通过rdzv_backend,可以使用不同的后端实现,如etcd、AWS S3等。当rdzv_backend选择为静态时,可以使用master_addr进行通信,并且无需指定rdzv_endpoint。但当rdzv_backend选择为动态时,则需要指定rdzv_endpoint,master_addr将不起作用。
要解决这个错误,可以检查代码中的分布式设置,确保正确地设置了rdzv_backend和rdzv_endpoint,或者使用静态rdzv_backend来使用master_addr。
注意:rdzv_backend选项是在分布式训练脚本中设置的,具体如何设置要根据使用的分布式工具和后端来确定。
- User
这是什么错误TypeError: Descriptors cannot not be created directly. If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0. If you cannot immediately regenerate your protos, some other possible workarounds are:
- Downgrade the protobuf package to 3.20.x or lower.
- Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).,如何解决
- Assistant
这是一个错误提示,表示由于生成的代码过时,无法直接创建描述符。解决此问题的方法是更新 protobuf 到 3.19.0 或更高版本,并重新生成对应的 protos 文件。如果无法立即重新生成 protos 文件,还有其他可能的解决方法:
- 降级 protobuf 包到 3.20.x 或更低版本。
- 设置环境变量 PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python(但这会使用纯 Python 解析,速度会比较慢)。
具体的解决方法取决于你的具体环境和使用的工具。确保按照提供的建议步骤进行操作,并确保更新任何与 protobuf 相关的依赖项。
- User
这是什么错误ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1113) of binary: /opt/conda/bin/python Traceback (most recent call last): File "/opt/conda/bin/torchrun", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper return f(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main run(args) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run elastic_launch( File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
- Assistant
这是一个错误提示。根据错误消息中的信息,发生了torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 错误。这通常表示由于某个子进程失败,导致整个分布式训练作业失败。
要解决这个错误,可以尝试以下步骤:
-
检查分布式训练作业的配置和参数是否正确。确保传递给torch.distributed.elastic.multiprocessing.launch函数的参数正确无误。
-
检查所使用的环境和依赖项是否正确安装。确保所有依赖库都是兼容的,并且已正确安装。
-
检查主机之间的网络连接和通信是否正常。确保所有的主机都能够互相通信,并且在分布式训练期间没有网络中断或问题。
-
查看详细的错误日志和堆栈跟踪信息,以获取更多关于失败的子进程的详细信息。根据这些信息,可能能够确定导致错误的具体原因,并相应地进行调查和解决。
如果以上步骤都没有解决问题,建议提供更多的上下文和详细信息,以便能够更准确地分析和解决该错误。
-
- User
这个错误如何解决master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified. 2023-09-01 08:53:28.761368: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-09-01 08:53:28.876443: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0. Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1086, in _get_module return importlib.import_module("." + module_name, self.name) File "/opt/conda/lib/python3.10/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 992, in _find_and_load_unlocked File "", line 241, in _call_with_frames_removed File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/opt/conda/lib/python3.10/site-packages/transformers/data/init.py", line 27, in from .processors import ( File "/opt/conda/lib/python3.10/site-packages/transformers/data/processors/init.py", line 15, in from .glue import glue_convert_examples_to_features, glue_output_modes, glue_processors, glue_tasks_num_labels File "/opt/conda/lib/python3.10/site-packages/transformers/data/processors/glue.py", line 30, in import tensorflow as tf File "/opt/conda/lib/python3.10/site-packages/tensorflow/init.py", line 37, in from tensorflow.python.tools import module_util as _module_util File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/init.py", line 37, in from tensorflow.python.eager import context File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/context.py", line 28, in from tensorflow.core.framework import function_pb2 File "/opt/conda/lib/python3.10/site-packages/tensorflow/core/framework/function_pb2.py", line 16, in from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2 File "/opt/conda/lib/python3.10/site-packages/tensorflow/core/framework/attr_value_pb2.py", line 16, in from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2 File "/opt/conda/lib/python3.10/site-packages/tensorflow/core/framework/tensor_pb2.py", line 16, in from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2 File "/opt/conda/lib/python3.10/site-packages/tensorflow/core/framework/resource_handle_pb2.py", line 16, in from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2 File "/opt/conda/lib/python3.10/site-packages/tensorflow/core/framework/tensor_shape_pb2.py", line 36, in _descriptor.FieldDescriptor( File "/opt/conda/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 561, in new _message.Message._CheckCalledFromGeneratedFile() TypeError: Descriptors cannot not be created directly. If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0. If you cannot immediately regenerate your protos, some other possible workarounds are:- Downgrade the protobuf package to 3.20.x or lower.
- Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/sage/home/project/src/04 自然语言/tuning2/main.py", line 34, in from transformers import ( File "", line 1075, in _handle_fromlist File "/opt/conda/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1076, in getattr module = self._get_module(self._class_to_module[name]) File "/opt/conda/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 1088, in _get_module raise RuntimeError( RuntimeError: Failed to import transformers.data.data_collator because of the following error (look up to see its traceback): Descriptors cannot not be created directly. If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0. If you cannot immediately regenerate your protos, some other possible workarounds are:
- Downgrade the protobuf package to 3.20.x or lower.
- Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1316) of binary: /opt/conda/bin/python Traceback (most recent call last): File "/opt/conda/bin/torchrun", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper return f(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main run(args) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run elastic_launch( File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
main.py FAILED
Failures: <NO_OTHER_FAILURES>
Root Cause (first observed failure): [0]: time : 2023-09-01_08:53:30 host : modelid-aeb04d0bcb6df7e509ed494795705dd1-143-0 rank : 0 (local_rank: 0) exitcode : 1 (pid: 1316) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
- Assistant
这个错误信息中有两个不同的问题,我们来分别解决:
- master_addr 错误:
根据错误提示 "master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified",这个错误意味着你使用了 master_addr,但是 rdzv_backend 选择为动态(dynamic)或者未指定 rdzv_endpoint。要解决此问题,有两个解决方法:
- 将 rdzv_backend 设置为静态(static),可以在分布式训练脚本中进行设置。
- 指定 rdzv_endpoint,确保指定了正确的地址和端口。
选择适合你的分布式训练设置的方法,并确保使用正确的参数。
- protobuf 版本不匹配错误:
根据错误消息 "TypeError: Descriptors cannot not be created directly" 和 "protobuf package to 3.20.x or lower",这意味着你的 protobuf 版本可能不兼容。为了解决这个问题,你可以尝试下面的解决方法:
- 降级 protobuf 版本为 3.20.x 或更低版本。
- 设置环境变量 PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python,但这会使用纯 Python 解析,速度会慢一些。
根据你的需要选择合适的解决方法,并确保环境变量或依赖项设置正确。
综上所述,解决这两个问题,你需要设置正确的分布式训练配置,并根据具体情况进行 protobuf 版本的调整。
- User
这是什么错误RuntimeError: Failed to import transformers.training_args_seq2seq because of the following error (look up to see its traceback): cannot import name 'PartialState' from 'accelerate.state' (/opt/conda/lib/python3.10/site-packages/accelerate/state.py) ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1536) of binary: /opt/conda/bin/python Traceback (most recent call last): File "/opt/conda/bin/torchrun", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper return f(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main run(args) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run elastic_launch( File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
- Assistant
这是一个错误提示,表示无法导入
transformers.training_args_seq2seq。错误的原因是找不到accelerate.state模块中的PartialState。这可能是由于使用了不兼容的版本或缺失依赖包导致的。要解决这个错误,可以尝试以下方法:
-
确保 transformers、accelerate 和相关依赖库的版本兼容。可以查看它们的文档或官方仓库,了解它们之间的兼容性要求,并确保使用相应的版本。
-
检查是否正确安装了所需的依赖库。可以使用
pip或其他包管理器,手动安装或更新所
-