进程结束前出现了如下错误:…
- User
进程结束前出现了如下错误: /home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/diffusers/configuration_utils.py:134: FutureWarning: Accessing config attribute
num_train_timestepsdirectly via 'DDPMScheduler' object attribute is deprecated. Please access 'num_train_timesteps' over 'DDPMScheduler's config object instead, e.g. 'scheduler.config.num_train_timesteps'. deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False) /home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/diffusers/configuration_utils.py:134: FutureWarning: Accessing config attributenum_train_timestepsdirectly via 'DDPMScheduler' object attribute is deprecated. Please access 'num_train_timesteps' over 'DDPMScheduler's config object instead, e.g. 'scheduler.config.num_train_timesteps'. deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False) /home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/utils/checkpoint.py:1399: FutureWarning:torch.cpu.amp.autocast(args...)is deprecated. Please usetorch.amp.autocast('cpu', args...)instead. with device_autocast_ctx, torch.cpu.amp.autocast(**cpu_autocast_kwargs), recompute_context: # type: ignore[attr-defined] /home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/utils/checkpoint.py:1399: FutureWarning:torch.cpu.amp.autocast(args...)is deprecated. Please usetorch.amp.autocast('cpu', args...)instead. with device_autocast_ctx, torch.cpu.amp.autocast(**cpu_autocast_kwargs), recompute_context: # type: ignore[attr-defined] Steps: 0%| | 0/30000 [00:16<?, ?it/s, lr=0, step_loss=0.0128][rank0]:[W919 19:13:45.327293009 Module.cpp:175] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1.../home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/transformers/deepspeed.py:24: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations warnings.warn( /home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/adamw.py:159: UserWarning: Unsupported unwinding pattern: Address not in range (Triggered internally at ../torch/csrc/profiler/unwind/unwind.cpp:209.) state["exp_avg_sq"] = torch.zeros_like( [rank0]: Traceback (most recent call last): [rank0]: File "/data/fl/Wonder3D-sketch/train_mvdiffusion_image_edit_test_only.py", line 863, in [rank0]: main(cfg) [rank0]: File "/data/fl/Wonder3D-sketch/train_mvdiffusion_image_edit_test_only.py", line 779, in main [rank0]: optimizer.step() [rank0]: File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/accelerate/optimizer.py", line 159, in step [rank0]: self.scaler.step(self.optimizer, closure) [rank0]: File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/amp/grad_scaler.py", line 454, in step [rank0]: retval = self.maybe_opt_step(optimizer, optimizer_state, *args, **kwargs) [rank0]: File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/amp/grad_scaler.py", line 352, in maybe_opt_step [rank0]: retval = optimizer.step(*args, **kwargs) [rank0]: File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/accelerate/optimizer.py", line 214, in patched_step [rank0]: return method(*args, **kwargs) [rank0]: File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 130, in wrapper [rank0]: return func.get(opt, opt.class)(*args, **kwargs) [rank0]: File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/optimizer.py", line 484, in wrapper [rank0]: out = func(args, kwargs) [rank0]: File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/optimizer.py", line 89, in _use_grad [rank0]: ret = func(self, args, kwargs) [rank0]: File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/adamw.py", line 216, in step [rank0]: has_complex = self._init_group( [rank0]: File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/optim/adamw.py", line 159, in _init_group [rank0]: state["exp_avg_sq"] = torch.zeros_like( [rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 26.00 MiB. GPU 0 has a total capacity of 31.73 GiB of which 18.38 MiB is free. Process 1140475 has 1.28 GiB memory in use. Process 893160 has 10.20 GiB memory in use. Including non-PyTorch memory, this process has 20.23 GiB memory in use. Of the allocated memory 19.62 GiB is allocated by PyTorch, and 130.02 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) [rank0]: Exception raised from malloc at ../c10/cuda/CUDACachingAllocator.cpp:1167 (most recent call first): [rank0]: C++ CapturedTraceback: [rank0]: #4 std::_Function_handler<std::shared_ptr<c10::LazyValuestd::string const> (), c10::SetStackTraceFetcher(std::function<std::string ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) from Logging.cpp:0 [rank0]: #5 c10::Error::Error(c10::SourceLocation, std::string) from ??:0 [rank0]: #6 c10::cuda::CUDACachingAllocator::Native::DeviceCachingAllocator::malloc(signed char, unsigned long, CUstream_st) from CUDACachingAllocator.cpp:0 [rank0]: #7 c10::cuda::CUDACachingAllocator::Native::NativeCachingAllocator::malloc(void, signed char, unsigned long, CUstream_st) from :0 [rank0]: #8 c10::cuda::CUDACachingAllocator::Native::NativeCachingAllocator::allocate(unsigned long) from :0 [rank0]: #9 at::TensorBase at::detail::_empty_strided_generic<c10::ArrayRef >(c10::ArrayRef, c10::ArrayRef, c10::Allocator, c10::DispatchKeySet, c10::ScalarType) from :0 [rank0]: #10 at::detail::empty_strided_generic(c10::ArrayRef, c10::ArrayRef, c10::Allocator, c10::DispatchKeySet, c10::ScalarType) from ??:0 [rank0]: #11 at::detail::empty_strided_cuda(c10::ArrayRef, c10::ArrayRef, c10::ScalarType, std::optionalc10::Device) from ??:0 [rank0]: #12 at::detail::empty_strided_cuda(c10::ArrayRef, c10::ArrayRef, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional) from ??:0 [rank0]: #13 at::native::empty_strided_cuda(c10::ArrayRef, c10::ArrayRef, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional) from ??:0 [rank0]: #14 at::(anonymous namespace)::(anonymous namespace)::wrapper_CUDA__empty_strided(c10::ArrayRefc10::SymInt, c10::ArrayRefc10::SymInt, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional) from RegisterCUDA.cpp:0 [rank0]: #15 c10::impl::wrap_kernel_functor_unboxed<c10::impl::detail::WrapFunctionIntoFunctor<c10::CompileTimeFunctionPointer<at::Tensor (c10::ArrayRefc10::SymInt, c10::ArrayRefc10::SymInt, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CUDA__empty_strided>, at::Tensor, c10::guts::typelist::typelist<c10::ArrayRefc10::SymInt, c10::ArrayRefc10::SymInt, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional > >, at::Tensor (c10::ArrayRefc10::SymInt, c10::ArrayRefc10::SymInt, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional)>::call(c10::OperatorKernel*, c10::DispatchKeySet, c10::ArrayRefc10::SymInt, c10::ArrayRefc10::SymInt, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional) from RegisterCUDA.cpp:0 [rank0]: #16 at::ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRefc10::SymInt, c10::ArrayRefc10::SymInt, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional) from ??:0 [rank0]: #17 c10::impl::wrap_kernel_functor_unboxed<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (c10::ArrayRefc10::SymInt, c10::ArrayRefc10::SymInt, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional), &at::(anonymous namespace)::empty_strided>, at::Tensor, c10::guts::typelist::typelist<c10::ArrayRefc10::SymInt, c10::ArrayRefc10::SymInt, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional > >, at::Tensor (c10::ArrayRefc10::SymInt, c10::ArrayRefc10::SymInt, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional)>::call(c10::OperatorKernel*, c10::DispatchKeySet, c10::ArrayRefc10::SymInt, c10::ArrayRefc10::SymInt, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional) from RegisterBackendSelect.cpp:0 [rank0]: #18 at::ops::empty_strided::call(c10::ArrayRefc10::SymInt, c10::ArrayRefc10::SymInt, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional) from ??:0 [rank0]: #19 at::native::empty_like(at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat) from ??:0 [rank0]: #20 c10::impl::wrap_kernel_functor_unboxed<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CompositeExplicitAutograd__empty_like>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat > >, at::Tensor (at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat) from RegisterCompositeExplicitAutograd.cpp:0 [rank0]: #21 at::ops::empty_like::call(at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat) from ??:0 [rank0]: #22 at::empty_like(at::Tensor const&, c10::TensorOptions, std::optionalc10::MemoryFormat) from :0 [rank0]: #23 at::native::zeros_like(at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat) from ??:0 [rank0]: #24 c10::impl::wrap_kernel_functor_unboxed<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CompositeExplicitAutograd__zeros_like>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat > >, at::Tensor (at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat) from RegisterCompositeExplicitAutograd.cpp:0 [rank0]: #25 at::_ops::zeros_like::call(at::Tensor const&, std::optionalc10::ScalarType, std::optionalc10::Layout, std::optionalc10::Device, std::optional, std::optionalc10::MemoryFormat) from ??:0 [rank0]: #26 torch::autograd::THPVariable_zeros_like(_object*, _object*, _object*) from python_torch_functions_0.cpp:0 [rank0]: #27 cfunction_call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/methodobject.c:539 [rank0]: #28 _PyObject_MakeTpCall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:191 [rank0]: #29 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:116 [rank0]: #30 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40 [rank0]: #31 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118 [rank0]: #32 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40 [rank0]: #33 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395 [rank0]: #34 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230 [rank0]: #35 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092 [rank0]: #36 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40 [rank0]: #37 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395 [rank0]: #38 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230 [rank0]: #39 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092 [rank0]: #40 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40 [rank0]: #41 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395 [rank0]: #42 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118 [rank0]: #43 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230 [rank0]: #44 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092 [rank0]: #45 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40 [rank0]: #46 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395 [rank0]: #47 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230 [rank0]: #48 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092 [rank0]: #49 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40 [rank0]: #50 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395 [rank0]: #51 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230 [rank0]: #52 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092 [rank0]: #53 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40 [rank0]: #54 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395 [rank0]: #55 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118 [rank0]: #56 PyVectorcall_Call from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:230 [rank0]: #57 do_call_core from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:5092 [rank0]: #58 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40 [rank0]: #59 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395 [rank0]: #60 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118 [rank0]: #61 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40 [rank0]: #62 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395 [rank0]: #63 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118 [rank0]: #64 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40 [rank0]: #65 _PyFunction_Vectorcall from /tmp/build/80754af9/python-split_1605449976777/work/Objects/call.c:395 [rank0]: #66 _PyObject_VectorcallTstate from /tmp/build/80754af9/python-split_1605449976777/work/Include/cpython/abstract.h:118 [rank0]: #67 _PyEval_EvalFrame from /tmp/build/80754af9/python-split_1605449976777/work/Include/internal/pycore_ceval.h:40 [rank0]: #68 _PyEval_EvalCodeWithName from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:4331 [rank0]: #69 PyEval_EvalCodeEx from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:4347 [rank0]: #70 PyEval_EvalCode from /tmp/build/80754af9/python-split_1605449976777/work/Python/ceval.c:809 [rank0]: #71 run_eval_code_obj from /tmp/build/80754af9/python-split_1605449976777/work/Python/pythonrun.c:1178 [rank0]: #72 run_mod from /tmp/build/80754af9/python-split_1605449976777/work/Python/pythonrun.c:1199 [rank0]: #73 PyRun_FileExFlags from /tmp/build/80754af9/python-split_1605449976777/work/Python/pythonrun.c:1116 [rank0]: #74 PyRun_SimpleFileExFlags from /tmp/build/80754af9/python-split_1605449976777/work/Python/pythonrun.c:438 [rank0]: #75 pymain_run_file from /tmp/build/80754af9/python-split_1605449976777/work/Modules/main.c:379 [rank0]: #76 Py_BytesMain from /tmp/build/80754af9/python-split_1605449976777/work/Modules/main.c:1129 [rank0]: #77 __libc_start_main from ??:0 [rank0]: #78 _start from /home/rdonnelly/mc/conda-bld/compilers_linux-64_1534865402226/work/.build/src/glibc-2.12.2/csu/../sysdeps/x86_64/elf/start.S:103 [rank0]: #79 from ??:0
Steps: 0%| | 0/30000 [00:20<?, ?it/s, lr=0, step_loss=0.0128] [rank1]:[E919 19:13:48.759261807 ProcessGroupGloo.cpp:143] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank. [rank1]:[W919 19:13:48.760329940 unwind.cpp:209] Warning: Unsupported unwinding pattern: Address not in range (function unwinderFor) [rank1]:[W919 19:13:48.760365428 Module.cpp:175] symbolizing C++ stack trace for exception; if this hangs, rerun with TORCH_DISABLE_ADDR2LINE=1...
W0919 19:13:48.848006 140368826976064 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 908789 closing signal SIGTERM E0919 19:13:49.668027 140368826976064 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 908788) of binary: /home/fl/.conda/envs/wonder3d-harsh/bin/python Traceback (most recent call last): File "/home/fl/.conda/envs/wonder3d-harsh/bin/accelerate", line 8, in sys.exit(main()) File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main args.func(args) File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/accelerate/commands/launch.py", line 1165, in launch_command multi_gpu_launcher(args) File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/accelerate/commands/launch.py", line 799, in multi_gpu_launcher distrib_run.run(args) File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/distributed/run.py", line 892, in run elastic_launch( File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 133, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/home/fl/.conda/envs/wonder3d-harsh/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
train_mvdiffusion_image_edit_test_only.py FAILED
Failures: <NO_OTHER_FAILURES>
- Assistant
It appears you're encountering a CUDA Out of Memory error during your training process. Here's the critical part of the error message:
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 26.00 MiB. GPU 0 has a total capacity of 31.73 GiB of which 18.38 MiB is free. Process 1140475 has 1.28 GiB memory in use. Process 893160 has 10.20 GiB memory in use. Including non-PyTorch memory, this process has 20.23 GiB memory in use. Of the allocated memory 19.62 GiB is allocated by PyTorch, and 130.02 MiB is reserved by PyTorch but unallocated.This error indicates that your GPU doesn't have enough available memory to allocate the requested amount (26.00 MiB in this case). Let's break down what's happening and explore potential solutions.
Understanding the Issue
-
Memory Usage Details:
- Total GPU Memory: 31.73 GiB
- Free Memory Before Allocation Attempt: 18.38 MiB (which is very low)
- Current Processes Using GPU Memory:
- Process 1140475: 1.28 GiB in use
- Process 893160: 10.20 GiB in use
- Total Memory in Use: 20.23 GiB (including non-PyTorch memory)
- Allocated by PyTorch: 19.62 GiB
- Memory Reserved but Unallocated by PyTorch: 130.02 MiB
-
Error Occurrence:
- Your script attempted to allocate an additional 26.00 MiB, but there wasn't enough free memory on the GPU to satisfy this request.
Potential Causes and Solutions
1. Other Processes Occupying GPU Memory
- Cause: Other processes (e.g., process 1140475 and 893160) are using significant GPU memory.
- Solution:
- Check Running Processes:
- Run
nvidia-smiin your terminal to view all processes using the GPU. - Look for any unnecessary or background processes that might be using GPU memory.
- Run
- Terminate Unnecessary Processes:
- If safe to do so, terminate these processes to free up GPU memory.
- Use
kill -9 <PID>to terminate a process (replace<PID>with the actual process ID).
- Check Running Processes:
2. Batch Size Too Large
- Cause: A large batch size increases memory consumption.
- Solution:
- Reduce Batch Size:
- Decrease the batch size in your training script.
- Even a small reduction can significantly reduce memory usage.
- Reduce Batch Size:
3. Model Complexity
- Cause: Large models consume more GPU memory.
- Solution:
- Use a Smaller Model:
- If possible, use a lighter version of the model.
- Model Pruning or Quantization:
- Apply techniques like pruning unused connections or quantizing weights to reduce model size.
- Use a Smaller Model:
4. Mixed Precision Training
- Cause: Models trained in full precision (FP32) consume more memory.
- Solution:
- Enable Mixed Precision:
- Use PyTorch's Automatic Mixed Precision (AMP) to train in FP16.
- Wrap your training code with
torch.cuda.amp.autocast()context. - Use
torch.cuda.amp.GradScaler()to scale gradients.
- Enable Mixed Precision:
5. Gradient Checkpointing
- Cause: Storing all intermediate activations for backpropagation consumes memory.
- Solution:
- Implement Gradient Checkpointing:
- Trades computation for memory by recomputing some activations during backward pass.
- Use
torch.utils.checkpoint.checkpoint()on parts of your model.
- Implement Gradient Checkpointing:
6. Memory Fragmentation
- Cause: Memory might be fragmented, making it hard to allocate large contiguous blocks.
- Solution:
- Set PyTorch CUDA Allocation Config:
- Set the environment variable to allow for expandable segments:
export PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:128 - This can help PyTorch manage memory more efficiently.
- Set the environment variable to allow for expandable segments:
- Set PyTorch CUDA Allocation Config:
7. Clear Cache Before Allocation
- Cause: Cached memory isn't being released, leading to OOM errors.
- Solution:
- Manually Clear Cache:
- Insert
torch.cuda.empty_cache()before the allocation step. - Note: This doesn't free up memory occupied by tensors, but it can help with fragmentation.
- Insert
- Manually Clear Cache:
8. Monitor Memory Usage During Training
- Cause: Memory leaks or unexpected spikes during training.
- Solution:
- Use Profiling Tools:
- Use tools like PyTorch's Profiler or external tools to monitor memory usage over time.
- Check Data Loading:
- Ensure that your data loaders aren't holding onto data unnecessarily.
- Set
pin_memory=Falseif not needed.
- Use Profiling Tools:
9. Use Smaller Input Sizes
- Cause: Large input data (e.g., high-resolution images) increase memory usage.
- Solution:
- Reduce Input Size:
- Resize images or sequences to smaller dimensions before feeding them to the model.
- Reduce Input Size:
10. Distribute the Load Across Multiple GPUs
- Cause: Single GPU can't handle the memory requirements.
- Solution:
- Use Data Parallelism:
- Distribute your model and data across multiple GPUs if available.
- Use
nn.DataParallelor, better yet,DistributedDataParallelfor multi-GPU setups.
- Use Data Parallelism:
11. Upgrade Hardware
- Cause: The GPU doesn't have sufficient memory for your task.
- Solution:
- Use a GPU with More Memory:
- If possible, switch to a GPU with more VRAM.
- Use a GPU with More Memory:
Next Steps
-
Check GPU Memory Usage:
Run the following command to see current GPU utilization and processes:
nvidia-smiLook for any processes that aren't part of your training job and terminate them if possible.
-
Adjust Training Parameters:
- Reduce Batch Size:
- Try reducing the batch size incrementally until the OOM error no longer occurs.
- Enable Mixed Precision:
- Modify your training script to use mixed-precision training.
- Reduce Batch Size:
-
Optimize Model and Code:
- Implement Gradient Checkpointing:
- Apply checkpointing to memory-intensive parts of your model.
- Optimize Data Loading:
- Ensure your data pipeline is efficient and doesn't hold unnecessary data in memory.
- Implement Gradient Checkpointing:
-
Set Environment Variables for PyTorch:
-
Add the following environment variable before running your script:
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:TrueThis can help reduce memory fragmentation.
-
-
Restart Your System or GPU Processes:
- Sometimes, residual memory usage can linger after processes have terminated.
- Restarting your system or resetting the GPU can help clear this memory.
Additional Considerations
-
Monitor Over Time:
- Keep an eye on memory usage as your training progresses. Some models may gradually consume more memory.
-
Exception Handling:
- Incorporate try-except blocks to catch OOM errors gracefully and possibly adjust parameters on the fly.
-
Check for Memory Leaks:
- Ensure that you're not inadvertently retaining references to tensors that are no longer needed.
Conclusion
The primary issue is that your GPU doesn't have enough free memory to allocate additional resources for your training process. By freeing up GPU memory from other processes and optimizing your training configuration, you should be able to resolve the
CUDA out of memoryerror.Please let me know if you need further assistance or if you have questions about any of the steps mentioned above.
-