Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x33239cd00> [DEBUG]…

  1. User

    DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x33239cd00> [DEBUG] Channel 'vh' image shape before transformations: (512, 512) [DEBUG] Channel 'vh' image shape after individual_transforms: (512, 512) [DEBUG] Channel 'vh' image shape after normalization: (512, 512) [DEBUG] Channel 'vv' image shape before transformations: (512, 512) [DEBUG] Channel 'vv' image shape after individual_transforms: (512, 512) [DEBUG] Channel 'vv' image shape after normalization: (512, 512) [DEBUG] Number of channels: 2 [DEBUG] Channel 0 shape: (512, 512) [DEBUG] Channel 1 shape: (512, 512) [DEBUG] Combined image shape after np.dstack: (512, 512, 2) [DEBUG] Keypoints being passed to albumentations: (237.0, 250.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (231.0, 226.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (238.0, 223.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (245.0, 217.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (267.0, 255.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (260.0, 260.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (261.0, 248.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (252.0, 209.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (260.0, 204.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (289.0, 237.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (281.0, 243.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (274.0, 249.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x33e6e3d60> DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x33e6e3dc0> options DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x33e6e3dc0>. DEBUG:rasterio.env:Exiting outermost env DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x33e6e3d60> [DEBUG] Channel 'vh' image shape before transformations: (512, 512) [DEBUG] Channel 'vh' image shape after individual_transforms: (512, 512) [DEBUG] Channel 'vh' image shape after normalization: (512, 512) [DEBUG] Channel 'vv' image shape before transformations: (512, 512) [DEBUG] Channel 'vv' image shape after individual_transforms: (512, 512) [DEBUG] Channel 'vv' image shape after normalization: (512, 512) [DEBUG] Number of channels: 2 [DEBUG] Channel 0 shape: (512, 512) [DEBUG] Channel 1 shape: (512, 512) [DEBUG] Combined image shape after np.dstack: (512, 512, 2) [DEBUG] Keypoints being passed to albumentations: (36.0, 37.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (25.0, 69.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (76.0, 120.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (37.0, 121.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (2.0, 172.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (259.0, 264.0, 2.214465379714966) [<class 'float'>, <class 'float'>, <class 'float'>] DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x338dc2a60> DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x338dcfee0> options DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x338dcfee0>. DEBUG:rasterio.env:Exiting outermost env DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x338dc2a60> DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x338dcfe50> DEBUG:rasterio.env:Starting outermost env DEBUG:rasterio.env:No GDAL environment exists DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x338dcfe80> created DEBUG:rasterio._env:GDAL_DATA found in environment. DEBUG:rasterio._env:PROJ_DATA found in environment. DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x338dcfe80>. DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x338dcfe50> DEBUG:rasterio._base:Sharing flag: 0 DEBUG:rasterio._base:Nodata success: 1, Nodata value: -32768.000000 DEBUG:rasterio._base:Dataset is started. DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x338dcfe50> DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x338dcfe80> options DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x338dcfe80>. DEBUG:rasterio.env:Exiting outermost env DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x338dcfe50> DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x338dc2a60> DEBUG:rasterio.env:Starting outermost env DEBUG:rasterio.env:No GDAL environment exists DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x338dcfe80> created DEBUG:rasterio._env:GDAL_DATA found in environment. DEBUG:rasterio._env:PROJ_DATA found in environment. DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x338dcfe80>. DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x338dc2a60> DEBUG:rasterio._io:Output nodata value read from file: -32768.0 DEBUG:rasterio._io:Output nodata values: [-32768.0] DEBUG:rasterio._io:all_valid: False DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],) DEBUG:rasterio._io:Jump straight to _read() DEBUG:rasterio._io:Window: Window(col_off=17311, row_off=22200, width=512, height=512) DEBUG:rasterio._io:IO window xoff=17311.0 yoff=22200.0 width=512.0 height=512.0 DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x338dc2a60> DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x338dcfe80> options DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x338dcfe80>. DEBUG:rasterio.env:Exiting outermost env DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x338dc2a60> DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x338dcfe50> DEBUG:rasterio.env:Starting outermost env DEBUG:rasterio.env:No GDAL environment exists DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x338dcfee0> created DEBUG:rasterio._env:GDAL_DATA found in environment. DEBUG:rasterio._env:PROJ_DATA found in environment. DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x338dcfee0>. DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x338dcfe50> DEBUG:rasterio._base:Sharing flag: 0 DEBUG:rasterio._base:Nodata success: 1, Nodata value: -32768.000000 DEBUG:rasterio._base:Dataset is started. DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x338dcfe50> DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x338dcfee0> options DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x338dcfee0>. DEBUG:rasterio.env:Exiting outermost env DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x338dcfe50> DEBUG:rasterio.env:Entering env context: <rasterio.env.Env object at 0x338dc2a60> DEBUG:rasterio.env:Starting outermost env DEBUG:rasterio.env:No GDAL environment exists DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x338dcfee0> created DEBUG:rasterio._env:GDAL_DATA found in environment. DEBUG:rasterio._env:PROJ_DATA found in environment. DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x338dcfee0>. DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x338dc2a60> DEBUG:rasterio._io:Output nodata value read from file: -32768.0 DEBUG:rasterio._io:Output nodata values: [-32768.0] DEBUG:rasterio._io:all_valid: False DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],) DEBUG:rasterio._io:Jump straight to _read() DEBUG:rasterio._io:Window: Window(col_off=17311, row_off=22200, width=512, height=512) DEBUG:rasterio._io:IO window xoff=17311.0 yoff=22200.0 width=512.0 height=512.0 DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x338dc2a60> DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x338dcfee0> options DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x338dcfee0>. DEBUG:rasterio.env:Exiting outermost env DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x338dc2a60> [DEBUG] Channel 'vh' image shape before transformations: (512, 512) [DEBUG] Channel 'vh' image shape after individual_transforms: (512, 512) [DEBUG] Channel 'vh' image shape after normalization: (512, 512) [DEBUG] Channel 'vv' image shape before transformations: (512, 512) [DEBUG] Channel 'vv' image shape after individual_transforms: (512, 512) [DEBUG] Channel 'vv' image shape after normalization: (512, 512) [DEBUG] Number of channels: 2 [DEBUG] Channel 0 shape: (512, 512) [DEBUG] Channel 1 shape: (512, 512) [DEBUG] Combined image shape after np.dstack: (512, 512, 2) [DEBUG] Keypoints being passed to albumentations: (447.0, 95.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (437.0, 95.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (463.0, 193.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (453.0, 193.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (327.0, 15.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (316.0, 16.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (348.0, 106.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (482.0, 291.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (472.0, 290.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (232.0, 37.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (219.0, 31.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (281.0, 321.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (267.0, 228.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (268.0, 319.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (257.0, 230.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (503.0, 383.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (493.0, 389.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (301.0, 422.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (292.0, 428.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (366.0, 213.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (250.0, 126.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (238.0, 128.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (384.0, 303.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (375.0, 311.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (420.0, 508.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (400.0, 406.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (391.0, 409.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] Error executing job with overrides: ['+config=configs/multilabel_circle_net.yaml'] Traceback (most recent call last): File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 529, in main() File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main _run_hydra( File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra _run_app( File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app run_and_report( File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report raise ex File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report return func() File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in lambda: hydra.run( File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run _ = ret.return_value File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value raise self._return_value File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job ret.return_value = task_function(task_cfg) File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 525, in main CenternetPipeline(cfg).train() File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 416, in train model = self.build_model(config, self.loaders) File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 207, in build_model model = nas.search(config.nas.num_iterations, loaders['train'], loaders['valid']) File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 35, in search scores = [self.evaluate_model(model, train_loader, val_loader) for model in population] File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 35, in scores = [self.evaluate_model(model, train_loader, val_loader) for model in population] File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 114, in evaluate_model outputs = model(inputs) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/centernet_unet.py", line 134, in forward neck_features = self.neck(decoded_features) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/necks.py", line 48, in forward return self.fpn(feature_maps) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/pytorch_toolbelt/modules/fpn.py", line 126, in forward layers.append(F.interpolate(f, size=dst_size, mode=self.mode, align_corners=self.align_corners)) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/functional.py", line 4110, in interpolate raise NotImplementedError( NotImplementedError: Input Error: Only 3D, 4D and 5D input Tensors supported (got 4D) for the modes: nearest | linear | bilinear | bicubic | trilinear | area | nearest-exact (got [512, 384, 256, 128]) (sam3) samerkharboush@Samers-MBP SAR-MPS-OptunaNasFbnYoloTransformers %

  2. Assistant

    The error you're encountering suggests that there's a mismatch in the tensor dimensions when using torch.nn.functional.interpolate. Here's what the error message is telling us:

    NotImplementedError: Input Error: Only 3D, 4D and 5D input Tensors supported (got 4D) for the modes: nearest | linear | bilinear | bicubic | trilinear | area | nearest-exact (got [512, 384, 256, 128])
    

    While it says that 3D, 4D, and 5D tensors are supported (and you have a 4D tensor), it's raising an error because the shape [512, 384, 256, 128] doesn't match the expected ordering of dimensions.

    Let's break down the tensor shape and see what's going on.

    In PyTorch, the typical shape for an input tensor for image data is:

    • For 4D tensors (batch of images):

      [batch_size, channels, height, width]
      
    • For 5D tensors (batch of videos or 3D data):

      [batch_size, channels, depth, height, width]
      

    Given that, let's look at your tensor shape:

    [512, 384, 256, 128]
    

    This suggests:

    • batch_size: 512
    • channels: 384
    • height: 256
    • width: 128

    A batch size of 512 is quite large and uncommon, especially for neural network training due to memory constraints. This hints that the dimensions may be misinterpreted.

    Possible Issue: The dimensions of your tensor might be in the wrong order. For example, channels might be in place of batch size, or some dimensions might be swapped.

    Here's how you can troubleshoot and fix the issue:

    Step 1: Inspect Tensor Shapes

    Before the call to F.interpolate, print out the shapes of the tensors involved.

    In your code (specifically in pytorch_toolbelt/modules/fpn.py, before line 126), add:

    print(f"Input tensor shape before interpolate: {f.shape}")
    print(f"Expected size for interpolate: {dst_size}")
    

    Also, in the parts of your code where decoded_features and feature_maps are created (e.g., in centernet_unet.py and necks.py), add print statements:

    # In centernet_unet.py
    print(f"decoded_features shapes:")
    for feat in decoded_features:
        print(feat.shape)
    
    # In necks.py
    print(f"Feature maps shapes:")
    for feat in feature_maps:
        print(feat.shape)
    

    This will help you verify what the shapes are at each stage.

    Step 2: Verify the Order of Dimensions

    Ensure that your tensors are in the expected [batch_size, channels, height, width] format.

    If you find that the dimensions are swapped (e.g., you have [batch_size, height, width, channels]), you can rearrange them using tensor.permute().

    For example:

    # Swap dimensions from [batch_size, height, width, channels] to [batch_size, channels, height, width]
    tensor = tensor.permute(0, 3, 1, 2)
    

    Step 3: Check the Feature Pyramid Network (FPN) Implementation

    In your FPN implementation (pytorch_toolbelt/modules/fpn.py), the forward method seems to be expecting feature maps in a particular shape.

    Ensure that the feature maps you're passing into the FPN have the correct shape and that their dimensions align with what the FPN expects.

    Step 4: Validate the mode Parameter in F.interpolate

    The error message mentions:

    Input Error: Only 3D, 4D and 5D input Tensors supported ... for the modes: nearest | linear | bilinear | bicubic | trilinear | area | nearest-exact
    

    Ensure that the mode you're using in F.interpolate is appropriate for the number of dimensions of your input tensor.

    For example:

    • Use mode='bilinear' for 4D inputs (images).
    • Use mode='trilinear' for 5D inputs (volumes).

    Step 5: Adjust the Batch Size if Necessary

    If you're unintentionally using a batch size of 512, consider reducing it. A common batch size is between 8 and 64, depending on your GPU memory.

    Example Correction

    Suppose you realize that your tensors are in [channels, height, width, batch_size] format. You can correct it by permuting the dimensions:

    # From [channels, height, width, batch_size] to [batch_size, channels, height, width]
    tensor = tensor.permute(3, 0, 1, 2)
    

    Step 6: Update the Model's Forward Method

    Ensure that the forward methods in your model correctly handle the data and maintain the proper tensor shapes throughout the network.

    Conclusion

    The error arises because the input tensor to F.interpolate doesn't have the expected shape. By checking and correcting the tensor shapes and ensuring they match the expected format for PyTorch functions, you should be able to resolve the issue.

    Action Items:

    • Add print statements to check tensor shapes at each stage.
    • Ensure tensors are in [batch_size, channels, height, width] format.
    • Correct any misaligned dimensions using permute.
    • Verify that the mode parameter in F.interpolate matches the tensor dimensions.

    If you continue to experience issues after these steps, please provide the printed shapes from your debugging statements, and we can investigate further.