DEBUG:rasterio.env:Starting outermost env DEBUG:rasterio.env:No GDAL environment exists…
- User
DEBUG:rasterio.env:Starting outermost env DEBUG:rasterio.env:No GDAL environment exists DEBUG:rasterio.env:New GDAL environment <rasterio._env.GDALEnv object at 0x3431d2f40> created DEBUG:rasterio._env:GDAL_DATA found in environment. DEBUG:rasterio._env:PROJ_DATA found in environment. DEBUG:rasterio._env:Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x3431d2f40>. DEBUG:rasterio.env:Entered env context: <rasterio.env.Env object at 0x3431e3a90> DEBUG:rasterio._io:Output nodata value read from file: -32768.0 DEBUG:rasterio._io:Output nodata values: [-32768.0] DEBUG:rasterio._io:all_valid: False DEBUG:rasterio._io:mask_flags: ([<MaskFlags.nodata: 8>],) DEBUG:rasterio._io:Jump straight to _read() DEBUG:rasterio._io:Window: Window(col_off=24763, row_off=17301, width=512, height=512) DEBUG:rasterio._io:IO window xoff=24763.0 yoff=17301.0 width=512.0 height=512.0 DEBUG:rasterio.env:Exiting env context: <rasterio.env.Env object at 0x3431e3a90> DEBUG:rasterio.env:Cleared existing <rasterio._env.GDALEnv object at 0x3431d2f40> options DEBUG:rasterio._env:Stopped GDALEnv <rasterio._env.GDALEnv object at 0x3431d2f40>. DEBUG:rasterio.env:Exiting outermost env DEBUG:rasterio.env:Exited env context: <rasterio.env.Env object at 0x3431e3a90> [DEBUG] Channel 0 shape: (512, 512) [DEBUG] Channel 1 shape: (512, 512) (184.0, 223.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (196.0, 218.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (206.0, 221.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (192.0, 232.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (199.0, 239.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (249.0, 242.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (211.0, 247.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (235.0, 249.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (247.0, 257.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (266.0, 271.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (257.0, 265.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (274.0, 259.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (266.0, 252.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (257.0, 247.0, 0.0) [<class 'float'>, <class 'float'>, <class 'float'>] (215.0, 228.0, 1.6873483657836914) [<class 'float'>, <class 'float'>, <class 'float'>] Error executing job with overrides: ['+config=configs/multilabel_circle_net.yaml'] Traceback (most recent call last): File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 528, in main() File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/main.py", line 94, in decorated_main _run_hydra( File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra _run_app( File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 457, in _run_app run_and_report( File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 223, in run_and_report raise ex File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 220, in run_and_report return func() File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/utils.py", line 458, in lambda: hydra.run( File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run _ = ret.return_value File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value raise self._return_value File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job ret.return_value = task_function(task_cfg) File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 524, in main CenternetPipeline(cfg).train() File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 415, in train model = self.build_model(config, self.loaders) File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/train_multilabel.py", line 206, in build_model model = nas.search(config.nas.num_iterations, loaders['train'], loaders['valid']) File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 36, in search scores = [self.evaluate_model(model, train_loader, val_loader) for model in population] File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 36, in scores = [self.evaluate_model(model, train_loader, val_loader) for model in population] File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/nas.py", line 112, in evaluate_model outputs = model(inputs) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/centernet_unet.py", line 172, in forward decoded_features = self.decoder(feature_maps) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/opt/anaconda3/envs/sam3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) File "/Users/samerkharboush/Downloads/SAR-MPS-OptunaNasFbnYoloTransformers/xview3/centernet/models/custom_encoder.py", line 65, in forward x = torch.cat([x, encoder_feature], dim=1) RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 256 but got size 255 for tensor number 1 in the list. (sam3) samerkharboush@Samers-MBP SAR-MPS-OptunaNasFbnYoloTransformers %
- Assistant
It looks like you're encountering a
RuntimeErrorduring the forward pass of your model, specifically when attempting to concatenate tensors within your custom decoder in the U-Net architecture. Here's the relevant portion of the error message:RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 256 but got size 255 for tensor number 1 in the list.This error indicates that while concatenating tensors along
dim=1(the channel dimension), the other dimensions (typically height and width in image data) do not match. Specifically, one tensor has a size of 256 along one of these dimensions, and the other tensor has a size of 255.Cause of the Error
In U-Net architectures, it's common to concatenate feature maps from the encoder path with those in the decoder path to allow the network to learn context at multiple scales. However, due to the way pooling and convolution operations work (especially with uneven kernel sizes and strides), the spatial dimensions (height and width) of the feature maps can sometimes differ by one pixel between the encoder and decoder.
This usually happens because:
- Pooling layers reduce spatial dimensions, and if the input dimensions are not perfectly divisible by the pooling window and stride, rounding can cause mismatches.
- Convolutional layers without proper padding can alter spatial dimensions.
- Upsampling methods (like interpolation) might not perfectly recover the original dimensions due to rounding errors.
How to Fix It
To resolve this issue, you need to ensure that the spatial dimensions of the tensors you're trying to concatenate match. Here are some steps and tips to help you fix the problem:
-
Inspect Your Encoder and Decoder Layers:
- Check the layers in your encoder and decoder to see how they alter the spatial dimensions.
- Ensure that any downsampling (e.g., MaxPooling) and upsampling (e.g.,
ConvTranspose2d, interpolation) operations are correctly paired.
-
Use Padding in Convolutional Layers:
- Apply appropriate padding in your convolutional layers to maintain spatial dimensions where necessary.
- For example, using
padding=1with akernel_size=3helps preserve the spatial dimensions.
nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=1, padding=1) -
Adjust Upsampling Methods:
- If you're using interpolation methods like
nn.Upsample, specify thealign_cornersparameter to ensure consistent output sizes. - Consider using
scale_factorcarefully to match the required dimensions.
nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True) - If you're using interpolation methods like
-
Crop or Center Crop Tensors Before Concatenation:
- Crop the larger tensor to match the dimensions of the smaller one before concatenation.
- Implement a cropping layer or function that crops the encoder features to the size of the decoder features.
def center_crop(enc_feature, dec_feature): _, _, h_dec, w_dec = dec_feature.size() _, _, h_enc, w_enc = enc_feature.size() start_h = (h_enc - h_dec) // 2 start_w = (w_enc - w_dec) // 2 return enc_feature[:, :, start_h:start_h+h_dec, start_w:start_w+w_dec]- Use it before concatenation:
encoder_feature_cropped = center_crop(encoder_feature, x) x = torch.cat([x, encoder_feature_cropped], dim=1) -
Debug the Shapes Before Concatenation:
- Add print statements or use a debugger to output the shapes of
xandencoder_featurejust before the concatenation.
print(f"x shape: {x.shape}, encoder_feature shape: {encoder_feature.shape}")- This will help you see exactly where the mismatch is occurring.
- Add print statements or use a debugger to output the shapes of
-
Ensure Consistent Input Sizes:
- If your model expects inputs of certain dimensions, ensure that all input images are appropriately resized or padded before being fed into the model.
Example Adjustment in Your Code
Based on the error message, the issue occurs in
custom_encoder.py, line 65:x = torch.cat([x, encoder_feature], dim=1)Modify your forward method to include cropping:
def forward(self, x, encoder_features): for idx in range(len(self.up_convs)): encoder_feature = encoder_features[idx] x = self.up_convs[idx](x) # Crop encoder_feature if necessary if x.size()[2:] != encoder_feature.size()[2:]: encoder_feature = self.center_crop(encoder_feature, x) x = torch.cat([x, encoder_feature], dim=1) x = self.double_convs[idx](x) return x def center_crop(self, enc_feature, dec_feature): _, _, h_dec, w_dec = dec_feature.size() _, _, h_enc, w_enc = enc_feature.size() start_h = (h_enc - h_dec) // 2 start_w = (w_enc - w_dec) // 2 return enc_feature[:, :, start_h:start_h+h_dec, start_w:start_w+w_dec]Additional Tips
- Check for Off-By-One Errors: Sometimes the discrepancy is due to an off-by-one error in indexing or dimension calculations.
- Consult the Documentation: Refer to PyTorch's documentation for functions like
nn.Conv2d,nn.MaxPool2d,nn.Upsample, etc., to understand how they affect tensor shapes. - Review U-Net Implementations: Look at open-source implementations of U-Net to see how they handle dimension mismatches. For example, the original U-Net paper discusses the need for cropping due to the loss of border pixels.
Conclusion
The error arises because the shapes of the tensors you're trying to concatenate don't match in all dimensions except the concatenation dimension (
dim=1). By ensuring that the spatial dimensions of the encoder and decoder feature maps match (either by careful architecture design, padding, or cropping), you can resolve this issue.Feel free to ask if you need more specific guidance or if you encounter further issues!