Skip to content
The Chat Archive
o1-preview-2024-09-12· OpenAI WildChat-4.8M (AI2)

#%% import torch import torchinfo from OMI.RectifiedFlow import RectifiedFlow from…

  1. User

    #%% import torch import torchinfo from OMI.RectifiedFlow import RectifiedFlow from OMI.dit import DiT from OMI.utils import RunningMean

    device='cuda' torch.set_float32_matmul_precision('high')

    import os from PIL import Image from torch.utils.data import Dataset from torch.utils.data import DataLoader from torchvision.transforms import v2

    class ImageDataset(Dataset): def init(self, img_dir, transform=None): """ Args: img_dir (string): Directory with all the images. transform (callable, optional): Optional transform to be applied on an image. """ self.img_dir = img_dir self.image_names = os.listdir(img_dir) self.transform = transform

    def __len__(self):
        return len(self.image_names)
        
    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        
        # Get image path
        img_name = self.image_names[idx]
        img_path = os.path.join(self.img_dir, img_name)
        
        # Load image
        image = Image.open(img_path).convert('RGB')  # Convert to RGB
        
        # Apply transformations if any
        if self.transform:
            image = self.transform(image)
        
        return image
    

    #%% def main(): model=DiT(384,8,3,3,1,8,2) print(torchinfo.summary(model, input_size=[(1,3,128,128),(1,1,1,1)]))

    modelT=RectifiedFlow(model,1000,device=device,lr=1e-4,emaStrength=0.999)
    
    #%%    
    from torch.utils.tensorboard import SummaryWriter
    import datetime
    logger=SummaryWriter(f'./omitest/{datetime.datetime.now().strftime("%Y%m%d-%H%M%S")}')
    #%%
    dataset = ImageDataset(img_dir='/mnt/h/minecraft/icons', transform=v2.Compose([
        v2.PILToTensor(),
        v2.ToDtype(torch.float32, scale=True),
        v2.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
        ]))
    

    Optionally, create a DataLoader to iterate over the dataset

    dataloader = DataLoader(dataset, batch_size=32, shuffle=True)
    #%%
    #Load weights
    #checkpoint = torch.load('./checkpoint.pt',map_location='cpu',weights_only=True)
    #modelT.load_state_dict(checkpoint)
    #%%
    step=-1
    while True:
        #Training
        loss=RunningMean()
        for image in dataloader:
            #image=torch.nn.functional.interpolate(image,(128,128))
            metrics=modelT.train_step(image.to('cuda'))
            loss.update(metrics['loss'])
            
            step+=1
            print(f"Step {step}, Loss: {metrics['loss']}",end='\r')
        
            if step%100==0:
                break
    
        #Logging
        logger.add_scalar(f'Generator Loss', loss.value, step)
        if(step % 100 == 0):
            with torch.inference_mode(), torch.amp.autocast('cuda',dtype=torch.float32):
                pred=modelT.call(steps=20,shape=(1,3,128,128))
                for i in range(1):
                    logger.add_image(f'Generated Image/{i}', pred[i].float(), step)
                    logger.add_image(f'Real Image/{i}', image[i], step)
                logger.flush()
    
        #Save all models
        #torch.save(modelT.state_dict(),'./checkpoint.pt')
    

    if name == 'main': main()

    i got noise as generated images, i think its because i normalizing images, how to fix them?

  2. Assistant

    The issue you're experiencing is likely due to the images being normalized in your dataset transformation and not being de-normalized before visualization. When you normalize images during preprocessing (with mean and standard deviation), their pixel values are adjusted to have a mean of 0 and a standard deviation of 1 (or whatever values you specify). This means that the pixel values are no longer in the range [0, 1] or [0, 255], which are the expected ranges for most image visualization tools.

    When you attempt to visualize these normalized images directly, they may appear as noise because the visualization tools interpret the adjusted pixel values incorrectly. To fix this, you need to de-normalize your images before visualization by reversing the normalization transformation.

    Here's how you can fix the issue:

    1. Define a De-normalization Function:

      Create a function to reverse the normalization process. This function will take a normalized tensor and apply the inverse transformation to recover the original pixel values.

      def denormalize(tensor, mean, std):
          mean = torch.as_tensor(mean, dtype=tensor.dtype, device=tensor.device).view(-1, 1, 1)
          std = torch.as_tensor(std, dtype=tensor.dtype, device=tensor.device).view(-1, 1, 1)
          tensor = tensor * std + mean
          return tensor.clamp(0, 1)  # Ensure the values are within [0, 1]
      
    2. Modify Your Logging Code:

      Update the code where you log or save the images to de-normalize them before visualization.

      mean = [0.485, 0.456, 0.406]
      std = [0.229, 0.224, 0.225]
      
      for i in range(1):
          # De-normalize generated image
          gen_image = denormalize(pred[i].cpu(), mean, std)
          # De-normalize real image
          real_image = denormalize(image[i].cpu(), mean, std)
          # Log images
          logger.add_image(f'Generated Image/{i}', gen_image, step)
          logger.add_image(f'Real Image/{i}', real_image, step)
      
    3. Ensure Consistency in Normalization Across Model and Data:

      Make sure the model's expected input range matches the normalization applied to the dataset. If your model expects inputs in [0, 1], you should adjust your normalization accordingly. Alternatively, if the model expects inputs normalized with a specific mean and standard deviation (like in ImageNet models), then your current normalization is appropriate.

    4. Check the Model's Output Range:

      When generating images with the model, verify the range of the output tensor. Some generative models output images in a specific range (e.g., [-1, 1]). If that's the case, you need to adjust your de-normalization function or scaling accordingly.

      For example, if the model outputs images in [-1, 1], you can adjust the de-normalization function:

      def denormalize_output(tensor):
          tensor = (tensor + 1) / 2  # Scale from [-1, 1] to [0, 1]
          return tensor.clamp(0, 1)
      

      And apply it to the generated images:

      gen_image = denormalize_output(pred[i].cpu())
      
    5. Verify the Image Shapes and Data Types:

      Ensure that the images have the correct shape and data type when logging them. The add_image function expects images in the shape [C, H, W] and typically in float32 data type with values in [0, 1].

    6. Adjust the Transformations if Necessary:

      If after de-normalizing the images still don't appear correctly, double-check your dataset transformations. You might need to adjust the normalization parameters or ensure that the scaling is appropriate.

    Summary:

    The noise in the generated images is due to not de-normalizing the images before visualization. By applying the inverse of your normalization transformation to both the generated and real images before logging them, you can visualize them correctly and verify if your model is generating meaningful images.

    Revised Logging Code with De-normalization:

    def denormalize(tensor):
        mean = torch.tensor([0.485, 0.456, 0.406], device=tensor.device).view(-1,1,1)
        std = torch.tensor([0.229, 0.224, 0.225], device=tensor.device).view(-1,1,1)
        tensor = tensor * std + mean
        return tensor.clamp(0,1)  # Clamp to [0,1]
    
    for i in range(1):
        # De-normalize the generated image
        gen_image = denormalize(pred[i].cpu())
        # De-normalize the real image
        real_image = denormalize(image[i].cpu())
        # Log the images
        logger.add_image(f'Generated Image/{i}', gen_image, step)
        logger.add_image(f'Real Image/{i}', real_image, step)
    

    By making these adjustments, you should be able to see meaningful images instead of noise when visualizing your generated and real images.