Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

I have this Mobile U-Net. I want to make it conditional and use it in my DDPM training…

  1. User

    I have this Mobile U-Net. I want to make it conditional and use it in my DDPM training loop. Mobile Unet is as follow: # Mobile UNet and Inverted Residual Block

    Author: Lafith Mattara

    Date: July 2022

    import torch import torch.nn as nn

    class InvertedResidualBlock(nn.Module): """ inverted residual block used in MobileNetV2 """ def init(self, in_c, out_c, stride, expansion_factor=6, deconvolve=False): super(InvertedResidualBlock, self).init() # check stride value assert stride in [1, 2] self.stride = stride self.in_c = in_c self.out_c = out_c # Skip connection if stride is 1 self.use_skip_connection = True if self.stride == 1 else False

        # expansion factor or t as mentioned in the paper
        ex_c = int(self.in_c * expansion_factor)
        if deconvolve:
            self.conv = nn.Sequential(
                # pointwise convolution
                nn.Conv2d(self.in_c, ex_c, 1, 1, 0, bias=False),
                nn.BatchNorm2d(ex_c),
                nn.ReLU6(inplace=True),
                # depthwise convolution
                nn.ConvTranspose2d(ex_c, ex_c, 4,self.stride,1, groups=ex_c, bias=False),
                nn.BatchNorm2d(ex_c),
                nn.ReLU6(inplace=True),
                # pointwise convolution
                nn.Conv2d(ex_c, self.out_c, 1, 1, 0, bias=False),
                nn.BatchNorm2d(self.out_c),
            )
        else:
            self.conv = nn.Sequential(
                # pointwise convolution
                nn.Conv2d(self.in_c, ex_c, 1, 1, 0, bias=False),
                nn.BatchNorm2d(ex_c),
                nn.ReLU6(inplace=True),
                # depthwise convolution
                nn.Conv2d(ex_c, ex_c, 3, self.stride, 1, groups=ex_c, bias=False),
                nn.BatchNorm2d(ex_c),
                nn.ReLU6(inplace=True),
                # pointwise convolution
                nn.Conv2d(ex_c, self.out_c, 1, 1, 0, bias=False),
                nn.BatchNorm2d(self.out_c),
            )
        self.conv1x1 = nn.Conv2d(self.in_c, self.out_c, 1, 1, 0, bias=False)
    
            
    
    def forward(self, x):
        if self.use_skip_connection:
            out = self.conv(x)
            if self.in_c != self.out_c:
                x = self.conv1x1(x)
            return x+out
        else:
            return self.conv(x)
    

    class MobileUNet(nn.Module): """ Modified UNet with inverted residual block and depthwise seperable convolution """

    def __init__(self):
        super(MobileUNet, self).__init__()
    
        # encoding arm
        self.conv3x3 = self.depthwise_conv(1, 32, p=1, s=2)
        self.irb_bottleneck1 = self.irb_bottleneck(32, 16, 1, 1, 1)
        self.irb_bottleneck2 = self.irb_bottleneck(16, 24, 2, 2, 6)
        self.irb_bottleneck3 = self.irb_bottleneck(24, 32, 3, 2, 6)
        self.irb_bottleneck4 = self.irb_bottleneck(32, 64, 4, 2, 6)
        self.irb_bottleneck5 = self.irb_bottleneck(64, 96, 3, 1, 6)
        self.irb_bottleneck6 = self.irb_bottleneck(96, 160, 3, 2, 6)
        self.irb_bottleneck7 = self.irb_bottleneck(160, 320, 1, 1, 6)
        self.conv1x1_encode = nn.Conv2d(320, 1280, kernel_size=1, stride=1)
        # decoding arm
        self.D_irb1 = self.irb_bottleneck(1280, 96, 1, 2, 6, True)
        self.D_irb2 = self.irb_bottleneck(96, 32, 1, 2, 6, True)
        self.D_irb3 = self.irb_bottleneck(32, 24, 1, 2, 6, True)
        self.D_irb4 = self.irb_bottleneck(24, 16, 1, 2, 6, True)
        self.DConv4x4 = nn.ConvTranspose2d(16,16,4,2,1,groups=16, bias=False)
        # Final layer: output channel number can be changed as per the usecase
        self.conv1x1_decode = nn.Conv2d(16, 1, kernel_size=1, stride=1)
    
    def depthwise_conv(self, in_c, out_c, k=3, s=1, p=0):
        """
        optimized convolution by combining depthwise convolution and
        pointwise convolution.
        """
        conv = nn.Sequential(
            nn.Conv2d(in_c, in_c, kernel_size=k, padding=p, groups=in_c, stride=s),
            nn.BatchNorm2d(num_features=in_c),
            nn.ReLU6(inplace=True),
            nn.Conv2d(in_c, out_c, kernel_size=1),
        )
        return conv
    
    def irb_bottleneck(self, in_c, out_c, n, s, t, d=False):
        """
        create a series of inverted residual blocks.
        """
        convs = []
        xx = InvertedResidualBlock(in_c, out_c, s, t, deconvolve=d)
        convs.append(xx)
        if n>1:
            for i in range(1,n):
                xx = InvertedResidualBlock(out_c, out_c, 1, t, deconvolve=d)
                convs.append(xx)
        conv = nn.Sequential(*convs)
        return conv
    
    def get_count(self, model):
        # simple function to get the count of parameters in a model.
        num = sum(p.numel() for p in model.parameters() if p.requires_grad)
        return num
    
    def forward(self, x):
        # Left arm/ Encoding arm
        #D1
        x1 = self.conv3x3(x) #(32, 112, 112)
        x2 = self.irb_bottleneck1(x1) #(16,112,112) s1
        x3 = self.irb_bottleneck2(x2) #(24,56,56) s2
        x4 = self.irb_bottleneck3(x3) #(32,28,28) s3
        x5 = self.irb_bottleneck4(x4) #(64,14,14)
        x6 = self.irb_bottleneck5(x5) #(96,14,14) s4
        x7 = self.irb_bottleneck6(x6) #(160,7,7)
        x8 = self.irb_bottleneck7(x7) #(320,7,7)
        x9 = self.conv1x1_encode(x8) #(1280,7,7) s5
    
        # Right arm / Decoding arm with skip connections
        d1 = self.D_irb1(x9) + x6
        d2 = self.D_irb2(d1) + x4
        d3 = self.D_irb3(d2) + x3
        d4 = self.D_irb4(d3) + x2
        d5 = self.DConv4x4(d4)
        out = self.conv1x1_decode(d5)
        return out
    

    I want to have it such that self.diffusion_model = MobileUNet() should be the Unet in diffusion process and the training step would be like model_output= self.diffusion_model(noisy_y, timesteps, x_latent.flatten(2))

  2. Assistant

    Sure! To adapt your MobileUNet to be conditional and use it in a DDPM training loop, you'll need to modify it to accept additional inputs like timesteps and x_latent (flattened as needed). You'll also need to adjust the architecture to incorporate these conditioning inputs appropriately.

    Here’s how you can modify your MobileUNet:

    1. Add Embedding Layers for Timesteps and Conditioning Variables:

      • Use a sinusoidal positional embedding for the timesteps.
      • Use a simple MLP to process the conditioning input (x_latent) to produce embeddings.
    2. Modify Your InvertedResidualBlock to Accept Embeddings:

      • Adjust InvertedResidualBlock to accept an embedding vector.
      • Use this embedding to produce scaling (gamma) and shifting (beta) parameters via an MLP.
      • Apply these parameters to modulate the activations using Feature-wise Linear Modulation (FiLM).
    3. Update Your MobileUNet to Use These Embeddings:

      • In the forward method, compute the embeddings for both timesteps and conditioning variables.
      • Pass these embeddings to each InvertedResidualBlock.

    Here's the modified code:

    import math
    import torch
    import torch.nn as nn
    
    class SinusoidalPositionEmbeddings(nn.Module):
        def __init__(self, dim):
            super().__init__()
            self.dim = dim
    
        def forward(self, time):
            device = time.device
            half_dim = self.dim // 2
            # Ensure time is a tensor
            if isinstance(time, int) or isinstance(time, float):
                time = torch.tensor([time], device=device)
            embeddings = math.log(10000) / (half_dim - 1)
            embeddings = torch.exp(torch.arange(half_dim, device=device) * -embeddings)
            embeddings = time[:, None] * embeddings[None, :]
            embeddings = torch.cat((torch.sin(embeddings), torch.cos(embeddings)), dim=-1)
            return embeddings
    
    class InvertedResidualBlock(nn.Module):
        """
        Inverted residual block used in MobileNetV2 with conditioning.
        """
        def __init__(self, in_c, out_c, stride, expansion_factor=6, deconvolve=False, embed_dim=512):
            super(InvertedResidualBlock, self).__init__()
            # check stride value
            assert stride in [1, 2]
            self.stride = stride
            self.in_c = in_c
            self.out_c = out_c
            # Skip connection if stride is 1
            self.use_skip_connection = True if self.stride == 1 else False
    
            # expansion factor or t as mentioned in the paper
            ex_c = int(self.in_c * expansion_factor)
            if deconvolve:
                self.conv = nn.Sequential(
                    # pointwise convolution
                    nn.ConvTranspose2d(self.in_c, ex_c, 1, 1, 0, bias=False),
                    nn.BatchNorm2d(ex_c),
                    nn.ReLU6(inplace=True),
                    # depthwise convolution
                    nn.ConvTranspose2d(ex_c, ex_c, 4, self.stride, 1, groups=ex_c, bias=False),
                    nn.BatchNorm2d(ex_c),
                    nn.ReLU6(inplace=True),
                    # pointwise convolution
                    nn.ConvTranspose2d(ex_c, self.out_c, 1, 1, 0, bias=False),
                    nn.BatchNorm2d(self.out_c),
                )
            else:
                self.conv = nn.Sequential(
                    # pointwise convolution
                    nn.Conv2d(self.in_c, ex_c, 1, 1, 0, bias=False),
                    nn.BatchNorm2d(ex_c),
                    nn.ReLU6(inplace=True),
                    # depthwise convolution
                    nn.Conv2d(ex_c, ex_c, 3, self.stride, 1, groups=ex_c, bias=False),
                    nn.BatchNorm2d(ex_c),
                    nn.ReLU6(inplace=True),
                    # pointwise convolution
                    nn.Conv2d(ex_c, self.out_c, 1, 1, 0, bias=False),
                    nn.BatchNorm2d(self.out_c),
                )
            if self.in_c != self.out_c:
                self.conv1x1 = nn.Conv2d(self.in_c, self.out_c, 1, 1, 0, bias=False)
    
            # Define an MLP to process the embedding and produce gamma and beta
            self.embed_dim = embed_dim
    
            self.gamma_beta = nn.Sequential(
                nn.SiLU(),
                nn.Linear(embed_dim, self.out_c * 2)
            )
    
        def forward(self, x, emb):
            out = self.conv(x)
            if self.use_skip_connection:
                if self.in_c != self.out_c:
                    x = self.conv1x1(x)
    
            # Get gamma and beta from embedding
            gamma_beta = self.gamma_beta(emb)
            gamma, beta = gamma_beta.chunk(2, dim=1)
            gamma = gamma[:, :, None, None]
            beta = beta[:, :, None, None]
            out = out * (1 + gamma) + beta
    
            if self.use_skip_connection:
                return x + out
            else:
                return out
    
    class MobileUNet(nn.Module):
        """
        Modified UNet with inverted residual block and depthwise separable convolution.
        Now accepts timesteps and conditioning variables.
        """
    
        def __init__(self, cond_dim=512, time_dim=512):
            super(MobileUNet, self).__init__()
    
            # Time and conditioning dimensionality
            self.time_dim = time_dim
    
            # Define time embedding MLP
            self.time_mlp = nn.Sequential(
                SinusoidalPositionEmbeddings(time_dim),
                nn.Linear(time_dim, time_dim * 4),
                nn.SiLU(),
                nn.Linear(time_dim * 4, time_dim)
            )
    
            # Define conditioning embedding MLP
            self.cond_mlp = nn.Sequential(
                nn.Linear(cond_dim, time_dim * 4),  # Adjust cond_dim as needed
                nn.SiLU(),
                nn.Linear(time_dim * 4, time_dim)
            )
    
            # Encoding arm
            self.conv3x3 = self.depthwise_conv(1, 32, p=1, s=2)
            self.irb_bottleneck1 = self.irb_bottleneck(32, 16, 1, 1, 1)
            self.irb_bottleneck2 = self.irb_bottleneck(16, 24, 2, 2, 6)
            self.irb_bottleneck3 = self.irb_bottleneck(24, 32, 3, 2, 6)
            self.irb_bottleneck4 = self.irb_bottleneck(32, 64, 4, 2, 6)
            self.irb_bottleneck5 = self.irb_bottleneck(64, 96, 3, 1, 6)
            self.irb_bottleneck6 = self.irb_bottleneck(96, 160, 3, 2, 6)
            self.irb_bottleneck7 = self.irb_bottleneck(160, 320, 1, 1, 6)
            self.conv1x1_encode = nn.Conv2d(320, 1280, kernel_size=1, stride=1)
            # Decoding arm
            self.D_irb1 = self.irb_bottleneck(1280, 96, 1, 2, 6, True)
            self.D_irb2 = self.irb_bottleneck(96, 32, 1, 2, 6, True)
            self.D_irb3 = self.irb_bottleneck(32, 24, 1, 2, 6, True)
            self.D_irb4 = self.irb_bottleneck(24, 16, 1, 2, 6, True)
            self.DConv4x4 = nn.ConvTranspose2d(16,16,4,2,1,groups=16, bias=False)
            # Final layer: output channel number can be changed as per the use case
            self.conv1x1_decode = nn.Conv2d(16, 1, kernel_size=1, stride=1)
    
        def depthwise_conv(self, in_c, out_c, k=3, s=1, p=0):
            """
            Optimized convolution by combining depthwise convolution and
            pointwise convolution.
            """
            conv = nn.Sequential(
                nn.Conv2d(in_c, in_c, kernel_size=k, padding=p, groups=in_c, stride=s),
                nn.BatchNorm2d(num_features=in_c),
                nn.ReLU6(inplace=True),
                nn.Conv2d(in_c, out_c, kernel_size=1),
            )
            return conv
    
        def irb_bottleneck(self, in_c, out_c, n, s, t, d=False):
            """
            Create a series of inverted residual blocks.
            """
            blocks = nn.ModuleList()
            blocks.append(InvertedResidualBlock(in_c, out_c, s, t, deconvolve=d, embed_dim=self.time_dim))
            if n > 1:
                for i in range(1, n):
                    blocks.append(InvertedResidualBlock(out_c, out_c, 1, t, deconvolve=d, embed_dim=self.time_dim))
            return blocks
    
        def forward_irb(self, x, blocks, emb):
            for block in blocks:
                x = block(x, emb)
            return x
    
        def get_count(self, model):
            # Simple function to get the count of parameters in a model.
            num = sum(p.numel() for p in model.parameters() if p.requires_grad)
            return num
    
        def forward(self, x, t, c):
            # Get time embeddings
            t_emb = self.time_mlp(t)
            # Get conditioning embeddings
            c_emb = self.cond_mlp(c)
            # Combine embeddings
            emb = t_emb + c_emb
    
            # Left arm/ Encoding arm
            # D1
            x1 = self.conv3x3(x)  # (32, 112, 112)
            x2 = self.forward_irb(x1, self.irb_bottleneck1, emb)  # (16,112,112) s1
            x3 = self.forward_irb(x2, self.irb_bottleneck2, emb)  # (24,56,56) s2
            x4 = self.forward_irb(x3, self.irb_bottleneck3, emb)  # (32,28,28) s3
            x5 = self.forward_irb(x4, self.irb_bottleneck4, emb)  # (64,14,14)
            x6 = self.forward_irb(x5, self.irb_bottleneck5, emb)  # (96,14,14) s4
            x7 = self.forward_irb(x6, self.irb_bottleneck6, emb)  # (160,7,7)
            x8 = self.forward_irb(x7, self.irb_bottleneck7, emb)  # (320,7,7)
            x9 = self.conv1x1_encode(x8)  # (1280,7,7) s5
    
            # Right arm / Decoding arm with skip connections
            d1 = self.forward_irb(x9, self.D_irb1, emb) + x6
            d2 = self.forward_irb(d1, self.D_irb2, emb) + x4
            d3 = self.forward_irb(d2, self.D_irb3, emb) + x3
            d4 = self.forward_irb(d3, self.D_irb4, emb) + x2
            d5 = self.DConv4x4(d4)
            out = self.conv1x1_decode(d5)
            return out
    

    Explanation of Modifications:

    • Embedding Layers:

      • SinusoidalPositionEmbeddings is used to create embeddings for the timesteps.
      • self.time_mlp processes the time embeddings.
      • self.cond_mlp processes the conditioning input embeddings.
    • Modified InvertedResidualBlock:

      • Accepts an embedding vector emb.
      • Uses an MLP (self.gamma_beta) to compute gamma and beta from the embedding.
      • Applies FiLM modulation to the activations: out = out * (1 + gamma) + beta.
    • Adjusted irb_bottleneck Method:

      • Returns an nn.ModuleList instead of an nn.Sequential to allow passing extra arguments.
      • The forward pass uses the helper method forward_irb to iterate through the blocks.
    • Updated Forward Method:

      • Computes embeddings for timesteps (t_emb) and conditioning variables (c_emb).
      • Combines them into a single embedding (emb).
      • Passes emb to each InvertedResidualBlock during the forward pass.

    Usage in Training Loop:

    Now, you can create an instance of your conditional MobileUNet:

    self.diffusion_model = MobileUNet(cond_dim=cond_dim, time_dim=time_dim)
    

    And during the training step, you can call:

    model_output = self.diffusion_model(noisy_y, timesteps, x_latent.flatten(2))
    

    Make sure that timesteps and x_latent.flatten(2) have the appropriate shapes. timesteps should be a tensor of shape [batch_size] or [batch_size, 1], and x_latent.flatten(2) should match the cond_dim specified in the MobileUNet.

    Additional Considerations:

    • Batch Sizes and Shapes:

      • Ensure that all tensors are correctly shaped and on the same device (e.g., GPU).
      • You might need to expand and reshape tensors to match expected dimensions.
    • Adjusting Dimensions:

      • If your conditioning vector x_latent.flatten(2) doesn’t match cond_dim, adjust the cond_dim or process x_latent accordingly.
    • Testing the Model:

      • Before integrating it into your training loop, test the modified MobileUNet with dummy inputs to ensure it runs without errors.

    Final Note:

    By incorporating the timesteps and conditioning variables into your MobileUNet, you're effectively turning it into a conditional U-Net suitable for use in a DDPM or other diffusion-based models. The use of FiLM modulation allows the embeddings to influence the activations throughout the network, enabling the model to generate outputs conditioned on the provided inputs.