I have this Mobile U-Net. I want to make it conditional and use it in my DDPM training…
- User
I have this Mobile U-Net. I want to make it conditional and use it in my DDPM training loop. Mobile Unet is as follow: # Mobile UNet and Inverted Residual Block
Author: Lafith Mattara
Date: July 2022
import torch import torch.nn as nn
class InvertedResidualBlock(nn.Module): """ inverted residual block used in MobileNetV2 """ def init(self, in_c, out_c, stride, expansion_factor=6, deconvolve=False): super(InvertedResidualBlock, self).init() # check stride value assert stride in [1, 2] self.stride = stride self.in_c = in_c self.out_c = out_c # Skip connection if stride is 1 self.use_skip_connection = True if self.stride == 1 else False
# expansion factor or t as mentioned in the paper ex_c = int(self.in_c * expansion_factor) if deconvolve: self.conv = nn.Sequential( # pointwise convolution nn.Conv2d(self.in_c, ex_c, 1, 1, 0, bias=False), nn.BatchNorm2d(ex_c), nn.ReLU6(inplace=True), # depthwise convolution nn.ConvTranspose2d(ex_c, ex_c, 4,self.stride,1, groups=ex_c, bias=False), nn.BatchNorm2d(ex_c), nn.ReLU6(inplace=True), # pointwise convolution nn.Conv2d(ex_c, self.out_c, 1, 1, 0, bias=False), nn.BatchNorm2d(self.out_c), ) else: self.conv = nn.Sequential( # pointwise convolution nn.Conv2d(self.in_c, ex_c, 1, 1, 0, bias=False), nn.BatchNorm2d(ex_c), nn.ReLU6(inplace=True), # depthwise convolution nn.Conv2d(ex_c, ex_c, 3, self.stride, 1, groups=ex_c, bias=False), nn.BatchNorm2d(ex_c), nn.ReLU6(inplace=True), # pointwise convolution nn.Conv2d(ex_c, self.out_c, 1, 1, 0, bias=False), nn.BatchNorm2d(self.out_c), ) self.conv1x1 = nn.Conv2d(self.in_c, self.out_c, 1, 1, 0, bias=False) def forward(self, x): if self.use_skip_connection: out = self.conv(x) if self.in_c != self.out_c: x = self.conv1x1(x) return x+out else: return self.conv(x)class MobileUNet(nn.Module): """ Modified UNet with inverted residual block and depthwise seperable convolution """
def __init__(self): super(MobileUNet, self).__init__() # encoding arm self.conv3x3 = self.depthwise_conv(1, 32, p=1, s=2) self.irb_bottleneck1 = self.irb_bottleneck(32, 16, 1, 1, 1) self.irb_bottleneck2 = self.irb_bottleneck(16, 24, 2, 2, 6) self.irb_bottleneck3 = self.irb_bottleneck(24, 32, 3, 2, 6) self.irb_bottleneck4 = self.irb_bottleneck(32, 64, 4, 2, 6) self.irb_bottleneck5 = self.irb_bottleneck(64, 96, 3, 1, 6) self.irb_bottleneck6 = self.irb_bottleneck(96, 160, 3, 2, 6) self.irb_bottleneck7 = self.irb_bottleneck(160, 320, 1, 1, 6) self.conv1x1_encode = nn.Conv2d(320, 1280, kernel_size=1, stride=1) # decoding arm self.D_irb1 = self.irb_bottleneck(1280, 96, 1, 2, 6, True) self.D_irb2 = self.irb_bottleneck(96, 32, 1, 2, 6, True) self.D_irb3 = self.irb_bottleneck(32, 24, 1, 2, 6, True) self.D_irb4 = self.irb_bottleneck(24, 16, 1, 2, 6, True) self.DConv4x4 = nn.ConvTranspose2d(16,16,4,2,1,groups=16, bias=False) # Final layer: output channel number can be changed as per the usecase self.conv1x1_decode = nn.Conv2d(16, 1, kernel_size=1, stride=1) def depthwise_conv(self, in_c, out_c, k=3, s=1, p=0): """ optimized convolution by combining depthwise convolution and pointwise convolution. """ conv = nn.Sequential( nn.Conv2d(in_c, in_c, kernel_size=k, padding=p, groups=in_c, stride=s), nn.BatchNorm2d(num_features=in_c), nn.ReLU6(inplace=True), nn.Conv2d(in_c, out_c, kernel_size=1), ) return conv def irb_bottleneck(self, in_c, out_c, n, s, t, d=False): """ create a series of inverted residual blocks. """ convs = [] xx = InvertedResidualBlock(in_c, out_c, s, t, deconvolve=d) convs.append(xx) if n>1: for i in range(1,n): xx = InvertedResidualBlock(out_c, out_c, 1, t, deconvolve=d) convs.append(xx) conv = nn.Sequential(*convs) return conv def get_count(self, model): # simple function to get the count of parameters in a model. num = sum(p.numel() for p in model.parameters() if p.requires_grad) return num def forward(self, x): # Left arm/ Encoding arm #D1 x1 = self.conv3x3(x) #(32, 112, 112) x2 = self.irb_bottleneck1(x1) #(16,112,112) s1 x3 = self.irb_bottleneck2(x2) #(24,56,56) s2 x4 = self.irb_bottleneck3(x3) #(32,28,28) s3 x5 = self.irb_bottleneck4(x4) #(64,14,14) x6 = self.irb_bottleneck5(x5) #(96,14,14) s4 x7 = self.irb_bottleneck6(x6) #(160,7,7) x8 = self.irb_bottleneck7(x7) #(320,7,7) x9 = self.conv1x1_encode(x8) #(1280,7,7) s5 # Right arm / Decoding arm with skip connections d1 = self.D_irb1(x9) + x6 d2 = self.D_irb2(d1) + x4 d3 = self.D_irb3(d2) + x3 d4 = self.D_irb4(d3) + x2 d5 = self.DConv4x4(d4) out = self.conv1x1_decode(d5) return outI want to have it such that self.diffusion_model = MobileUNet() should be the Unet in diffusion process and the training step would be like model_output= self.diffusion_model(noisy_y, timesteps, x_latent.flatten(2))
- Assistant
Sure! To adapt your
MobileUNetto be conditional and use it in a DDPM training loop, you'll need to modify it to accept additional inputs liketimestepsandx_latent(flattened as needed). You'll also need to adjust the architecture to incorporate these conditioning inputs appropriately.Here’s how you can modify your
MobileUNet:-
Add Embedding Layers for Timesteps and Conditioning Variables:
- Use a sinusoidal positional embedding for the timesteps.
- Use a simple MLP to process the conditioning input (
x_latent) to produce embeddings.
-
Modify Your InvertedResidualBlock to Accept Embeddings:
- Adjust
InvertedResidualBlockto accept an embedding vector. - Use this embedding to produce scaling (
gamma) and shifting (beta) parameters via an MLP. - Apply these parameters to modulate the activations using Feature-wise Linear Modulation (FiLM).
- Adjust
-
Update Your MobileUNet to Use These Embeddings:
- In the
forwardmethod, compute the embeddings for both timesteps and conditioning variables. - Pass these embeddings to each
InvertedResidualBlock.
- In the
Here's the modified code:
import math import torch import torch.nn as nn class SinusoidalPositionEmbeddings(nn.Module): def __init__(self, dim): super().__init__() self.dim = dim def forward(self, time): device = time.device half_dim = self.dim // 2 # Ensure time is a tensor if isinstance(time, int) or isinstance(time, float): time = torch.tensor([time], device=device) embeddings = math.log(10000) / (half_dim - 1) embeddings = torch.exp(torch.arange(half_dim, device=device) * -embeddings) embeddings = time[:, None] * embeddings[None, :] embeddings = torch.cat((torch.sin(embeddings), torch.cos(embeddings)), dim=-1) return embeddings class InvertedResidualBlock(nn.Module): """ Inverted residual block used in MobileNetV2 with conditioning. """ def __init__(self, in_c, out_c, stride, expansion_factor=6, deconvolve=False, embed_dim=512): super(InvertedResidualBlock, self).__init__() # check stride value assert stride in [1, 2] self.stride = stride self.in_c = in_c self.out_c = out_c # Skip connection if stride is 1 self.use_skip_connection = True if self.stride == 1 else False # expansion factor or t as mentioned in the paper ex_c = int(self.in_c * expansion_factor) if deconvolve: self.conv = nn.Sequential( # pointwise convolution nn.ConvTranspose2d(self.in_c, ex_c, 1, 1, 0, bias=False), nn.BatchNorm2d(ex_c), nn.ReLU6(inplace=True), # depthwise convolution nn.ConvTranspose2d(ex_c, ex_c, 4, self.stride, 1, groups=ex_c, bias=False), nn.BatchNorm2d(ex_c), nn.ReLU6(inplace=True), # pointwise convolution nn.ConvTranspose2d(ex_c, self.out_c, 1, 1, 0, bias=False), nn.BatchNorm2d(self.out_c), ) else: self.conv = nn.Sequential( # pointwise convolution nn.Conv2d(self.in_c, ex_c, 1, 1, 0, bias=False), nn.BatchNorm2d(ex_c), nn.ReLU6(inplace=True), # depthwise convolution nn.Conv2d(ex_c, ex_c, 3, self.stride, 1, groups=ex_c, bias=False), nn.BatchNorm2d(ex_c), nn.ReLU6(inplace=True), # pointwise convolution nn.Conv2d(ex_c, self.out_c, 1, 1, 0, bias=False), nn.BatchNorm2d(self.out_c), ) if self.in_c != self.out_c: self.conv1x1 = nn.Conv2d(self.in_c, self.out_c, 1, 1, 0, bias=False) # Define an MLP to process the embedding and produce gamma and beta self.embed_dim = embed_dim self.gamma_beta = nn.Sequential( nn.SiLU(), nn.Linear(embed_dim, self.out_c * 2) ) def forward(self, x, emb): out = self.conv(x) if self.use_skip_connection: if self.in_c != self.out_c: x = self.conv1x1(x) # Get gamma and beta from embedding gamma_beta = self.gamma_beta(emb) gamma, beta = gamma_beta.chunk(2, dim=1) gamma = gamma[:, :, None, None] beta = beta[:, :, None, None] out = out * (1 + gamma) + beta if self.use_skip_connection: return x + out else: return out class MobileUNet(nn.Module): """ Modified UNet with inverted residual block and depthwise separable convolution. Now accepts timesteps and conditioning variables. """ def __init__(self, cond_dim=512, time_dim=512): super(MobileUNet, self).__init__() # Time and conditioning dimensionality self.time_dim = time_dim # Define time embedding MLP self.time_mlp = nn.Sequential( SinusoidalPositionEmbeddings(time_dim), nn.Linear(time_dim, time_dim * 4), nn.SiLU(), nn.Linear(time_dim * 4, time_dim) ) # Define conditioning embedding MLP self.cond_mlp = nn.Sequential( nn.Linear(cond_dim, time_dim * 4), # Adjust cond_dim as needed nn.SiLU(), nn.Linear(time_dim * 4, time_dim) ) # Encoding arm self.conv3x3 = self.depthwise_conv(1, 32, p=1, s=2) self.irb_bottleneck1 = self.irb_bottleneck(32, 16, 1, 1, 1) self.irb_bottleneck2 = self.irb_bottleneck(16, 24, 2, 2, 6) self.irb_bottleneck3 = self.irb_bottleneck(24, 32, 3, 2, 6) self.irb_bottleneck4 = self.irb_bottleneck(32, 64, 4, 2, 6) self.irb_bottleneck5 = self.irb_bottleneck(64, 96, 3, 1, 6) self.irb_bottleneck6 = self.irb_bottleneck(96, 160, 3, 2, 6) self.irb_bottleneck7 = self.irb_bottleneck(160, 320, 1, 1, 6) self.conv1x1_encode = nn.Conv2d(320, 1280, kernel_size=1, stride=1) # Decoding arm self.D_irb1 = self.irb_bottleneck(1280, 96, 1, 2, 6, True) self.D_irb2 = self.irb_bottleneck(96, 32, 1, 2, 6, True) self.D_irb3 = self.irb_bottleneck(32, 24, 1, 2, 6, True) self.D_irb4 = self.irb_bottleneck(24, 16, 1, 2, 6, True) self.DConv4x4 = nn.ConvTranspose2d(16,16,4,2,1,groups=16, bias=False) # Final layer: output channel number can be changed as per the use case self.conv1x1_decode = nn.Conv2d(16, 1, kernel_size=1, stride=1) def depthwise_conv(self, in_c, out_c, k=3, s=1, p=0): """ Optimized convolution by combining depthwise convolution and pointwise convolution. """ conv = nn.Sequential( nn.Conv2d(in_c, in_c, kernel_size=k, padding=p, groups=in_c, stride=s), nn.BatchNorm2d(num_features=in_c), nn.ReLU6(inplace=True), nn.Conv2d(in_c, out_c, kernel_size=1), ) return conv def irb_bottleneck(self, in_c, out_c, n, s, t, d=False): """ Create a series of inverted residual blocks. """ blocks = nn.ModuleList() blocks.append(InvertedResidualBlock(in_c, out_c, s, t, deconvolve=d, embed_dim=self.time_dim)) if n > 1: for i in range(1, n): blocks.append(InvertedResidualBlock(out_c, out_c, 1, t, deconvolve=d, embed_dim=self.time_dim)) return blocks def forward_irb(self, x, blocks, emb): for block in blocks: x = block(x, emb) return x def get_count(self, model): # Simple function to get the count of parameters in a model. num = sum(p.numel() for p in model.parameters() if p.requires_grad) return num def forward(self, x, t, c): # Get time embeddings t_emb = self.time_mlp(t) # Get conditioning embeddings c_emb = self.cond_mlp(c) # Combine embeddings emb = t_emb + c_emb # Left arm/ Encoding arm # D1 x1 = self.conv3x3(x) # (32, 112, 112) x2 = self.forward_irb(x1, self.irb_bottleneck1, emb) # (16,112,112) s1 x3 = self.forward_irb(x2, self.irb_bottleneck2, emb) # (24,56,56) s2 x4 = self.forward_irb(x3, self.irb_bottleneck3, emb) # (32,28,28) s3 x5 = self.forward_irb(x4, self.irb_bottleneck4, emb) # (64,14,14) x6 = self.forward_irb(x5, self.irb_bottleneck5, emb) # (96,14,14) s4 x7 = self.forward_irb(x6, self.irb_bottleneck6, emb) # (160,7,7) x8 = self.forward_irb(x7, self.irb_bottleneck7, emb) # (320,7,7) x9 = self.conv1x1_encode(x8) # (1280,7,7) s5 # Right arm / Decoding arm with skip connections d1 = self.forward_irb(x9, self.D_irb1, emb) + x6 d2 = self.forward_irb(d1, self.D_irb2, emb) + x4 d3 = self.forward_irb(d2, self.D_irb3, emb) + x3 d4 = self.forward_irb(d3, self.D_irb4, emb) + x2 d5 = self.DConv4x4(d4) out = self.conv1x1_decode(d5) return outExplanation of Modifications:
-
Embedding Layers:
SinusoidalPositionEmbeddingsis used to create embeddings for the timesteps.self.time_mlpprocesses the time embeddings.self.cond_mlpprocesses the conditioning input embeddings.
-
Modified InvertedResidualBlock:
- Accepts an embedding vector
emb. - Uses an MLP (
self.gamma_beta) to computegammaandbetafrom the embedding. - Applies FiLM modulation to the activations:
out = out * (1 + gamma) + beta.
- Accepts an embedding vector
-
Adjusted
irb_bottleneckMethod:- Returns an
nn.ModuleListinstead of annn.Sequentialto allow passing extra arguments. - The forward pass uses the helper method
forward_irbto iterate through the blocks.
- Returns an
-
Updated Forward Method:
- Computes embeddings for timesteps (
t_emb) and conditioning variables (c_emb). - Combines them into a single embedding (
emb). - Passes
embto eachInvertedResidualBlockduring the forward pass.
- Computes embeddings for timesteps (
Usage in Training Loop:
Now, you can create an instance of your conditional
MobileUNet:self.diffusion_model = MobileUNet(cond_dim=cond_dim, time_dim=time_dim)And during the training step, you can call:
model_output = self.diffusion_model(noisy_y, timesteps, x_latent.flatten(2))Make sure that
timestepsandx_latent.flatten(2)have the appropriate shapes.timestepsshould be a tensor of shape[batch_size]or[batch_size, 1], andx_latent.flatten(2)should match thecond_dimspecified in theMobileUNet.Additional Considerations:
-
Batch Sizes and Shapes:
- Ensure that all tensors are correctly shaped and on the same device (e.g., GPU).
- You might need to expand and reshape tensors to match expected dimensions.
-
Adjusting Dimensions:
- If your conditioning vector
x_latent.flatten(2)doesn’t matchcond_dim, adjust thecond_dimor processx_latentaccordingly.
- If your conditioning vector
-
Testing the Model:
- Before integrating it into your training loop, test the modified
MobileUNetwith dummy inputs to ensure it runs without errors.
- Before integrating it into your training loop, test the modified
Final Note:
By incorporating the timesteps and conditioning variables into your
MobileUNet, you're effectively turning it into a conditional U-Net suitable for use in a DDPM or other diffusion-based models. The use of FiLM modulation allows the embeddings to influence the activations throughout the network, enabling the model to generate outputs conditioned on the provided inputs. -