class MutanFusion(AbstractFusion): def init(self, _C, visualembedding=True,…
- User
class MutanFusion(AbstractFusion):
def __init__(self, __C, visual_embedding=True, question_embedding=True): super(MutanFusion, self).__init__(__C) self.visual_embedding = visual_embedding self.question_embedding = question_embedding if self.visual_embedding: self.linear_v = nn.Linear(__C.IMG_FEAT_SIZE, __C.HIDDEN_SIZE) else: print('no visual embedding!') if self.question_embedding: self.linear_q = nn.Linear(1024, __C.HIDDEN_SIZE) else: print('no question embedding!') self.list_linear_hv = nn.ModuleList([ nn.Linear(__C.HIDDEN_SIZE, 1024) for i in range(5) ]) self.list_linear_hq = nn.ModuleList([ nn.Linear(__C.HIDDEN_SIZE, 1024) for i in range(5) ]) def forward(self, input_v, input_q): if input_v.dim() != input_q.dim() and input_v.dim() != 2: raise ValueError batch_size = input_v.size(0) if self.visual_embedding: x_v = F.dropout(input_v, p = 0.5, training=self.training) x_v = self.linear_v(x_v) x_v = getattr(torch, "tanh")(x_v) else: x_v = input_v if self.question_embedding: x_q = F.dropout(input_q, p = 0.5, training=self.training) x_q = self.linear_q(x_q) x_q = getattr(torch, "tanh")(x_q) else: x_q = input_q x_mm = [] for i in range(5): x_hv = F.dropout(x_v, p=0, training=self.training) x_hv = self.list_linear_hv[i](x_hv) x_hv = getattr(torch, "tanh")(x_hv) x_hq = F.dropout(x_q, p=0, training=self.training) x_hq = self.list_linear_hq[i](x_hq) x_hq = getattr(torch, "tanh")(x_hq) x_mm.append(torch.mul(x_hq, x_hv)) #矩阵点乘 x_mm = torch.stack(x_mm, dim=1) #拼接 x_mm = x_mm.sum(1).view(64, 1024) x_mm = getattr(torch, "tanh")(x_mm) return x_mm - Assistant
This code defines a class called MutanFusion that extends the AbstractFusion class. It implements a fusion method using the Mutan fusion model.
The MutanFusion class has two main attributes: visual_embedding and question_embedding, which indicate whether visual embedding and question embedding should be performed, respectively.
In the constructor, the class initializes the parent class (AbstractFusion) and sets the visual_embedding and question_embedding attributes. It also initializes linear_v and linear_q, which are linear layers used for visual and question embedding, respectively. The module lists list_linear_hv and list_linear_hq are used to store linear layers for the fusion operation.
The forward method takes two input tensors, input_v and input_q, which represent the visual and question features, respectively. If visual_embedding is set to true, it applies dropout to input_v and passes it through linear_v and tanh activation. If question_embedding is set to true, it applies dropout to input_q and passes it through linear_q and tanh activation.
The method then performs the fusion operation for the given inputs using the Mutan fusion model. For each of the five fusion levels, it applies dropout to x_v and x_q, passes them through the corresponding linear layers in list_linear_hv and list_linear_hq, and applies tanh activation. It then multiplies x_hq and x_hv (element-wise multiplication) and adds the result to the x_mm list.
After the loop, x_mm is stacked along the second dimension and summed along that dimension to obtain a single tensor. This tensor is then passed through tanh activation and returned as the output of the fusion operation.
- User
Finish! Evaluation: [step 0/ 23] Traceback (most recent call last): File "run.py", line 141, in execution.run(__C.RUN_MODE) File "/root/autodl-tmp/mcan-vqa-master1/core/exec.py", line 526, in run self.eval(self.dataset, valid=True) File "/root/autodl-tmp/mcan-vqa-master1/core/exec.py", line 362, in eval pred = net( File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/root/autodl-tmp/mcan-vqa-master1/core/model/net.py", line 231, in forward proj_feat = self.mutanfusion(img_feat,lang_feat) File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/root/autodl-tmp/mcan-vqa-master1/core/model/net.py", line 129, in forward x_mm = x_mm.sum(1).view(64, 1024) RuntimeError: shape '[64, 1024]' is invalid for input of size 32768
- Assistant
The error "RuntimeError: shape '[64, 1024]' is invalid for input of size 32768" indicates that there is a mismatch in the input sizes.
In the forward method of the MutanFusion class, there is a line where the x_mm tensor is reshaped using the view method to have a size of [64, 1024]. However, the input tensor size is actually [batch_size, 1024].
To fix this error, you need to replace the hard-coded size in the view method with the actual batch size, which can be obtained using the batch_size variable. Modify the line as follows:
x_mm = x_mm.sum(1).view(batch_size, 1024)
This will ensure that the reshaping is done correctly based on the actual batch size.