Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0301· openaiHugging Face Suzhen/CodeChat

the following pytorch model is giving me "TypeError: forward() got an unexpected keyword…

  1. User

    the following pytorch model is giving me "TypeError: forward() got an unexpected keyword argument 'input_mask'":

    class transformer_test(nn.Module): def init(self, dim_feedforward=100, num_encoder_layers=5): super(transformer_test, self).init() self.vocab_size = 256 self.emb_size = 512 self.nhead = 4 self.embedding = nn.Embedding(self.vocab_size, self.emb_size) self.transformer = nn.Transformer(d_model=self.emb_size, nhead=self.nhead, num_encoder_layers=num_encoder_layers, dim_feedforward=dim_feedforward) self.decoder = nn.Linear(self.emb_size, self.vocab_size)

    def forward(self, input):
    	input = input.cuda()
    	input = self.embedding(input)
    	input_mask = self._generate_square_subsequent_mask(len(input)).to(input.device)
    	output = self.transformer(input, input_mask=input_mask)
    	output = self.decoder(output[-1, :, :])
    	return output
    
    def _generate_square_subsequent_mask(self, sz):
    	mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
    	mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
    	return mask
    
  2. Assistant

    The error message suggests that the forward() function is being called with an unexpected keyword argument input_mask. This could be happening if you are calling the forward() function with incorrect argument names.

    To solve the issue, make sure that you call the forward() function with the correct argument names. In this case, the input_mask argument is expected by the transformer module inside the forward() function. Therefore, you need to pass the input_mask argument as shown below:

    model = transformer_test()
    input = torch.rand(10, 20)
    input_mask = model._generate_square_subsequent_mask(20)
    output = model(input, input_mask=input_mask)
    

    Here, input_mask is generated using the _generate_square_subsequent_mask() function defined in the model. You can also pass a pre-defined input_mask tensor instead of generating it using the function.

  3. User

    This error happend at the line "output = self.transformer(input, input_mask=input_mask)".

  4. Assistant

    Yes, that's correct. The forward() method is trying to pass input_mask as a keyword argument to the self.transformer() method, but the self.transformer() method is not expecting this keyword argument. So, you need to modify the forward() method to remove this argument.

    To fix this issue, you can change the forward() method as follows:

    class transformer_test(nn.Module): 
        def __init__(self, dim_feedforward=100, num_encoder_layers=5):
            super(transformer_test, self).__init__()
            self.vocab_size = 256
            self.emb_size = 512
            self.nhead = 4
            self.embedding = nn.Embedding(self.vocab_size, self.emb_size)
            self.transformer = nn.Transformer(d_model=self.emb_size, nhead=self.nhead, num_encoder_layers=num_encoder_layers, dim_feedforward=dim_feedforward)
            self.decoder = nn.Linear(self.emb_size, self.vocab_size)
        
        def forward(self, input, input_mask=None):
            input = input.cuda()
            input = self.embedding(input)
            if input_mask is not None:
                output = self.transformer(input, input_mask=input_mask)
            else:
                output = self.transformer(input)
            output = self.decoder(output[-1, :, :])
            return output
    

    In this modified version of the forward() method, we have added an optional parameter input_mask=None to the function header. If an input mask is provided, then we use that mask while calling the self.transformer() method. If no input mask is provided, we call the self.transformer() method without any keyword arguments.

    This should solve the current error that you are facing.