is this applied to each token individually? Or is it applied to a sequence of tokens…
- User
is this applied to each token individually? Or is it applied to a sequence of tokens that talk to each other?
class MLP(nn.Module):
def __init__(self, config): super().__init__() self.c_fc = nn.Linear(config.n_embd, 4 * config.n_embd, bias=config.bias) self.gelu = nn.GELU() self.c_proj = nn.Linear(4 * config.n_embd, config.n_embd, bias=config.bias) self.dropout = nn.Dropout(config.dropout) def forward(self, x): x = self.c_fc(x) x = self.gelu(x) x = self.c_proj(x) x = self.dropout(x) return x Assistant
The MLP is applied to each token individually.
The input
xto the MLP is a tensor of shape(batch_size, sequence_length, hidden_size)wherebatch_sizeis the number of sequences in the batch,sequence_lengthis the length of each sequence, andhidden_sizeis the size of the hidden state of each token.The output of the MLP is a tensor of the same shape as the input, where each token has been independently transformed by the MLP.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.