Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-pro· GoogleHugging Face Arena Human Preference 55K

is this applied to each token individually? Or is it applied to a sequence of tokens…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with vicuna-33b’s answer.
  1. User

    is this applied to each token individually? Or is it applied to a sequence of tokens that talk to each other?

    class MLP(nn.Module):

    def __init__(self, config):
        super().__init__()
        self.c_fc    = nn.Linear(config.n_embd, 4 * config.n_embd, bias=config.bias)
        self.gelu    = nn.GELU()
        self.c_proj  = nn.Linear(4 * config.n_embd, config.n_embd, bias=config.bias)
        self.dropout = nn.Dropout(config.dropout)
    
    def forward(self, x):
        x = self.c_fc(x)
        x = self.gelu(x)
        x = self.c_proj(x)
        x = self.dropout(x)
        return x
    
  2. Gemini Assistant

    The MLP is applied to each token individually.

    The input x to the MLP is a tensor of shape (batch_size, sequence_length, hidden_size) where batch_size is the number of sequences in the batch, sequence_length is the length of each sequence, and hidden_size is the size of the hidden state of each token.

    The output of the MLP is a tensor of the same shape as the input, where each token has been independently transformed by the MLP.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.