Skip to content
TheChatArchiveThe Chat Archive
o3-mini· OpenAIHugging Face LMArena Human Preference 140K

if HIDDEN_DIM=100?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Googlegemma-3-27b-it’s answer.
  1. User

    if HIDDEN_DIM=100?

  2. Assistant

    Changing HIDDEN_DIM from 1 to 100 means you want your internal (hidden) representation to be a 100-dimensional vector rather than a 1-dimensional value. In a neural network, the hidden dimension controls the capacity of the model—that is, how many features the network can learn to represent from its input before the final prediction layer. However, there are some important points to consider:

    1. Increased Representation Power:
      Increasing HIDDEN_DIM to 100 generally gives the network more capacity to learn complex patterns from the data. The hidden layers, which perform nonlinear transformations on the input, will now be working with more features.

    2. Architectural Consistency:
      In your code, after the embedding layer you take a mean over the sequence dimension, so you end up with a tensor of shape [batch, EMB_DIM]. Because you use seven activation layers that operate elementwise, the tensor’s shape doesn’t change—it remains [batch, EMB_DIM] through those layers. Finally, you pass it to the output layer, which is constructed as:

      Linear::new(
          vs.get((HIDDEN_DIM, VOCAB_SIZE as usize), "output.weight")?,
          Some(vs.get(VOCAB_SIZE as usize, "output.bias")?),
      );
      

      Here the output layer is expecting an input of shape [batch, HIDDEN_DIM].

      If you change HIDDEN_DIM to 100 without adjusting EMB_DIM or adding an intermediate linear layer to convert your 1-dimensional embeddings into 100 dimensions, you'll encounter a shape mismatch. Essentially, your model will still be outputting [batch, 1] from the embeddings and activations, but the output layer is now expecting [batch, 100].

    3. What Needs to Change:

      • Either update EMB_DIM to 100 so that the initial representation is directly 100-dimensional, ensuring that after the average pooling and activations the tensor has shape [batch, 100].
      • Or insert an additional transformation (for example, a linear layer that maps from EMB_DIM to HIDDEN_DIM) before feeding the data to the seven activation layers or just prior to the output layer.
    4. Example Adjustments:
      If you’d like to keep your embedding dimension small (say, 1) but still want a 100-dimensional hidden state, you could modify your model like this:

      struct TheNn {
          embed: Embedding,
          to_hidden: Linear,
          l1: Activation,
          l2: Activation,
          l3: Activation,
          l4: Activation,
          l5: Activation,
          l6: Activation,
          l7: Activation,
          output: Linear,
      }
      
      impl TheNn {
          fn new(vs: VarBuilder) -> Result<Self> {
              let embed = Embedding::new(
                  vs.get((VOCAB_SIZE as usize, EMB_DIM), "embed.weight")?,
                  VOCAB_SIZE as _,
              );
      
              // Map from EMB_DIM to HIDDEN_DIM:
              let to_hidden = Linear::new(
                  vs.get((EMB_DIM, HIDDEN_DIM), "to_hidden.weight")?,
                  Some(vs.get(HIDDEN_DIM, "to_hidden.bias")?),
              );
      
              // Define seven activation layers (as before)
              let [l1, l2, l3, l4, l5, l6, l7] = [Activation::Swiglu; 7];
      
              let output = Linear::new(
                  vs.get((HIDDEN_DIM, VOCAB_SIZE as usize), "output.weight")?,
                  Some(vs.get(VOCAB_SIZE as usize, "output.bias")?),
              );
      
              Ok(Self {
                  embed,
                  to_hidden,
                  l1,
                  l2,
                  l3,
                  l4,
                  l5,
                  l6,
                  l7,
                  output,
              })
          }
      
          fn forward(&self, ids: &Tensor) -> Result<Tensor> {
              let x = self.embed.forward(ids)?; // (b, s, emb)
              let x = x.mean(D::Minus2)?; // global average: (b, emb)
              let x = self.to_hidden.forward(&x)?; // now (b, hidden)
      
              let x = self.l1.forward(&x)?;
              let x = self.l2.forward(&x)?;
              let x = self.l3.forward(&x)?;
              let x = self.l4.forward(&x)?;
              let x = self.l5.forward(&x)?;
              let x = self.l6.forward(&x)?;
              let x = self.l7.forward(&x)?;
      
              self.output.forward(&x)
          }
      }
      

      With this change, if you set:

      const HIDDEN_DIM: usize = 100;
      

      Even if EMB_DIM is still 1, the to_hidden layer will convert the 1-dimensional embedding into a 100-dimensional hidden representation.

    5. In Summary:
      If you set HIDDEN_DIM = 100, you are increasing the model’s representational capacity in the hidden layers. But you must ensure that the data flowing into the output layer has the correct shape—either by increasing the embedding dimension or by adding a transformation to map the current dimension to 100. Otherwise, you will run into shape mismatches.

    Let me know if you need any further clarification!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.