Skip to content
The Chat Archive
claude-2.1· Anthropic Arena Human Preference 55K

what happens with temperature <1.0? @torch.nograd() def generate(self, idx,…

  1. User

    what happens with temperature <1.0?

    @torch.no_grad() def generate(self, idx, max_new_tokens, temperature=1.0, top_k=None): """ Take a conditioning sequence of indices idx (LongTensor of shape (b,t)) and complete the sequence max_new_tokens times, feeding the predictions back into the model each time. Most likely you'll want to make sure to be in model.eval() mode of operation for this. """ for _ in range(max_new_tokens): # if the sequence context is growing too long we must crop it at block_size idx_cond = idx if idx.size(1) <= self.config.block_size else idx[:, -self.config.block_size:] # forward the model to get the logits for the index in the sequence logits, _ = self(idx_cond) # pluck the logits at the final step and scale by desired temperature logits = logits[:, -1, :] / temperature # optionally crop the logits to only the top k options if top_k is not None: v, _ = torch.topk(logits, min(top_k, logits.size(-1))) logits[logits < v[:, [-1]]] = -float('Inf') # apply softmax to convert logits to (normalized) probabilities probs = F.softmax(logits, dim=-1) # sample from the distribution idx_next = torch.multinomial(probs, num_samples=1) # append sampled index to the running sequence and continue idx = torch.cat((idx, idx_next), dim=1)

        return idx
    
  2. Assistant

    Good question! Here are a few things that can happen when the temperature parameter is set below 1.0:

    • Sampling becomes more deterministic and less diverse. As the temperature approaches 0, the model will tend to always sample the token with the highest probability. This reduces randomness and diversity in the generated text.

    • Logits get "sharpened". Scaling the logits by a temperature <1 makes the probabilities more "peaked" on the few tokens with highest original logits. The model becomes more confident in its top predictions.

    • Increased repetition. With very low temperatures (e.g. 0.1 or less), the sampling can sometimes get "stuck" repeating the same tokens over and over again rather than generating more diverse continuations.

    • Model errors get amplified. If the model makes a mistake and assigns unduly high probability to an incorrect token at some point, that error can persist for many tokens in low temperatures.

    So in summary, low temperatures reduce randomness and diversity to favor the model's current top predictions. This can be useful when you want the model to commit to a prediction, but too low and sampling becomes repetitive. Values between 0.5-1.0 are commonly used during generation.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.