Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0613· OpenAIHugging Face Arena Human Preference 55K

with torch.nograd(): with torch.autocast('cuda', dtype=torch.float32): outpute =…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4-1106-preview’s answer.
  1. User

    with torch.no_grad(): with torch.autocast('cuda', dtype=torch.float32):

    outpute = modeltt(input, labels=input, use_cache = False)

    outpute = modeltt(input, use_cache = False)

    Выбираем наиболее вероятные токены _, max_indices = torch.max(logits, dim=-1)

    надо получить вот это

    print(f"\nGreedy Search") for i, tok in enumerate(...): score = ...[0][i].cpu() print(f"| {tok:5d} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}| {tokenizer.decode([tok]):8s} ")

  2. Assistant

    Here's a quick guide on how you can achieve this:

    import torch
    import numpy as np
    
    with torch.no_grad():
        with torch.cuda.amp.autocast():
            # outpute = modeltt(input, labels=input, use_cache = False)
            output = modeltt(input, use_cache = False)
    
    # Get the logits from the model output
    logits = output.logits
    
    # Choose the most probable tokens
    _, max_indices = torch.max(logits, dim=-1)
    
    # Convert tensor to list for iteration
    max_indices = max_indices.tolist()
    
    print("\nGreedy Search")
    for i, tok in enumerate(max_indices[0]):
        # Get the logit (score) for the token
        score = logits[0][i].cpu().detach()
    
        print(f"| {tok:5d} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}| {tokenizer.decode([tok]):8s} ")
    

    This script assumes that you have a model called modeltt and an input tensor called input. Also, make sure that tokenizer is initialized with the appropriate tokenizer.

    Please note that the code is using the PyTorch automatic mixed precision (torch.cuda.amp.autocast) which can speed up the computations on your GPU. If you don't have a GPU or you don't want to use mixed precision, you can remove the with torch.cuda.amp.autocast(): line.

  3. User

    with torch.no_grad(): with torch.autocast('cuda', dtype=torch.float32): # Assuming 'input' is your input tensor outpute = modeltt(input, labels=input, use_cache = False) # This line is commented out

    logits are typically the last hidden state in the output of a transformer-based language model

    logits = outpute.logits

    Selecting the most probable tokens

    _, max_indices = torch.max(logits, dim=-1)

    тут видимо не хватает нормализации max_indices

    print(f"\nGreedy Search") for i, tok in enumerate(max_indices.squeeze()): score = logits[0][i][tok].cpu() # Get the score for the specific token print(f"| {tok.item():5d} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%} | {tokenizer.decode([tok]):8s} ")

    Please note the following:

  4. Assistant

    Your code is almost correct. However, there is a small mistake. In the line where you are getting the score for the specific token, you should use squeeze() on max_indices and item() on tok to convert it to a python number. Here's the correct code:

    import torch
    import numpy as np
    
    # Assuming 'input' is your input tensor
    with torch.no_grad():
        with torch.cuda.amp.autocast('cuda', dtype=torch.float32):
            outpute = modeltt(input, labels=input, use_cache = False)
    
    # Logits are typically the last hidden state in the output of a transformer-based language model
    logits = outpute.logits
    
    # Selecting the most probable tokens
    _, max_indices = torch.max(logits, dim=-1)
    
    # Squeeze max_indices to remove the unnecessary dimension
    max_indices = max_indices.squeeze()
    
    print(f"\nGreedy Search")
    for i, tok in enumerate(max_indices):
        tok = tok.item()  # Convert torch.Tensor to python number
        score = logits[0][i][tok].cpu()  # Get the score for the specific token
        print(f"| {tok:5d} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%} | {tokenizer.decode([tok]):8s} ")
    

    This script assumes that you have a model called modeltt and an input tensor called input. Also, make sure that tokenizer is initialized with the appropriate tokenizer.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.