with torch.nograd(): with torch.autocast('cuda', dtype=torch.float32): outpute =…
- User
with torch.no_grad(): with torch.autocast('cuda', dtype=torch.float32):
outpute = modeltt(input, labels=input, use_cache = False)
outpute = modeltt(input, use_cache = False)
Выбираем наиболее вероятные токены _, max_indices = torch.max(logits, dim=-1)
надо получить вот это
print(f"\nGreedy Search") for i, tok in enumerate(...): score = ...[0][i].cpu() print(f"| {tok:5d} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}| {tokenizer.decode([tok]):8s} ")
- Assistant
Here's a quick guide on how you can achieve this:
import torch import numpy as np with torch.no_grad(): with torch.cuda.amp.autocast(): # outpute = modeltt(input, labels=input, use_cache = False) output = modeltt(input, use_cache = False) # Get the logits from the model output logits = output.logits # Choose the most probable tokens _, max_indices = torch.max(logits, dim=-1) # Convert tensor to list for iteration max_indices = max_indices.tolist() print("\nGreedy Search") for i, tok in enumerate(max_indices[0]): # Get the logit (score) for the token score = logits[0][i].cpu().detach() print(f"| {tok:5d} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}| {tokenizer.decode([tok]):8s} ")This script assumes that you have a model called
modelttand an input tensor calledinput. Also, make sure thattokenizeris initialized with the appropriate tokenizer.Please note that the code is using the PyTorch automatic mixed precision (
torch.cuda.amp.autocast) which can speed up the computations on your GPU. If you don't have a GPU or you don't want to use mixed precision, you can remove thewith torch.cuda.amp.autocast():line. - User
with torch.no_grad(): with torch.autocast('cuda', dtype=torch.float32): # Assuming 'input' is your input tensor outpute = modeltt(input, labels=input, use_cache = False) # This line is commented out
logits are typically the last hidden state in the output of a transformer-based language model
logits = outpute.logits
Selecting the most probable tokens
_, max_indices = torch.max(logits, dim=-1)
тут видимо не хватает нормализации max_indices
print(f"\nGreedy Search") for i, tok in enumerate(max_indices.squeeze()): score = logits[0][i][tok].cpu() # Get the score for the specific token print(f"| {tok.item():5d} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%} | {tokenizer.decode([tok]):8s} ")
Please note the following:
- Assistant
Your code is almost correct. However, there is a small mistake. In the line where you are getting the score for the specific token, you should use
squeeze()onmax_indicesanditem()ontokto convert it to a python number. Here's the correct code:import torch import numpy as np # Assuming 'input' is your input tensor with torch.no_grad(): with torch.cuda.amp.autocast('cuda', dtype=torch.float32): outpute = modeltt(input, labels=input, use_cache = False) # Logits are typically the last hidden state in the output of a transformer-based language model logits = outpute.logits # Selecting the most probable tokens _, max_indices = torch.max(logits, dim=-1) # Squeeze max_indices to remove the unnecessary dimension max_indices = max_indices.squeeze() print(f"\nGreedy Search") for i, tok in enumerate(max_indices): tok = tok.item() # Convert torch.Tensor to python number score = logits[0][i][tok].cpu() # Get the score for the specific token print(f"| {tok:5d} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%} | {tokenizer.decode([tok]):8s} ")This script assumes that you have a model called
modelttand an input tensor calledinput. Also, make sure thattokenizeris initialized with the appropriate tokenizer.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.