Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0125-preview· OpenAIAi2 WildChat-4.8M (AI2)

Hello Chatgpt, I have a problem. I am attempting to manually apply a TopPLogitsWarper to…

  1. User

    Hello Chatgpt, I have a problem. I am attempting to manually apply a TopPLogitsWarper to the output of a GPT2LMHeadModel. My code is as below for the external part of my code which extracts the LM's logits.

        with torch.no_grad():
            out_logits = model(encoded_batch, attention_mask=attn_mask).logits
            # torch.save(out_logits, "outlogits.pt")
    
        from transformers import TopPLogitsWarper
        logits_warp = TopPLogitsWarper(0.9)
        shift_logits = out_logits[..., :-1, :].contiguous()
        out_logits = logits_warp.__call__(input_ids=None, scores = shift_logits)
        .... 
    

    However, when attempting to use the TopPLogitsWarper, with the code below, it gives the following error at the line 'indices_to_remove = sorted_indices_to_remove.scatter(1, sorted_indices, sorted_indices_to_remove)' where it states ' [86,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.

    Below is the code for call:

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
        sorted_logits, sorted_indices = torch.sort(scores, descending=False)
        cumulative_probs = sorted_logits.softmax(dim=-1).cumsum(dim=-1)
    
        # Remove tokens with cumulative top_p above the threshold (token with 0 are kept)
        sorted_indices_to_remove = cumulative_probs <= (1 - self.top_p)
        # Keep at least min_tokens_to_keep
        sorted_indices_to_remove[..., -self.min_tokens_to_keep :] = 0
    
        # scatter sorted tensors to original indexing
        indices_to_remove = sorted_indices_to_remove.scatter(1, sorted_indices, sorted_indices_to_remove)
        scores = scores.masked_fill(indices_to_remove, self.filter_value)
        return scores
    

    can you please help me?

  2. Assistant

    The error you are encountering with the scatter operation indicates that there's a mismatch between the dimensions of the indices and the tensor you're trying to modify. This often happens due to misalignment in shapes expected by the operation. Let's go through your code and the potential issue in detail.

    1. Understanding the Error: The error message suggests an issue with the dimensions being incompatible for the scatter operation. The scatter method expects the index tensor's size in the scattering dimension to match the source tensor's dimension size. The scatter call in your snippet is:

      indices_to_remove = sorted_indices_to_remove.scatter(1, sorted_indices, sorted_indices_to_remove)
      

      The purpose here is to rearrange sorted_indices_to_remove to its original order using sorted_indices as a guide. However, the error suggests a mismatch in expected dimensions.

    2. Debugging the Issue:

      • First, check the shape of sorted_indices_to_remove and sorted_indices to ensure they align with what scatter expects.
      • The scatter function signature in this context would normally look something like scatter(dim, index, src), where src has to broadcastable to the shape that the index tensor dictates.
    3. Potential Solution:

      The issue seems to stem from how sorted_indices_to_remove is used in both the index and src parameters, which might not be aligning with how scatter expects these tensors.

      The correct approach would require ensuring that the src argument (in this case, sorted_indices_to_remove) matches the shape expected by the index tensor's shape. However, you're using the same tensor for both index and src, which might not be the root issue.

      Here's an adjustment to focus on:

      Since the aim is to create a mask (indices_to_remove) that can be applied to the original scores tensor, a crucial step is ensuring that the scatter operation translates sorted_indices_to_remove back correctly. Given the error, and assuming the remaining parts of your code are correct, you might want to review if the manipulation done prior to scatter is correct and if sorted_indices is appropriately prepared for this operation.

      Here's a conceptual fix, focusing on ensuring the dimensions match and using correct scatter arguments:

      # Ensure dimensions alignment for scatter
      indices_to_remove = torch.zeros_like(scores, dtype=torch.bool)
      indices_to_remove.scatter_(1, sorted_indices, sorted_indices_to_remove)
      

      This adjustment assumes scores have the appropriate shape and uses scatter_ for an in-place operation, ensuring that indices_to_remove is initially a tensor of zeros with the same shape as scores. It also explicitly sets the data type to match boolean conditions expected by masked_fill.

    4. General Advice:

      • Ensure the shapes of all tensors involved in operations like scatter are exactly as expected. Print and verify these shapes as a debugging step.
      • Understand the dimensions and how scatter operations work, especially in terms of broadcasting rules and indexing.

    If after applying these suggestions you still face issues, consider isolating the operation and testing it with minimal data to understand the behavior more intuitively. This might help pinpoint the exact mismatch in dimensions or logic.

  3. User

    I double checked the dimensions and the dimensions are the same. Is there any other possible reason for this?

  4. Assistant

    Given that you've checked the dimensions and they align, but you're still encountering the error, let's reconsider the situation with more detail. The operation sequence leading to the scatter call seems theoretically sound, so the issue might lie in a subtle misuse or misunderstanding of the scatter operation or an oversight in tensor shapes at runtime. Here are a few additional insights and checks that you might find useful:

    Revisiting scatter Usage

    The scatter operation is used to rearrange elements of the source tensor into a new tensor according to indices provided. The signature of the scatter operation you are trying to use should indeed be like:

    scatter(dim, index, src)
    

    where dim is the dimension along which to index, index is the tensor of indices, and src is the tensor of values to scatter or a single value to scatter.

    Your usage intends to unsort the sorted_indices_to_remove boolean mask back to the original ordering of scores. Assuming sorted_indices correctly maps sorted positions back to original positions, the operation should theoretically reorder sorted_indices_to_remove correctly.

    Possible Issue with Scatter Arguments

    Given your confirmation that tensor dimensions align, let's re-examine the scatter parameters. The error message suggests a problem with indexing bounds, which might hint at a misuse in how indices are mapped:

    • Ensure sorted_indices has the same shape as sorted_indices_to_remove and contains valid indices for scattering. Each element in sorted_indices should be an index in the dimension you are scattering into (dim=1), and none of these indices should be out of bounds for the dimension size of scores.

    In-Depth Investigation

    1. Verify Index Values: Ensure all values in sorted_indices are within the bounds of the corresponding dimensions of your scores tensor. An out-of-bounds index value in sorted_indices could cause this assertion error.

    2. Dimensionality Check:

      • Confirm once again the exact shapes of scores, sorted_indices, and sorted_indices_to_remove right before the failing operation. This helps in identifying any unexpected shape change.
    3. Alternative Approach: If the direct troubleshooting doesn't resolve the problem, consider working around the issue by trying a different method to apply the mask:

      • Instead of using scatter to revert the order of sorted_indices_to_remove, another strategy could involve directly using sorted_indices to gather the necessary elements from scores and then apply the mask.

      Here’s a conceptual outline that avoids using scatter for reordering:

      # Assuming 'scores' and 'sorted_indices' are prepared as per your code
      # And 'sorted_indices_to_remove' is correctly computed
      
      # Create a new tensor filled with 'filter_value' for 'scores' that will be dropped
      updated_scores = scores.clone().fill_(self.filter_value)
      
      # Use a boolean mask to select elements to retain, and directly update 'updated_scores'
      retained_mask = ~sorted_indices_to_remove
      updated_scores.scatter_(1, sorted_indices, scores[torch.arange(scores.size(0))[:, None], sorted_indices][retained_mask])
      

      This approach circumvents the direct inversion of the mask and selectively updates the scores with retained values. Note that this pseudo-code might require adaptation based on your exact use case and the structure of your tensors.

    Final Note

    If the problem persists despite all the checks and alternative approaches, it might be helpful to isolate the snippet causing the issue in a minimal, standalone script. This could help in identifying whether the problem lies in the environment, version-specific issues with PyTorch or the Transformers library, or other overlooked aspects in the larger context of the code.

    Lastly, considering the complexities involved, it might also be beneficial to revisit the documentation for the PyTorch version you're using, ensuring there haven't been any changes or deprecations in the scatter method or related operations that could affect your implementation.