Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-mini-2024-07-18· OpenAIAi2 WildChat-4.8M (AI2)

outputids = [30522] pastkeyvalues = None while outputids[-1] != 102 and len(outputids) <…

  1. User

    output_ids = [30522] past_key_values = None while output_ids[-1] != 102 and len(output_ids) < 50: logits = model.text_decoder.bert( input_ids=torch.tensor([[output_ids[-1]]]), # torch.Size([1, 1]) attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int), # torch.Size([1, i]) encoder_hidden_states=text_outputs[0], # torch.Size([1, 'N_TOK_IN', 768]) encoder_attention_mask=torch.ones((1, text_outputs[0].size(1)), dtype=torch.int), # torch.Size([1, 'N_TOK_IN']) past_key_values=past_key_values, return_dict=True, is_decoder=True, ) # torch.Size([1, 1, 768])

    past_key_values = None  # tuple(tuple(tensor)); we keep it empty for now
    
    output = model.text_decoder.cls(logits.last_hidden_state)  # torch.Size([1, 1, 30524])
    
    new_id = int(output.argmax(-1))  # torch.Size([1, 1]) -> int
    
    output_ids.append(new_id)
    

    answer = processor.decode(output_ids, skip_special_tokens=True) print(answer)

    The first iteration works well, but the second one produces:

    RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [12, 2] but got: [12, 1].

  2. Assistant

    The error message you're encountering indicates that there is a mismatch in the expected dimension sizes when processing tensors for your model. Specifically, the issue arises when the model expects tensors of a certain shape based on the previous iteration's outputs, but it receives a different shape instead.

    From your code snippet, the most likely cause of this issue is related to how you're managing the input tensors on each iteration, particularly when it comes to the attention masks and the hidden states.

    Diagnosis of the Issue

    In each iteration, the size of the attention_mask and the encoder_hidden_states should be consistent with the past context or sequence length. In your loop:

    • You are constructing the input tensor for input_ids as a single-element array, which is fine for the first iteration, but subsequent iterations might break the expected shape as the sequence length increases.
    • It's likely that the past_key_values should not be reset to None on every iteration if you are attempting to utilize past context for your decoder.

    Suggested Fixes

    1. Update past_key_values Appropriately: Instead of resetting past_key_values to None in each loop iteration, you should use the output from the model for this purpose. The past_key_values should maintain the hidden states of previous iterations, enabling the model to retrieve context from earlier tokens.

    2. Manage Attention Masks Properly: Ensure that the attention_mask correctly reflects the sequence length of output_ids in each iteration. As output_ids grows, the size of the attention_mask should also be updated accordingly.

    Here's an updated version of your loop with these considerations:

    output_ids = [30522]
    past_key_values = None
    
    while output_ids[-1] != 102 and len(output_ids) < 50:
        input_tensor = torch.tensor([[output_ids[-1]]])  # Current token
        attention_mask = torch.ones((1, len(output_ids)), dtype=torch.int)  # Include all input
    
        logits = model.text_decoder.bert(
            input_ids=input_tensor,  # Current input token
            attention_mask=attention_mask,  # Updated attention mask for current tokens
            encoder_hidden_states=text_outputs[0],  # Encoder outputs
            encoder_attention_mask=torch.ones((1, text_outputs[0].size(1)), dtype=torch.int),  # Encoder mask
            past_key_values=past_key_values,  # Use past key values for context
            return_dict=True,
            is_decoder=True,
        )
        
        # Use the output to update past_key_values for next iteration
        past_key_values = logits.past_key_values  # Save past_key_values for next iteration
        
        output = model.text_decoder.cls(logits.last_hidden_state)  # Final output layer over last hidden state
        
        new_id = int(output.argmax(-1))  # Get the new generated token
        
        output_ids.append(new_id)  # Append the new token to output_ids
    
    answer = processor.decode(output_ids, skip_special_tokens=True)
    print(answer)
    

    Summary

    The key changes are:

    • Maintaining the past_key_values based on model outputs rather than resetting it to None.
    • Ensuring that the attention_mask correctly reflects the full sequence every iteration.

    By addressing these issues, you should be able to avoid the runtime error and continue generating tokens successfully.

  3. User

    how many billion barameters does chat gpt 4 have? also, how many billion parameters does a human have?

  4. Assistant

    As of my last knowledge update in October 2023, OpenAI has not disclosed the exact number of parameters for GPT-4, the model behind ChatGPT-4. However, estimates suggest that GPT-4 has hundreds of billions of parameters. Reports have varied, with some sources estimating it could be in the range of 100 to 200 billion parameters or more.

    When it comes to the human brain, the number of neurons is often cited as a comparison point. The human brain is estimated to have approximately 86 billion neurons, each of which can have thousands of synapses (the connections between neurons). If you were to consider synapses or connections as parameters in a very loose analogy, estimates suggest the number of synapses in the brain could be in the range of 100 trillion to 500 trillion.

    It's important to note that comparing the number of parameters in an AI model and the complexity of the human brain is not straightforward, as they operate on fundamentally different principles and architectures. The parameters in a neural network do not directly correspond to the functions of neurons and synapses in the human brain.

  5. User

    we have the following code snippet

    image_inputs = inputs['pixel_values']  # torch.Size([1, 3, 384, 384])
    question_inputs = inputs['input_ids']  # torch.Size([1, N_TOK_IN])
    attention_masks = inputs['attention_mask']  # torch.Size([1, N_TOK_IN])
    
    # =============================================================
    
    # Forward pass through the vision model
    vision_outputs = model.vision_model(pixel_values=image_inputs)
    #print(type(vision_outputs))  # BaseModelOutputWithPooling
    #print(vision_outputs.last_hidden_state.shape)  # torch.Size([1, 577, 768])
    #print(vision_outputs.pooler_output.shape)  # torch.Size([1, 768])
    
    vision_embedding = vision_outputs.last_hidden_state
    
    # =============================================================
    
    # Forward pass through the text encoder
    enc_att_mask = torch.ones((1, vision_embedding.size(1)))
    
    text_outputs = model.text_encoder(
        input_ids=question_inputs,  # torch.Size([1, N_TOK_IN])
        attention_mask=attention_masks,  # torch.Size([1, N_TOK_IN])
        encoder_hidden_states=vision_embedding,  # torch.Size([1, 577, 768])
        encoder_attention_mask=enc_att_mask,  # torch.Size([1, 577])
        return_dict=False,
        is_decoder=False,
    )
    #print(type(text_outputs))  # tuple, len=2
    #print(type(text_outputs[0]))  # torch.Tensor
    #print(type(text_outputs[1]))  # NoneType
    #print(text_outputs[0].shape)  # torch.Size([1, N_TOK_IN, 768])
    
    # You can get the features for the question as well
    text_embedding = text_outputs[0]
    
    
    # Generate the output token IDs one by one, starting with 30522, until you reach 102 or generate max_new_tokens
    output_ids = [30522]
    past_key_values = None
    while output_ids[-1] != 102 and len(output_ids) < max_new_tokens:
        logits = model.text_decoder.bert(
            input_ids=torch.tensor([[output_ids[-1]]]),  # torch.Size([1, 1])
            attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int),  # torch.Size([1, i])
            encoder_hidden_states=text_embedding,  # torch.Size([1, N_TOK_IN, 768])
            encoder_attention_mask=torch.ones((1, text_embedding.size(1)), dtype=torch.int),  # torch.Size([1, N_TOK_IN])
            past_key_values=past_key_values,
            return_dict=True,
            is_decoder=True,
        )
        #print(type(logits))  # BaseModelOutputWithPoolingAndCrossAttentions
        #print(logits.last_hidden_state.shape)  # torch.Size([1, 1, 768])
        #print(type(logits.past_key_values))  # tuple, len=12
        #print(type(logits.past_key_values[9]))  # tuple, len=2
        #print(logits.past_key_values[0][0].shape)  # torch.Size([1, 12, i, 64])
    

    How can I name the "logits" variable? I want a more suitable name?

  6. Assistant

    Choosing a more descriptive name for the logits variable in your code can improve code readability and help convey the purpose of this variable more clearly. Since logits typically refers to the raw output scores (unnormalized predictions) from the model right before applying an activation function (like softmax) for converting them into probabilities, it may be beneficial to choose a name that reflects this role.

    Here are a few suggestions for renaming the logits variable based on its function within your code:

    1. decoder_outputs: This name indicates that these are the outputs from the decoder model, which is generating token predictions.

    2. output_scores: This highlights that the variable contains score values from the model indicating the predicted likelihood for each token in the vocabulary.

    3. token_logits: This suggests that the variable represents the logits specifically associated with the output tokens being generated.

    4. prediction_logits: This indicates that these logits are for predicting the next token in the sequence.

    5. raw_output_scores: This emphasizes that these are the raw (unnormalized) scores given by the model.

    6. sequence_logits: Refers specifically to the logits used for generating the next sequence of tokens.

    7. next_token_logits: This name reflects the purpose of these logits as they relate to predicting the next token in the sequence.

    Here’s how you could implement one of these names in your code:

    # Update from logits to decoder_outputs
    decoder_outputs = model.text_decoder.bert(
        input_ids=torch.tensor([[output_ids[-1]]]),  # torch.Size([1, 1])
        attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int),  # torch.Size([1, i])
        encoder_hidden_states=text_embedding,  # torch.Size([1, N_TOK_IN, 768])
        encoder_attention_mask=torch.ones((1, text_embedding.size(1)), dtype=torch.int),  # torch.Size([1, N_TOK_IN])
        past_key_values=past_key_values,
        return_dict=True,
        is_decoder=True,
    )
    
    # You may also want to rename your use of logits accordingly
    # print(type(decoder_outputs))  # BaseModelOutputWithPoolingAndCrossAttentions
    # print(decoder_outputs.last_hidden_state.shape)  # torch.Size([1, 1, 768])
    

    Choose a name that best fits your understanding of the variable and the context of its use within your application.

  7. User

    thank you

  8. Assistant

    You're welcome! If you have any more questions or need further assistance, feel free to ask. Happy coding!

  9. User

    now here is the following part of the code:

    # Generate the output token IDs one by one, starting with 30522, until you reach 102 or generate max_new_tokens
    output_ids = [30522]
    past_key_values = None
    while output_ids[-1] != 102 and len(output_ids) < max_new_tokens:
        decoder_outputs = model.text_decoder.bert(
            input_ids=torch.tensor([[output_ids[-1]]]),  # torch.Size([1, 1])
            attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int),  # torch.Size([1, i])
            encoder_hidden_states=text_embedding,  # torch.Size([1, N_TOK_IN, 768])
            encoder_attention_mask=torch.ones((1, text_embedding.size(1)), dtype=torch.int),  # torch.Size([1, N_TOK_IN])
            past_key_values=past_key_values,
            return_dict=True,
            is_decoder=True,
        )
        #print(type(decoder_outputs))  # BaseModelOutputWithPoolingAndCrossAttentions
        #print(decoder_outputs.last_hidden_state.shape)  # torch.Size([1, 1, 768])
        #print(type(decoder_outputs.past_key_values))  # tuple, len=12
        #print(type(decoder_outputs.past_key_values[9]))  # tuple, len=2
        #print(decoder_outputs.past_key_values[0][0].shape)  # torch.Size([1, 12, i, 64])
        
        past_key_values = decoder_outputs.past_key_values
        
        output = model.text_decoder.cls(decoder_outputs.last_hidden_state)  # torch.Size([1, 1, 30524])
        
        new_id = int(output.argmax(-1))  # torch.Size([1, 1]) -> int
        output_ids.append(new_id)
    

    Which of "decoder_outputs.last_hidden_state" or "output" are the logits? Also, can I give a more suitable name to any of the variables used there?

  10. Assistant

    In the context of your code snippet, decoder_outputs.last_hidden_state contains the hidden states produced by the last layer of the decoder for the current input. This is often utilized to extract features or representations, but it is not the logits themselves.

    On the other hand, output is created by passing decoder_outputs.last_hidden_state through the cls layer of your text decoder. Therefore, output represents the logits, as it consists of the raw scores (unnormalized predictions) from which you derive the predicted token IDs.

    Variable Naming Suggestions

    You could improve the clarity of your variable names as follows:

    1. decoder_outputs

      • Current Name: decoder_outputs
      • Suggested Name: decoder_response or decoder_model_outputs. This name indicates that it contains the output response from the decoder model, encapsulating all necessary data.
    2. output

      • Current Name: output
      • Suggested Name: logits or predicted_logits. This name reflects that these are the raw scores that are being used to generate the token predictions.
    3. new_id

      • Current Name: new_id
      • Suggested Name: predicted_token_id or next_token_id. This makes it explicit that this variable holds the ID of the token predicted by the model.

    Here’s how the code would look with these suggested name changes:

    # Generate the output token IDs one by one, starting with 30522, until you reach 102 or generate max_new_tokens
    output_ids = [30522]
    past_key_values = None
    while output_ids[-1] != 102 and len(output_ids) < max_new_tokens:
        decoder_response = model.text_decoder.bert(
            input_ids=torch.tensor([[output_ids[-1]]]),  # torch.Size([1, 1])
            attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int),  # torch.Size([1, i])
            encoder_hidden_states=text_embedding,  # torch.Size([1, N_TOK_IN, 768])
            encoder_attention_mask=torch.ones((1, text_embedding.size(1)), dtype=torch.int),  # torch.Size([1, N_TOK_IN])
            past_key_values=past_key_values,
            return_dict=True,
            is_decoder=True,
        )
    
        past_key_values = decoder_response.past_key_values
    
        logits = model.text_decoder.cls(decoder_response.last_hidden_state)  # torch.Size([1, 1, 30524])
    
        predicted_token_id = int(logits.argmax(-1))  # torch.Size([1, 1]) -> int
        output_ids.append(predicted_token_id)
    

    Summary

    • decoder_response: Indicates the output from the decoder.
    • logits: Represents the scores or predictions used to determine the next token.
    • predicted_token_id: Clearly states that this is the ID of the token predicted by the model.

    These changes improve the readability of the code by making variable purposes clearer.

  11. User

    can i name decoder_outputs.last_hidden_state as "decoder_embedding" or something similar?

  12. Assistant

    Yes, naming decoder_outputs.last_hidden_state as decoder_embedding is a suitable choice. The term "embedding" in this context implies that the output represents a learned representation of the input data—specifically, the sequence of tokens processed by the decoder at that moment in the generation process.

    Using the term "embedding" can be helpful in emphasizing that you are working with a dense, continuous representation of the input rather than raw scores or logits. However, it's also important to note that "embedding" might imply some sort of lower-dimensional representation typically associated with the initial layers in a model, whereas the outputs from the last layer are the final hidden states after passing through all the transformations.

    If you decide to use decoder_embedding, it might be useful to clarify its context in comments or documentation to ensure that others (or future you) understand its origin as the last hidden states of the decoder.

    Here’s how the code would look with that naming change:

    # Generate the output token IDs one by one, starting with 30522, until you reach 102 or generate max_new_tokens
    output_ids = [30522]
    past_key_values = None
    while output_ids[-1] != 102 and len(output_ids) < max_new_tokens:
        decoder_response = model.text_decoder.bert(
            input_ids=torch.tensor([[output_ids[-1]]]),  # torch.Size([1, 1])
            attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int),  # torch.Size([1, i])
            encoder_hidden_states=text_embedding,  # torch.Size([1, N_TOK_IN, 768])
            encoder_attention_mask=torch.ones((1, text_embedding.size(1)), dtype=torch.int),  # torch.Size([1, N_TOK_IN])
            past_key_values=past_key_values,
            return_dict=True,
            is_decoder=True,
        )
    
        past_key_values = decoder_response.past_key_values
    
        decoder_embedding = decoder_response.last_hidden_state  # torch.Size([1, 1, 768])
        logits = model.text_decoder.cls(decoder_embedding)  # torch.Size([1, 1, 30524])
    
        predicted_token_id = int(logits.argmax(-1))  # torch.Size([1, 1]) -> int
        output_ids.append(predicted_token_id)
    

    Summary

    • decoder_embedding: Using this name is appropriate as it reflects the embeddings obtained from the last hidden states of the decoder.
    • Ensure to maintain clear comments or documentation for clarity if others are reading the code, as it helps to contextualize "embedding" in this instance.

    Feel free to rename it as you see fit to improve clarity!