outputids = [30522] pastkeyvalues = None while outputids[-1] != 102 and len(outputids) <…
- User
output_ids = [30522] past_key_values = None while output_ids[-1] != 102 and len(output_ids) < 50: logits = model.text_decoder.bert( input_ids=torch.tensor([[output_ids[-1]]]), # torch.Size([1, 1]) attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int), # torch.Size([1, i]) encoder_hidden_states=text_outputs[0], # torch.Size([1, 'N_TOK_IN', 768]) encoder_attention_mask=torch.ones((1, text_outputs[0].size(1)), dtype=torch.int), # torch.Size([1, 'N_TOK_IN']) past_key_values=past_key_values, return_dict=True, is_decoder=True, ) # torch.Size([1, 1, 768])
past_key_values = None # tuple(tuple(tensor)); we keep it empty for now output = model.text_decoder.cls(logits.last_hidden_state) # torch.Size([1, 1, 30524]) new_id = int(output.argmax(-1)) # torch.Size([1, 1]) -> int output_ids.append(new_id)answer = processor.decode(output_ids, skip_special_tokens=True) print(answer)
The first iteration works well, but the second one produces:
RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [12, 2] but got: [12, 1].
- Assistant
The error message you're encountering indicates that there is a mismatch in the expected dimension sizes when processing tensors for your model. Specifically, the issue arises when the model expects tensors of a certain shape based on the previous iteration's outputs, but it receives a different shape instead.
From your code snippet, the most likely cause of this issue is related to how you're managing the input tensors on each iteration, particularly when it comes to the attention masks and the hidden states.
Diagnosis of the Issue
In each iteration, the size of the
attention_maskand theencoder_hidden_statesshould be consistent with the past context or sequence length. In your loop:- You are constructing the input tensor for
input_idsas a single-element array, which is fine for the first iteration, but subsequent iterations might break the expected shape as the sequence length increases. - It's likely that the
past_key_valuesshould not be reset toNoneon every iteration if you are attempting to utilize past context for your decoder.
Suggested Fixes
-
Update
past_key_valuesAppropriately: Instead of resettingpast_key_valuestoNonein each loop iteration, you should use the output from the model for this purpose. Thepast_key_valuesshould maintain the hidden states of previous iterations, enabling the model to retrieve context from earlier tokens. -
Manage Attention Masks Properly: Ensure that the
attention_maskcorrectly reflects the sequence length ofoutput_idsin each iteration. Asoutput_idsgrows, the size of theattention_maskshould also be updated accordingly.
Here's an updated version of your loop with these considerations:
output_ids = [30522] past_key_values = None while output_ids[-1] != 102 and len(output_ids) < 50: input_tensor = torch.tensor([[output_ids[-1]]]) # Current token attention_mask = torch.ones((1, len(output_ids)), dtype=torch.int) # Include all input logits = model.text_decoder.bert( input_ids=input_tensor, # Current input token attention_mask=attention_mask, # Updated attention mask for current tokens encoder_hidden_states=text_outputs[0], # Encoder outputs encoder_attention_mask=torch.ones((1, text_outputs[0].size(1)), dtype=torch.int), # Encoder mask past_key_values=past_key_values, # Use past key values for context return_dict=True, is_decoder=True, ) # Use the output to update past_key_values for next iteration past_key_values = logits.past_key_values # Save past_key_values for next iteration output = model.text_decoder.cls(logits.last_hidden_state) # Final output layer over last hidden state new_id = int(output.argmax(-1)) # Get the new generated token output_ids.append(new_id) # Append the new token to output_ids answer = processor.decode(output_ids, skip_special_tokens=True) print(answer)Summary
The key changes are:
- Maintaining the
past_key_valuesbased on model outputs rather than resetting it toNone. - Ensuring that the
attention_maskcorrectly reflects the full sequence every iteration.
By addressing these issues, you should be able to avoid the runtime error and continue generating tokens successfully.
- You are constructing the input tensor for
- User
how many billion barameters does chat gpt 4 have? also, how many billion parameters does a human have?
- Assistant
As of my last knowledge update in October 2023, OpenAI has not disclosed the exact number of parameters for GPT-4, the model behind ChatGPT-4. However, estimates suggest that GPT-4 has hundreds of billions of parameters. Reports have varied, with some sources estimating it could be in the range of 100 to 200 billion parameters or more.
When it comes to the human brain, the number of neurons is often cited as a comparison point. The human brain is estimated to have approximately 86 billion neurons, each of which can have thousands of synapses (the connections between neurons). If you were to consider synapses or connections as parameters in a very loose analogy, estimates suggest the number of synapses in the brain could be in the range of 100 trillion to 500 trillion.
It's important to note that comparing the number of parameters in an AI model and the complexity of the human brain is not straightforward, as they operate on fundamentally different principles and architectures. The parameters in a neural network do not directly correspond to the functions of neurons and synapses in the human brain.
- User
we have the following code snippet
image_inputs = inputs['pixel_values'] # torch.Size([1, 3, 384, 384]) question_inputs = inputs['input_ids'] # torch.Size([1, N_TOK_IN]) attention_masks = inputs['attention_mask'] # torch.Size([1, N_TOK_IN]) # ============================================================= # Forward pass through the vision model vision_outputs = model.vision_model(pixel_values=image_inputs) #print(type(vision_outputs)) # BaseModelOutputWithPooling #print(vision_outputs.last_hidden_state.shape) # torch.Size([1, 577, 768]) #print(vision_outputs.pooler_output.shape) # torch.Size([1, 768]) vision_embedding = vision_outputs.last_hidden_state # ============================================================= # Forward pass through the text encoder enc_att_mask = torch.ones((1, vision_embedding.size(1))) text_outputs = model.text_encoder( input_ids=question_inputs, # torch.Size([1, N_TOK_IN]) attention_mask=attention_masks, # torch.Size([1, N_TOK_IN]) encoder_hidden_states=vision_embedding, # torch.Size([1, 577, 768]) encoder_attention_mask=enc_att_mask, # torch.Size([1, 577]) return_dict=False, is_decoder=False, ) #print(type(text_outputs)) # tuple, len=2 #print(type(text_outputs[0])) # torch.Tensor #print(type(text_outputs[1])) # NoneType #print(text_outputs[0].shape) # torch.Size([1, N_TOK_IN, 768]) # You can get the features for the question as well text_embedding = text_outputs[0] # Generate the output token IDs one by one, starting with 30522, until you reach 102 or generate max_new_tokens output_ids = [30522] past_key_values = None while output_ids[-1] != 102 and len(output_ids) < max_new_tokens: logits = model.text_decoder.bert( input_ids=torch.tensor([[output_ids[-1]]]), # torch.Size([1, 1]) attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int), # torch.Size([1, i]) encoder_hidden_states=text_embedding, # torch.Size([1, N_TOK_IN, 768]) encoder_attention_mask=torch.ones((1, text_embedding.size(1)), dtype=torch.int), # torch.Size([1, N_TOK_IN]) past_key_values=past_key_values, return_dict=True, is_decoder=True, ) #print(type(logits)) # BaseModelOutputWithPoolingAndCrossAttentions #print(logits.last_hidden_state.shape) # torch.Size([1, 1, 768]) #print(type(logits.past_key_values)) # tuple, len=12 #print(type(logits.past_key_values[9])) # tuple, len=2 #print(logits.past_key_values[0][0].shape) # torch.Size([1, 12, i, 64])How can I name the "logits" variable? I want a more suitable name?
- Assistant
Choosing a more descriptive name for the
logitsvariable in your code can improve code readability and help convey the purpose of this variable more clearly. Sincelogitstypically refers to the raw output scores (unnormalized predictions) from the model right before applying an activation function (like softmax) for converting them into probabilities, it may be beneficial to choose a name that reflects this role.Here are a few suggestions for renaming the
logitsvariable based on its function within your code:-
decoder_outputs: This name indicates that these are the outputs from the decoder model, which is generating token predictions. -
output_scores: This highlights that the variable contains score values from the model indicating the predicted likelihood for each token in the vocabulary. -
token_logits: This suggests that the variable represents the logits specifically associated with the output tokens being generated. -
prediction_logits: This indicates that these logits are for predicting the next token in the sequence. -
raw_output_scores: This emphasizes that these are the raw (unnormalized) scores given by the model. -
sequence_logits: Refers specifically to the logits used for generating the next sequence of tokens. -
next_token_logits: This name reflects the purpose of these logits as they relate to predicting the next token in the sequence.
Here’s how you could implement one of these names in your code:
# Update from logits to decoder_outputs decoder_outputs = model.text_decoder.bert( input_ids=torch.tensor([[output_ids[-1]]]), # torch.Size([1, 1]) attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int), # torch.Size([1, i]) encoder_hidden_states=text_embedding, # torch.Size([1, N_TOK_IN, 768]) encoder_attention_mask=torch.ones((1, text_embedding.size(1)), dtype=torch.int), # torch.Size([1, N_TOK_IN]) past_key_values=past_key_values, return_dict=True, is_decoder=True, ) # You may also want to rename your use of logits accordingly # print(type(decoder_outputs)) # BaseModelOutputWithPoolingAndCrossAttentions # print(decoder_outputs.last_hidden_state.shape) # torch.Size([1, 1, 768])Choose a name that best fits your understanding of the variable and the context of its use within your application.
-
- User
thank you
- Assistant
You're welcome! If you have any more questions or need further assistance, feel free to ask. Happy coding!
- User
now here is the following part of the code:
# Generate the output token IDs one by one, starting with 30522, until you reach 102 or generate max_new_tokens output_ids = [30522] past_key_values = None while output_ids[-1] != 102 and len(output_ids) < max_new_tokens: decoder_outputs = model.text_decoder.bert( input_ids=torch.tensor([[output_ids[-1]]]), # torch.Size([1, 1]) attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int), # torch.Size([1, i]) encoder_hidden_states=text_embedding, # torch.Size([1, N_TOK_IN, 768]) encoder_attention_mask=torch.ones((1, text_embedding.size(1)), dtype=torch.int), # torch.Size([1, N_TOK_IN]) past_key_values=past_key_values, return_dict=True, is_decoder=True, ) #print(type(decoder_outputs)) # BaseModelOutputWithPoolingAndCrossAttentions #print(decoder_outputs.last_hidden_state.shape) # torch.Size([1, 1, 768]) #print(type(decoder_outputs.past_key_values)) # tuple, len=12 #print(type(decoder_outputs.past_key_values[9])) # tuple, len=2 #print(decoder_outputs.past_key_values[0][0].shape) # torch.Size([1, 12, i, 64]) past_key_values = decoder_outputs.past_key_values output = model.text_decoder.cls(decoder_outputs.last_hidden_state) # torch.Size([1, 1, 30524]) new_id = int(output.argmax(-1)) # torch.Size([1, 1]) -> int output_ids.append(new_id)Which of "decoder_outputs.last_hidden_state" or "output" are the logits? Also, can I give a more suitable name to any of the variables used there?
- Assistant
In the context of your code snippet,
decoder_outputs.last_hidden_statecontains the hidden states produced by the last layer of the decoder for the current input. This is often utilized to extract features or representations, but it is not the logits themselves.On the other hand,
outputis created by passingdecoder_outputs.last_hidden_statethrough theclslayer of your text decoder. Therefore,outputrepresents the logits, as it consists of the raw scores (unnormalized predictions) from which you derive the predicted token IDs.Variable Naming Suggestions
You could improve the clarity of your variable names as follows:
-
decoder_outputs- Current Name:
decoder_outputs - Suggested Name:
decoder_responseordecoder_model_outputs. This name indicates that it contains the output response from the decoder model, encapsulating all necessary data.
- Current Name:
-
output- Current Name:
output - Suggested Name:
logitsorpredicted_logits. This name reflects that these are the raw scores that are being used to generate the token predictions.
- Current Name:
-
new_id- Current Name:
new_id - Suggested Name:
predicted_token_idornext_token_id. This makes it explicit that this variable holds the ID of the token predicted by the model.
- Current Name:
Here’s how the code would look with these suggested name changes:
# Generate the output token IDs one by one, starting with 30522, until you reach 102 or generate max_new_tokens output_ids = [30522] past_key_values = None while output_ids[-1] != 102 and len(output_ids) < max_new_tokens: decoder_response = model.text_decoder.bert( input_ids=torch.tensor([[output_ids[-1]]]), # torch.Size([1, 1]) attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int), # torch.Size([1, i]) encoder_hidden_states=text_embedding, # torch.Size([1, N_TOK_IN, 768]) encoder_attention_mask=torch.ones((1, text_embedding.size(1)), dtype=torch.int), # torch.Size([1, N_TOK_IN]) past_key_values=past_key_values, return_dict=True, is_decoder=True, ) past_key_values = decoder_response.past_key_values logits = model.text_decoder.cls(decoder_response.last_hidden_state) # torch.Size([1, 1, 30524]) predicted_token_id = int(logits.argmax(-1)) # torch.Size([1, 1]) -> int output_ids.append(predicted_token_id)Summary
decoder_response: Indicates the output from the decoder.logits: Represents the scores or predictions used to determine the next token.predicted_token_id: Clearly states that this is the ID of the token predicted by the model.
These changes improve the readability of the code by making variable purposes clearer.
-
- User
can i name decoder_outputs.last_hidden_state as "decoder_embedding" or something similar?
- Assistant
Yes, naming
decoder_outputs.last_hidden_stateasdecoder_embeddingis a suitable choice. The term "embedding" in this context implies that the output represents a learned representation of the input data—specifically, the sequence of tokens processed by the decoder at that moment in the generation process.Using the term "embedding" can be helpful in emphasizing that you are working with a dense, continuous representation of the input rather than raw scores or logits. However, it's also important to note that "embedding" might imply some sort of lower-dimensional representation typically associated with the initial layers in a model, whereas the outputs from the last layer are the final hidden states after passing through all the transformations.
If you decide to use
decoder_embedding, it might be useful to clarify its context in comments or documentation to ensure that others (or future you) understand its origin as the last hidden states of the decoder.Here’s how the code would look with that naming change:
# Generate the output token IDs one by one, starting with 30522, until you reach 102 or generate max_new_tokens output_ids = [30522] past_key_values = None while output_ids[-1] != 102 and len(output_ids) < max_new_tokens: decoder_response = model.text_decoder.bert( input_ids=torch.tensor([[output_ids[-1]]]), # torch.Size([1, 1]) attention_mask=torch.ones((1, len(output_ids)), dtype=torch.int), # torch.Size([1, i]) encoder_hidden_states=text_embedding, # torch.Size([1, N_TOK_IN, 768]) encoder_attention_mask=torch.ones((1, text_embedding.size(1)), dtype=torch.int), # torch.Size([1, N_TOK_IN]) past_key_values=past_key_values, return_dict=True, is_decoder=True, ) past_key_values = decoder_response.past_key_values decoder_embedding = decoder_response.last_hidden_state # torch.Size([1, 1, 768]) logits = model.text_decoder.cls(decoder_embedding) # torch.Size([1, 1, 30524]) predicted_token_id = int(logits.argmax(-1)) # torch.Size([1, 1]) -> int output_ids.append(predicted_token_id)Summary
decoder_embedding: Using this name is appropriate as it reflects the embeddings obtained from the last hidden states of the decoder.- Ensure to maintain clear comments or documentation for clarity if others are reading the code, as it helps to contextualize "embedding" in this instance.
Feel free to rename it as you see fit to improve clarity!