Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0613· OpenAIAi2 WildChat-4.8M (AI2)

Concatenate the chunked embeddings to get the final corpusembeddings corpusembeddings =…

  1. User

    Concatenate the chunked embeddings to get the final corpus_embeddings

    corpus_embeddings = np.concatenate(chunked_embeddings, axis=0)

    cunked_embeddings is a super long list of np arrays. should i batch it

  2. Assistant

    It depends on the size of the list and the available memory. If the list of embeddings is too large to fit in memory, you can consider batching it to concatenate smaller batches at a time.

    Here's an example of how you can batch and concatenate the embeddings using a for loop:

    batch_size = 1000  # Choose an appropriate batch size
    corpus_embeddings = []
    
    for i in range(0, len(chunked_embeddings), batch_size):
        batch_embeddings = np.concatenate(chunked_embeddings[i:i+batch_size], axis=0)
        corpus_embeddings.append(batch_embeddings)
    
    corpus_embeddings = np.concatenate(corpus_embeddings, axis=0)
    

    By dividing the chunked embeddings into smaller batches, you can concatenate them gradually and reduce memory usage. Adjust the batch_size based on your available memory and computational resources.