gpt-3.5-turbo-0613· OpenAI
WildChat-4.8M (AI2)
Concatenate the chunked embeddings to get the final corpusembeddings corpusembeddings =…
- User
Concatenate the chunked embeddings to get the final corpus_embeddings
corpus_embeddings = np.concatenate(chunked_embeddings, axis=0)
cunked_embeddings is a super long list of np arrays. should i batch it
- Assistant
It depends on the size of the list and the available memory. If the list of embeddings is too large to fit in memory, you can consider batching it to concatenate smaller batches at a time.
Here's an example of how you can batch and concatenate the embeddings using a for loop:
batch_size = 1000 # Choose an appropriate batch size corpus_embeddings = [] for i in range(0, len(chunked_embeddings), batch_size): batch_embeddings = np.concatenate(chunked_embeddings[i:i+batch_size], axis=0) corpus_embeddings.append(batch_embeddings) corpus_embeddings = np.concatenate(corpus_embeddings, axis=0)By dividing the chunked embeddings into smaller batches, you can concatenate them gradually and reduce memory usage. Adjust the
batch_sizebased on your available memory and computational resources.