I want to do retrieval augmented generation with my LLM. can you suggest how can I put…
- User
I want to do retrieval augmented generation with my LLM. can you suggest how can I put the data and retrieve it from the vector database? I tried just making chunks of 512 tokens and retrieve them based on cosine similarity to the prompt, but the result is trash - the chunks don't make sense sometimes and they make the model confused
Assistant
Here are a few suggestions for doing retrieval augmented generation with an LLM in a more effective way:
-
Use larger context chunks rather than 512 tokens. Try chunks of 1,000-2,000 tokens to help ensure the retrieved text is coherent.
-
Preprocess the context chunks to remove any private/sensitive information before storing. This helps avoid confusing or worrying the model.
-
When retrieving, consider the top 2-5 most similar chunks rather than just the top 1. Then stitch together the most coherent parts from multiple chunks.
-
Weight the cosine similarity by things like chunk length - longer chunks that still match reasonably well may be better than very short exact matches.
-
Filter retrieved chunks against the current conversation context to avoid introducing irrelevant information.
-
Gradually fade in the retrieved context, don't just replace the model's own generation. Blend the retrieved and generated text.
-
Continue letting the model generate on its own after a certain number of retrieved chunks, to keep the conversation natural.
-
Store context chunks organized by topic/domain to only retrieve relevant chunks, not random out-of-domain text.
-
Continuously evaluate the quality of retrieved vs generated responses to refine your retrieval approach over time.
The key is finding the right balance between retrieved context and model generation to augment without overwhelming the model. It takes some trial and error.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.