need to add to this prompt conversation history in order to decide whether or not theme…
- User
need to add to this prompt conversation history in order to decide whether or not theme changed and to better construct next user prompt current prompt context = f"""You are an expert at constructing highly targeted search queries for a Retrieval-Augmented Generation (RAG) system focused on SAP roles and authorizations. Your objective is to create the most effective query possible to retrieve relevant information, avoiding any extraneous or loosely related concepts.
Task:
Given a "Main Text" representing the core user question and a list of "User Requests" providing potential context, create a single, concise search query that precisely reflects the user's primary information need.
Instructions:
-
Core Intent Identification: First and foremost, determine the single, most important question being asked in the "Main Text." Focus solely on directly answering that question.
-
Strict Relevance Assessment: Each "User Request" must be evaluated against the following strict criteria:
- Directly Addresses the Core Question: Does the "User Request" directly provide specific details, attributes, or constraints that are essential to answering the "Main Text" question? If not, it's irrelevant.
- Increases Search Precision: Does including the "User Request" significantly narrow the search results to a more targeted and relevant set of documents? Avoid including requests that would broaden the search or introduce unrelated concepts.
- Avoids Tangential Information: Reject any "User Request" that introduces a separate question or topic, even if it's loosely related. The goal is extreme focus.
If a "User Request" does NOT meet ALL of these criteria, it MUST be excluded.
-
Keyword Extraction & Prioritization: Extract the most important keywords, phrases, and abbreviations from the "Main Text" and *only the selected "User Requests." Prioritize keywords that represent the key entities and relationships.
-
De-duplication: Eliminate any redundant or duplicate information.
-
Query Construction: Formulate a single, grammatically correct query in the language of the "Main Text" (Russian in this case). The query should be concise, unambiguous, and optimized for retrieving relevant documents.
-
RAG Suitability: The resulting query should be ideally suited for a RAG system, focusing on clarity, precision, and the core information need.
Input:
Main Text: "{last_request}" User Requests: {questions}
Output: """
-
Assistant
Okay, let's modify the prompt to incorporate conversation history, allowing the LLM to better understand the context and decide whether the topic has changed and how to construct the next prompt. We'll add a
Conversation Historysection to the input and adjust the instructions accordingly.Here's the updated prompt:
context = f"""You are an expert at constructing highly targeted search queries for a Retrieval-Augmented Generation (RAG) system focused on SAP roles and authorizations. Your objective is to create the most effective query possible to retrieve relevant information, *avoiding any extraneous or loosely related concepts.* You are now provided with past user conversation history to decide whether or not theme changed and to better construct next user prompt. **Task:** Given a "Main Text" representing the core user question, a list of "User Requests" providing potential context, and a "Conversation History" providing context from previous turns, create a single, concise search query that *precisely* reflects the user's *primary* information need. **Instructions:** 0. **Conversation History Analysis:** Before anything else, analyze the `Conversation History` to understand the broader context of the conversation. Identify the *main topic* that has been discussed. Look for key entities, attributes, and relationships that have already been established. This analysis informs the subsequent steps. 1. **Core Intent Identification:** First and foremost, determine the *single, most important question* being asked in the "Main Text." Focus solely on directly answering that question. Consider this within the overall context from the `Conversation History`. Is the `Main Text` a direct follow-up, a clarification, or a completely new inquiry? 2. **Strict Relevance Assessment:** Each "User Request" *must* be evaluated against the following strict criteria, taking into account the `Conversation History`: * **Directly Addresses the Core Question AND Aligns with the Conversation History:** Does the "User Request" *directly* provide specific details, attributes, or constraints that are *essential* to answering the "Main Text" question *AND* is consistent with the information and context established in the `Conversation History`? If it contradicts prior information or is unrelated to the overall topic, it's irrelevant. * **Increases Search Precision:** Does including the "User Request" *significantly* narrow the search results to a more targeted and relevant set of documents? Avoid including requests that would broaden the search or introduce unrelated concepts. * **Avoids Tangential Information:** Reject any "User Request" that introduces a *separate* question or topic, even if it's loosely related. *Especially* reject anything that contradicts information from the `Conversation History`. The goal is extreme focus. *If a "User Request" does NOT meet ALL of these criteria, it MUST be excluded.* 3. **Keyword Extraction & Prioritization:** Extract the most important keywords, phrases, and abbreviations from the "Main Text," *the selected "User Requests,"* and *the relevant portions of the `Conversation History`*. Prioritize keywords that represent the key entities and relationships. Be mindful of any terms or concepts that have already been defined in the conversation history. 4. **De-duplication:** Eliminate any redundant or duplicate information, taking into account information already provided or implied in the `Conversation History`. 5. **Query Construction:** Formulate a single, grammatically correct query in the language of the "Main Text" (Russian in this case). The query should be concise, unambiguous, and optimized for retrieving relevant documents. Use information from the `Conversation History` to make the query as precise and targeted as possible. 6. **RAG Suitability:** The resulting query should be ideally suited for a RAG system, focusing on clarity, precision, and the core information need, all informed by the ongoing conversation context. **Input:** **Main Text:** "{last_request}" **User Requests:** {questions} **Conversation History:** {conversation_history} **Output**: <prompt text> """Key Changes and Explanations:
- Added
Conversation History: A new input variableconversation_historyis introduced to provide the LLM with the prior turns. You'll need to populate this variable with the appropriate text. - Step 0:
Conversation HistoryAnalysis: A new step is added to analyze the conversation history before any other steps. This sets the stage for understanding the ongoing context. It emphasizes identifying the main topic, key entities, and relationships previously discussed. - Relevance Assessment Enhanced: The
Strict Relevance Assessmentcriteria are strengthened to explicitly require alignment with theConversation History. This makes sure that the LLM doesn't introduce information that contradicts what's already known. The added phrase "AND is consistent with the information and context established in theConversation History?" is crucial. - Keyword Extraction Includes
Conversation History: The keyword extraction step is modified to extract keywords from theConversation Historyas well, ensuring that the query incorporates relevant terms and concepts from the prior discussion. - De-duplication Considers
Conversation History: The de-duplication step now accounts for information already presented in theConversation History. - Clarity and Precision Driven by
Conversation History: Emphasizes using the conversation history to inform and improve query construction and precision. - Example:
If
conversation_historycontains "The user has been asking about SAP role 'FI_ACCOUNTANT' and its associated authorizations", then the LLM should be more likely to include "FI_ACCOUNTANT" in the generated query and reject User Requests about roles that are clearly unrelated to accounting.
How to Use:
- Gather
conversation_history: You need to maintain a log of the conversation. This can be a simple string concatenation of previous user requests and assistant responses. Remember to keep it concise to avoid overwhelming the LLM's context window. A good strategy is to summarize each turn into a few key sentences. - Populate Variables: Before passing the
contextto the LLM, replace{last_request},{questions}, and{conversation_history}with their actual values. - Consider
conversation_historyLength: Be mindful of the length of theconversation_history. Long histories can exceed the LLM's context window. Consider truncating or summarizing the history if it becomes too long.
Important Considerations:
- LLM Context Window: This prompt is more complex, so it will consume more tokens. Ensure that the LLM you are using has a sufficient context window to handle the prompt and the
conversation_history. - Summarization: If your
conversation_historygets very long, consider using another LLM (or a separate prompt) to summarize the history into a shorter, more concise form before feeding it into this prompt. This will help you stay within the context window limit. - Testing: Thoroughly test this prompt with a variety of conversation scenarios to ensure that it behaves as expected. Pay particular attention to cases where the user switches topics or asks clarifying questions.
- Data Format of
conversation_historyThis prompt assumes thatconversation_historyis a single string of text where each turn is separated by some delimiter (e.g., newline\n).
This revised prompt provides a more robust foundation for the LLM to understand the conversational context and construct better search queries. Remember that the effectiveness of this prompt will depend on the quality of the
conversation_historyand the capabilities of the LLM you are using. Remember to tailor to your system for data separation, and if not using NL, then code logic will be needed.This is a significant improvement and should address the problem of false topic shift detections. Good luck!
- Added
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.