suppose we are solving a problem of providing construction by a user request to llm…
- User
suppose we are solving a problem of providing construction by a user request to llm (string) all the nescessary context and tools related, e.g. if we want the llm to fix unit tests, it should discover the project (read search list tools needed), run unit tests (exec tool needed) get the failed tests info. I want to have s system that by user input string can perform a context retrieval, tool description search. The system can include a RAG with chroma db, a graph with neo4j, an embedding calculation (ollama txt embeddings) it may include as well classifiers with classical ML on top of embeddings, suggest how would you approch this task
- Assistant
Here's one approach to designing such a system:
1. Define System Requirements and Workflow
- User Input: The system receives a plain text request from the user (e.g., “fix the unit tests”).
- Context Discovery: The system should automatically discover the appropriate context (e.g., the project structure, specific files, or modules).
- Tool and Task Mapping: Based on the request and discovered context, determine which tools and actions to invoke—for example:
- RAG (Retrieval-Augmented Generation): Use Chroma DB to retrieve relevant parts of the project documentation or code snippets.
- Graph Traversal: Use Neo4j to query relationships or tool descriptions stored in a graph (e.g., see which modules affect unit tests).
- Embeddings & Classification: Calculate sentence or document embeddings (using Ollama TXT embeddings) to compare the request to potential tasks and also employ classical ML classifiers over embeddings to decide the intent or select the most suitable tool.
- Tool Execution: Once the relevant tools are identified, execute them (e.g., run the unit tests, gather their output, and then feed the failures back to the system for further processing).
2. Architectural Components
-
User Interface (UI) / API Layer
- Input Module: Accepts the user’s request string.
- Output Module: Displays the results and any feedback from tool invocations.
-
Query Processing and Orchestration Module
- Intent Analyzer: Uses the LLM to understand the higher-level intent from the user request.
- Action Planner: Decides the sequence of operations (e.g., retrieve context → search tool descriptions → execute tools).
- Workflow Executor: Manages the orchestration, triggering various modules as required.
-
Context Retrieval Module (RAG with Chroma DB)
- Indexing: Maintain an indexed version of project documentation, code comments, specifications, etc.
- Querying: When a request comes in, use the query to retrieve relevant sections that can inform the task.
- Embedding Calculation: Use Ollama TXT embeddings to convert both the query and documents into embeddings to compute similarity.
-
Knowledge Base / Graph Module (using Neo4j)
- Graph Data Store: Store metadata about the project, relationships between files, descriptions of available tools, dependencies, etc.
- Query Engine: When the orchestrator needs to discover which tools or modules are related to unit testing (for instance), it can query this graph.
-
Classifier Module
- Embedding-based Classifiers: Use classical machine learning models to classify the query based on its embedding (e.g., is it a “fix” request, a “run” request, an “explore” request, etc.).
- Task Recommendation: Based on the classification, recommend the appropriate action(s).
-
Tool Execution Module
- Tool Registry: Maintain a list of available tools (e.g., unit test runner, static code analyzer, context retriever script, etc.) along with their descriptions and execution commands.
- Dynamic Execution: Once the task plan is decided, the module executes the corresponding tools (could involve shell commands, API calls, etc.).
- Result Gathering: Collect the output from tool executions (e.g., results of unit test failures) to feed back into the LLM.
3. Proposed Workflow Example (Unit Tests Fixing)
-
Initial Query Processing:
- User input: "Please fix the unit tests."
- The LLM uses its language understanding to break down the request, realizing that the task involves project diagnosis, running tests, and fixing issues.
-
Context Retrieval:
- The system computes embeddings for the input query using Ollama TXT embeddings.
- It queries the Chroma DB to retrieve relevant project documents (e.g., test files, error logs, related documentation).
-
Graph Query and Tool Mapping:
- The system turns to the Neo4j graph to see which modules are related to unit tests.
- It maps out the process: discover test files → run unit tests using a designated execution tool → collect failure logs.
-
Classification and Decision Making:
- A classifier (running on the embedding of the input) confirms that the request involves code correction for testing.
- The system identifies which tools need to be executed (a test runner, code fixer module, etc.).
-
Tool Execution and Iterative Loop:
- The tool execution module calls the unit test execution tool, gathers the output (e.g., error logs).
- Using the retrieved error logs, the system may further ask the LLM to suggest fixes or even automate changes.
- Optionally, re-run unit tests to confirm whether the fixes were applied successfully.
-
Feedback:
- The system communicates back to the user the steps taken, the changes made, or requests further clarifications if needed.
4. Integration Considerations
- Middleware / Message Bus: Implement a message bus to orchestrate communication between modules asynchronously if necessary.
- Logging & Traceability: Maintain detailed logs across modules (especially in production scenarios) to trace steps from the user’s request to the final execution.
- Security & Resource Management: Ensure that code execution tools operate in a sandboxed environment and that context retrieval doesn’t unintentionally leak sensitive information.
- Extensibility: Design modules such that new tools, classifiers, or context sources can be added without overhauling the whole system.
5. Development and Testing Strategies
- Prototyping: Start with a simple end-to-end flow that takes a user query and executes one tool, then progressively add context retrieval and graph-based reasoning.
- Simulation: Simulate different user requests to see how the system maps them to appropriate tool execution.
- User Feedback Loop: Once deployed, gather user feedback to improve classification rules and the mapping between context and actions.
This architecture uses a combination of modern retrieval techniques (RAG with Chroma DB), structured knowledge (Neo4j graph), and semantic understanding (Ollama embeddings with ML classifiers), all coordinated by an intelligent orchestrator that leverages the strengths of an LLM. This combined approach ensures that the system can dynamically respond to varied user requests, discover and retrieve context, and execute the appropriate toolchain to address the user’s needs.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.