write a python code to query over documents like excel, pdf files, etc using langchain
- User
write a python code to query over documents like excel, pdf files, etc using langchain
- Assistant
To query over various document types such as PDFs and Excel files using LangChain, you can follow these steps:
- Install Required Libraries
- Load and Parse Documents
- Create Embeddings and Vector Store
- Set Up the Question-Answering Chain
- Query the Documents
Below is a comprehensive Python script that accomplishes these tasks. This script assumes you have a directory containing PDF and Excel files that you want to query.
1. Install Required Libraries
First, ensure you have all the necessary libraries installed. You can install them using
pip:pip install langchain openai faiss-cpu pandas PyPDF2 openpyxl python-dotenv2. Set Up Environment Variables
It's good practice to keep your API keys secure. You can store your OpenAI API key in a
.envfile.Create a file named
.envin your project directory and add your OpenAI API key:OPENAI_API_KEY=your-openai-api-key3. Python Script to Query Documents
Here's the complete Python script:
import os from typing import List from langchain import OpenAI from langchain.embeddings import OpenAIEmbeddings from langchain.vectorstores import FAISS from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain.chains import RetrievalQA from langchain.document_loaders import PyPDFLoader import pandas as pd from dotenv import load_dotenv # Load environment variables from .env load_dotenv() # Initialize OpenAI API key OPENAI_API_KEY = os.getenv("OPENAI_API_KEY") if not OPENAI_API_KEY: raise ValueError("Please set the OPENAI_API_KEY environment variable.") # Initialize OpenAI embeddings embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY) def load_pdf_documents(file_path: str): """Load PDF documents using PyPDFLoader.""" loader = PyPDFLoader(file_path) documents = loader.load_and_split() return documents def load_excel_documents(file_path: str): """Load Excel documents by converting sheets to text.""" df = pd.read_excel(file_path, sheet_name=None) # Load all sheets documents = [] for sheet_name, sheet_df in df.items(): text = f"Sheet Name: {sheet_name}\n" + sheet_df.to_string(index=False) documents.append({"page_content": text, "metadata": {"source": file_path, "sheet": sheet_name}}) return documents def load_documents_from_directory(directory: str) -> List[dict]: """Load all PDF and Excel documents from a directory.""" all_documents = [] for filename in os.listdir(directory): file_path = os.path.join(directory, filename) if filename.lower().endswith(".pdf"): print(f"Loading PDF: {filename}") pdf_docs = load_pdf_documents(file_path) all_documents.extend(pdf_docs) elif filename.lower().endswith((".xlsx", ".xls")): print(f"Loading Excel: {filename}") excel_docs = load_excel_documents(file_path) # Convert dicts to LangChain Document objects for doc in excel_docs: from langchain.schema import Document document = Document(page_content=doc["page_content"], metadata=doc["metadata"]) all_documents.append(document) else: print(f"Unsupported file type: {filename}") return all_documents def main(): # Specify the directory containing your documents docs_directory = "path_to_your_documents_directory" # Load documents print("Loading documents...") documents = load_documents_from_directory(docs_directory) print(f"Total documents loaded: {len(documents)}") if not documents: print("No documents found. Exiting.") return # Optionally, split the documents into smaller chunks text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200) print("Splitting documents into chunks...") split_documents = text_splitter.split_documents(documents) print(f"Total document chunks: {len(split_documents)}") # Create a FAISS vector store from the documents print("Creating embeddings and building the vector store...") vector_store = FAISS.from_documents(split_documents, embeddings) # Initialize the language model llm = OpenAI(temperature=0, openai_api_key=OPENAI_API_KEY) # Set up the RetrievalQA chain qa = RetrievalQA.from_chain_type( llm=llm, chain_type="stuff", retriever=vector_store.as_retriever(), return_source_documents=True ) print("The system is ready to answer your questions!") while True: query = input("\nPlease enter your question (or type 'exit' to quit): ") if query.lower() in {"exit", "quit"}: print("Exiting the program. Goodbye!") break # Get the answer from the QA system result = qa({"query": query}) print("\nAnswer:") print(result['result']) print("\nSource Documents:") for doc in result['source_documents']: print(f"- {doc.metadata.get('source')} (Sheet: {doc.metadata.get('sheet', 'N/A')})") if __name__ == "__main__": main()4. Explanation of the Script
-
Imports and Environment Setup:
- The script imports necessary libraries and loads the OpenAI API key from the
.envfile.
- The script imports necessary libraries and loads the OpenAI API key from the
-
Document Loaders:
- PDFs: Uses
PyPDFLoaderfrom LangChain to load and split PDF documents into manageable chunks. - Excel: Uses
pandasto read Excel files. Each sheet is converted into a text block with its sheet name. These are then transformed into LangChainDocumentobjects.
- PDFs: Uses
-
Loading Documents from a Directory:
- The
load_documents_from_directoryfunction scans a specified directory for PDF and Excel files, loading them appropriately. Unsupported file types are skipped with a notification.
- The
-
Text Splitting:
- To handle large documents efficiently, the
RecursiveCharacterTextSplitteris used to split documents into chunks of 1000 characters with an overlap of 200 characters.
- To handle large documents efficiently, the
-
Creating Embeddings and Vector Store:
- Embeddings are created for each document chunk using OpenAI's embeddings.
- These embeddings are stored in a FAISS vector store, enabling efficient similarity search.
-
Setting Up the QA Chain:
- An instance of
RetrievalQAis created using an OpenAI language model. - This chain uses the vector store as a retriever to fetch relevant document chunks based on the query.
- An instance of
-
Interactive Query Interface:
- The script enters an interactive loop where users can input questions.
- For each query, the system retrieves the answer and displays the source documents.
5. Usage Instructions
-
Prepare Your Documents:
- Place all your PDF and Excel files in a single directory. Note the path to this directory.
-
Update the Script:
- Replace
"path_to_your_documents_directory"in the script with the actual path to your documents directory.
- Replace
-
Run the Script:
- Execute the script using Python:
python your_script_name.py -
Interact with the System:
- Once running, input your questions when prompted.
- Type
exitorquitto terminate the program.
6. Additional Considerations
-
Handling Other File Types:
- You can extend the
load_documents_from_directoryfunction to handle other file types (e.g., Word documents) by adding appropriate loaders.
- You can extend the
-
Performance Optimization:
- For large document collections, consider persisting the FAISS index to disk to avoid recomputing embeddings on every run.
-
Error Handling:
- Enhance the script with more robust error handling as needed, especially when dealing with various file types and potential reading issues.
-
Security:
- Ensure that your OpenAI API key is kept secure and not exposed in your code repositories.
7. Example Output
After running the script and loading your documents, you might interact as follows:
Loading documents... Loading PDF: example.pdf Loading Excel: data.xlsx Total documents loaded: 15 Splitting documents into chunks... Total document chunks: 30 Creating embeddings and building the vector store... The system is ready to answer your questions! Please enter your question (or type 'exit' to quit): What is the sales trend in Q1? Answer: The sales in Q1 showed a steady increase, starting at $50,000 in January and reaching $75,000 by March, indicating a positive growth trend. Source Documents: - sales_report.pdf (Sheet: N/A) - data.xlsx (Sheet: Sales_Q1)This output shows an answer generated based on the content of your loaded documents, along with references to the source files and specific sheets if applicable.
This script provides a solid foundation for querying a mix of PDF and Excel documents using LangChain. You can further customize and expand it based on your specific requirements and the types of documents you work with.