Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

write a python code to query over documents like excel, pdf files, etc using langchain

  1. User

    write a python code to query over documents like excel, pdf files, etc using langchain

  2. Assistant

    To query over various document types such as PDFs and Excel files using LangChain, you can follow these steps:

    1. Install Required Libraries
    2. Load and Parse Documents
    3. Create Embeddings and Vector Store
    4. Set Up the Question-Answering Chain
    5. Query the Documents

    Below is a comprehensive Python script that accomplishes these tasks. This script assumes you have a directory containing PDF and Excel files that you want to query.

    1. Install Required Libraries

    First, ensure you have all the necessary libraries installed. You can install them using pip:

    pip install langchain openai faiss-cpu pandas PyPDF2 openpyxl python-dotenv
    

    2. Set Up Environment Variables

    It's good practice to keep your API keys secure. You can store your OpenAI API key in a .env file.

    Create a file named .env in your project directory and add your OpenAI API key:

    OPENAI_API_KEY=your-openai-api-key
    

    3. Python Script to Query Documents

    Here's the complete Python script:

    import os
    from typing import List
    
    from langchain import OpenAI
    from langchain.embeddings import OpenAIEmbeddings
    from langchain.vectorstores import FAISS
    from langchain.text_splitter import RecursiveCharacterTextSplitter
    from langchain.chains import RetrievalQA
    
    from langchain.document_loaders import PyPDFLoader
    import pandas as pd
    
    from dotenv import load_dotenv
    
    # Load environment variables from .env
    load_dotenv()
    
    # Initialize OpenAI API key
    OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
    if not OPENAI_API_KEY:
        raise ValueError("Please set the OPENAI_API_KEY environment variable.")
    
    # Initialize OpenAI embeddings
    embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY)
    
    def load_pdf_documents(file_path: str):
        """Load PDF documents using PyPDFLoader."""
        loader = PyPDFLoader(file_path)
        documents = loader.load_and_split()
        return documents
    
    def load_excel_documents(file_path: str):
        """Load Excel documents by converting sheets to text."""
        df = pd.read_excel(file_path, sheet_name=None)  # Load all sheets
        documents = []
        for sheet_name, sheet_df in df.items():
            text = f"Sheet Name: {sheet_name}\n" + sheet_df.to_string(index=False)
            documents.append({"page_content": text, "metadata": {"source": file_path, "sheet": sheet_name}})
        return documents
    
    def load_documents_from_directory(directory: str) -> List[dict]:
        """Load all PDF and Excel documents from a directory."""
        all_documents = []
        for filename in os.listdir(directory):
            file_path = os.path.join(directory, filename)
            if filename.lower().endswith(".pdf"):
                print(f"Loading PDF: {filename}")
                pdf_docs = load_pdf_documents(file_path)
                all_documents.extend(pdf_docs)
            elif filename.lower().endswith((".xlsx", ".xls")):
                print(f"Loading Excel: {filename}")
                excel_docs = load_excel_documents(file_path)
                # Convert dicts to LangChain Document objects
                for doc in excel_docs:
                    from langchain.schema import Document
                    document = Document(page_content=doc["page_content"], metadata=doc["metadata"])
                    all_documents.append(document)
            else:
                print(f"Unsupported file type: {filename}")
        return all_documents
    
    def main():
        # Specify the directory containing your documents
        docs_directory = "path_to_your_documents_directory"
    
        # Load documents
        print("Loading documents...")
        documents = load_documents_from_directory(docs_directory)
        print(f"Total documents loaded: {len(documents)}")
    
        if not documents:
            print("No documents found. Exiting.")
            return
    
        # Optionally, split the documents into smaller chunks
        text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
        print("Splitting documents into chunks...")
        split_documents = text_splitter.split_documents(documents)
        print(f"Total document chunks: {len(split_documents)}")
    
        # Create a FAISS vector store from the documents
        print("Creating embeddings and building the vector store...")
        vector_store = FAISS.from_documents(split_documents, embeddings)
    
        # Initialize the language model
        llm = OpenAI(temperature=0, openai_api_key=OPENAI_API_KEY)
    
        # Set up the RetrievalQA chain
        qa = RetrievalQA.from_chain_type(
            llm=llm,
            chain_type="stuff",
            retriever=vector_store.as_retriever(),
            return_source_documents=True
        )
    
        print("The system is ready to answer your questions!")
    
        while True:
            query = input("\nPlease enter your question (or type 'exit' to quit): ")
            if query.lower() in {"exit", "quit"}:
                print("Exiting the program. Goodbye!")
                break
    
            # Get the answer from the QA system
            result = qa({"query": query})
    
            print("\nAnswer:")
            print(result['result'])
    
            print("\nSource Documents:")
            for doc in result['source_documents']:
                print(f"- {doc.metadata.get('source')} (Sheet: {doc.metadata.get('sheet', 'N/A')})")
    
    if __name__ == "__main__":
        main()
    

    4. Explanation of the Script

    1. Imports and Environment Setup:

      • The script imports necessary libraries and loads the OpenAI API key from the .env file.
    2. Document Loaders:

      • PDFs: Uses PyPDFLoader from LangChain to load and split PDF documents into manageable chunks.
      • Excel: Uses pandas to read Excel files. Each sheet is converted into a text block with its sheet name. These are then transformed into LangChain Document objects.
    3. Loading Documents from a Directory:

      • The load_documents_from_directory function scans a specified directory for PDF and Excel files, loading them appropriately. Unsupported file types are skipped with a notification.
    4. Text Splitting:

      • To handle large documents efficiently, the RecursiveCharacterTextSplitter is used to split documents into chunks of 1000 characters with an overlap of 200 characters.
    5. Creating Embeddings and Vector Store:

      • Embeddings are created for each document chunk using OpenAI's embeddings.
      • These embeddings are stored in a FAISS vector store, enabling efficient similarity search.
    6. Setting Up the QA Chain:

      • An instance of RetrievalQA is created using an OpenAI language model.
      • This chain uses the vector store as a retriever to fetch relevant document chunks based on the query.
    7. Interactive Query Interface:

      • The script enters an interactive loop where users can input questions.
      • For each query, the system retrieves the answer and displays the source documents.

    5. Usage Instructions

    1. Prepare Your Documents:

      • Place all your PDF and Excel files in a single directory. Note the path to this directory.
    2. Update the Script:

      • Replace "path_to_your_documents_directory" in the script with the actual path to your documents directory.
    3. Run the Script:

      • Execute the script using Python:
      python your_script_name.py
      
    4. Interact with the System:

      • Once running, input your questions when prompted.
      • Type exit or quit to terminate the program.

    6. Additional Considerations

    • Handling Other File Types:

      • You can extend the load_documents_from_directory function to handle other file types (e.g., Word documents) by adding appropriate loaders.
    • Performance Optimization:

      • For large document collections, consider persisting the FAISS index to disk to avoid recomputing embeddings on every run.
    • Error Handling:

      • Enhance the script with more robust error handling as needed, especially when dealing with various file types and potential reading issues.
    • Security:

      • Ensure that your OpenAI API key is kept secure and not exposed in your code repositories.

    7. Example Output

    After running the script and loading your documents, you might interact as follows:

    Loading documents...
    Loading PDF: example.pdf
    Loading Excel: data.xlsx
    Total documents loaded: 15
    Splitting documents into chunks...
    Total document chunks: 30
    Creating embeddings and building the vector store...
    The system is ready to answer your questions!
    
    Please enter your question (or type 'exit' to quit): What is the sales trend in Q1?
    
    Answer:
    The sales in Q1 showed a steady increase, starting at $50,000 in January and reaching $75,000 by March, indicating a positive growth trend.
    
    Source Documents:
    - sales_report.pdf (Sheet: N/A)
    - data.xlsx (Sheet: Sales_Q1)
    

    This output shows an answer generated based on the content of your loaded documents, along with references to the source files and specific sheets if applicable.


    This script provides a solid foundation for querying a mix of PDF and Excel documents using LangChain. You can further customize and expand it based on your specific requirements and the types of documents you work with.