Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

Following this instruction : 1. Strictly adhere to PEP-8 standards, ensuring code is…

  1. User

    Following this instruction :

    1. Strictly adhere to PEP-8 standards, ensuring code is clean, well-structured, and follows Python's style guide for readability.
    2. Use appropriate modules like typing for type hints, ensuring code clarity, maintainability, and leveraging Python's standard libraries effectively.
    3. Focus on time, space, robustness, scalability; write optimized, efficient code that can handle large datasets and scale seamlessly.
    4. Implement comprehensive error handling, catching exceptions gracefully, providing meaningful error messages, and ensuring code robustness under unexpected conditions.
    5. Ensure the developer agent writes advanced, maintainable Python code, balancing performance and readability while adhering to best practices.

    TASK : main problem statement user query , we should relevent content , meta_data including relevent file name also

    1. using user folder (reccusively in nature) we should convert into folder(with .txt files) write properly so that chucking will every easy next step import os import logging from typing import List, Optional from pathlib import Path from concurrent.futures import ThreadPoolExecutor import shutil import pandas as pd import json import yaml import docx import pptx import fitz

    import nltk from nltk.tokenize import word_tokenize from nltk.corpus import stopwords

    from raglogger import setup_logger

    logger=setup_logger()

    Download necessary NLTK resources quietly

    nltk.download('punkt', quiet=True) nltk.download('stopwords', quiet=True)

    1. folder(with .txt files) , user query , k=selectinon , int

    import datasets from typing import List, Dict, Optional from langchain.docstore.document import Document as LangchainDocument from langchain.text_splitter import RecursiveCharacterTextSplitter from transformers import AutoTokenizer from tqdm import tqdm import logging from concurrent.futures import ProcessPoolExecutor from langchain_community.vectorstores import FAISS from langchain_huggingface import HuggingFaceEmbeddings from langchain_community.vectorstores.utils import DistanceStrategy

    from raglogger import setup_logger

    logger = setup_logger()

    Constants for embedding

    EMBEDDING_MODEL_NAME = "thenlper/gte-large" # Replace with your preferred model

    Define markdown separators

    MARKDOWN_SEPARATORS = [ "\n#{1,6} ", "```\n", "\n\\\*+\n", "\n---+\n", "\n___+\n", "\n\n", "\n", " ", # Include space as separator "", # Empty string to finalize small chunks ]

    1. saving and loading vector data base Serialization save

    Saves an object to a disk file.

    load

    Loads an object saved with torch.save() from a file.

    Parallelism get_num_threads

    Returns the number of threads used for parallelizing CPU operations

    set_num_threads

    Sets the number of threads used for intraop parallelism on CPU.

    get_num_interop_threads

    Returns the number of threads used for inter-op parallelism on CPU (e.g.

    set_num_interop_threads

    Sets the number of threads used for interop parallelism (e.g.

  2. Assistant

    Here is a Python module that accomplishes your specified tasks:

    1. Recursively process a user's folder, converting files into .txt files for easy chunking.

    2. Process the folder with .txt files, perform text embedding, and retrieve the top k relevant documents based on a user query.

    3. Implement saving and loading of the vector database.

    The code is structured for readability, maintainability, and adheres to PEP-8 standards.

    import os
    import logging
    import shutil
    from typing import List, Optional, Dict
    from pathlib import Path
    from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
    
    import json
    import yaml
    import pandas as pd
    import docx
    import pptx
    import fitz  # PyMuPDF for PDF processing
    import nltk
    from nltk.tokenize import word_tokenize
    from nltk.corpus import stopwords
    
    from datasets import Dataset
    from transformers import AutoTokenizer
    from tqdm import tqdm
    from langchain.docstore.document import Document as LangchainDocument
    from langchain.text_splitter import RecursiveCharacterTextSplitter
    from langchain_community.vectorstores import FAISS
    from langchain_huggingface import HuggingFaceEmbeddings
    from langchain_community.vectorstores.utils import DistanceStrategy
    
    from raglogger import setup_logger
    
    # Initialize logger
    logger = setup_logger()
    
    # Download necessary NLTK resources quietly
    nltk.download('punkt', quiet=True)
    nltk.download('stopwords', quiet=True)
    
    # Constants
    EMBEDDING_MODEL_NAME = "thenlper/gte-large"  # Replace with your preferred model
    CHUNK_SIZE = 500  # Adjust based on your needs
    CHUNK_OVERLAP = 50  # Overlap between chunks for context
    

    Step 1: Convert Files to Text

    def extract_text_from_pdf(file_path: Path) -> str:
        """Extract text from a PDF file."""
        try:
            with fitz.open(file_path) as doc:
                text = ""
                for page in doc:
                    text += page.get_text()
            return text
        except Exception as e:
            logger.error(f"Error reading PDF file {file_path}: {e}")
            return ""
    
    def extract_text_from_docx(file_path: Path) -> str:
        """Extract text from a Word document."""
        try:
            doc = docx.Document(file_path)
            text = "\n".join([para.text for para in doc.paragraphs])
            return text
        except Exception as e:
            logger.error(f"Error reading DOCX file {file_path}: {e}")
            return ""
    
    def extract_text_from_pptx(file_path: Path) -> str:
        """Extract text from a PowerPoint presentation."""
        try:
            prs = pptx.Presentation(file_path)
            text = ""
            for slide in prs.slides:
                for shape in slide.shapes:
                    if hasattr(shape, "text"):
                        text += shape.text + "\n"
            return text
        except Exception as e:
            logger.error(f"Error reading PPTX file {file_path}: {e}")
            return ""
    
    def extract_text_from_json(file_path: Path) -> str:
        """Extract text from a JSON file."""
        try:
            with open(file_path, 'r', encoding='utf-8') as f:
                data = json.load(f)
            return json.dumps(data)
        except Exception as e:
            logger.error(f"Error reading JSON file {file_path}: {e}")
            return ""
    
    def extract_text_from_yaml(file_path: Path) -> str:
        """Extract text from a YAML file."""
        try:
            with open(file_path, 'r', encoding='utf-8') as f:
                data = yaml.safe_load(f)
            return yaml.dump(data)
        except Exception as e:
            logger.error(f"Error reading YAML file {file_path}: {e}")
            return ""
    
    def extract_text_from_csv(file_path: Path) -> str:
        """Extract text from a CSV file."""
        try:
            df = pd.read_csv(file_path)
            return df.to_csv(index=False)
        except Exception as e:
            logger.error(f"Error reading CSV file {file_path}: {e}")
            return ""
    
    def extract_text(file_path: Path) -> str:
        """Extract text from a file based on its extension."""
        ext = file_path.suffix.lower()
        if ext == '.pdf':
            return extract_text_from_pdf(file_path)
        elif ext == '.docx':
            return extract_text_from_docx(file_path)
        elif ext == '.pptx':
            return extract_text_from_pptx(file_path)
        elif ext == '.json':
            return extract_text_from_json(file_path)
        elif ext in ['.yml', '.yaml']:
            return extract_text_from_yaml(file_path)
        elif ext == '.csv':
            return extract_text_from_csv(file_path)
        elif ext == '.txt':
            try:
                with open(file_path, 'r', encoding='utf-8') as f:
                    return f.read()
            except Exception as e:
                logger.error(f"Error reading TXT file {file_path}: {e}")
                return ""
        else:
            logger.warning(f"Unsupported file type: {file_path}")
            return ""
    
    def process_files(input_dir: Path, output_dir: Path):
        """Recursively process files in input_dir and save text files to output_dir."""
        if not output_dir.exists():
            output_dir.mkdir(parents=True)
    
        def process_file(file_path: Path):
            text = extract_text(file_path)
            if text:
                relative_path = file_path.relative_to(input_dir)
                output_file = output_dir.joinpath(relative_path).with_suffix('.txt')
                output_file.parent.mkdir(parents=True, exist_ok=True)
                try:
                    with open(output_file, 'w', encoding='utf-8') as f:
                        f.write(text)
                    logger.info(f"Processed and saved text from {file_path} to {output_file}")
                except Exception as e:
                    logger.error(f"Error writing to file {output_file}: {e}")
    
        files = [p for p in input_dir.rglob('*') if p.is_file()]
        with ThreadPoolExecutor() as executor:
            executor.map(process_file, files)
    

    Step 2: Build Vector Database and Query

    def load_documents(txt_folder: Path) -> List[LangchainDocument]:
        """Load text files and return as LangchainDocument objects."""
        documents = []
        files = list(txt_folder.rglob('*.txt'))
        for file_path in tqdm(files, desc="Loading documents"):
            try:
                with open(file_path, 'r', encoding='utf-8') as f:
                    text = f.read()
                metadata = {
                    'source': str(file_path),
                    'file_name': file_path.name,
                    'file_path': str(file_path)
                }
                document = LangchainDocument(page_content=text, metadata=metadata)
                documents.append(document)
            except Exception as e:
                logger.error(f"Error loading document {file_path}: {e}")
        return documents
    
    def split_documents(documents: List[LangchainDocument]) -> List[LangchainDocument]:
        """Split documents into smaller chunks."""
        text_splitter = RecursiveCharacterTextSplitter(
            chunk_size=CHUNK_SIZE,
            chunk_overlap=CHUNK_OVERLAP,
            separators=["\n\n", "\n", " ", ""]
        )
        split_docs = []
        for doc in tqdm(documents, desc="Splitting documents"):
            splits = text_splitter.split_text(doc.page_content)
            for i, content in enumerate(splits):
                metadata = doc.metadata.copy()
                metadata['chunk'] = i
                split_docs.append(LangchainDocument(page_content=content, metadata=metadata))
        return split_docs
    
    def create_embeddings(documents: List[LangchainDocument]) -> FAISS:
        """Create embeddings and build FAISS index."""
        embeddings = HuggingFaceEmbeddings(
            model_name=EMBEDDING_MODEL_NAME,
            model_kwargs={'device': 'cpu'}  # Change to 'cuda' if using GPU
        )
        vector_store = FAISS.from_documents(
            documents,
            embedding=embeddings,
            distance_strategy=DistanceStrategy.EUCLIDEAN
        )
        return vector_store
    
    def save_vector_store(vector_store: FAISS, index_path: Path, metadata_path: Path):
        """Save the FAISS index and metadata."""
        vector_store.save_local(str(index_path), str(metadata_path))
        logger.info(f"Vector store saved to {index_path} and {metadata_path}")
    
    def load_vector_store(index_path: Path, metadata_path: Path) -> FAISS:
        """Load the FAISS index and metadata."""
        embeddings = HuggingFaceEmbeddings(model_name=EMBEDDING_MODEL_NAME)
        vector_store = FAISS.load_local(str(index_path), embeddings, str(metadata_path))
        logger.info(f"Vector store loaded from {index_path} and {metadata_path}")
        return vector_store
    
    def query_vector_store(vector_store: FAISS, query: str, k: int = 5) -> List[Dict]:
        """Query the vector store and get top k documents."""
        docs = vector_store.similarity_search(query, k=k)
        results = []
        for doc in docs:
            result = {
                'content': doc.page_content,
                'metadata': doc.metadata
            }
            results.append(result)
        return results
    

    Step 3: Main Execution Logic

    def main():
        # Define input and output directories
        input_dir = Path('/path/to/user_folder')  # Replace with your input directory
        txt_output_dir = Path('/path/to/output_txt_folder')  # Replace with your output directory
    
        # Step 1: Process files and convert to .txt
        process_files(input_dir, txt_output_dir)
    
        # Step 2: Load documents and build vector database
        documents = load_documents(txt_output_dir)
        split_docs = split_documents(documents)
        vector_store = create_embeddings(split_docs)
    
        # Save the vector store
        index_path = Path('faiss_index')
        metadata_path = Path('faiss_index_metadata')
        save_vector_store(vector_store, index_path, metadata_path)
    
        # Load the vector store (optional, to demonstrate loading)
        vector_store = load_vector_store(index_path, metadata_path)
    
        # Query the vector store
        user_query = "Enter your query here"  # Replace with the user's query
        top_k = 5  # Replace with desired number of results
        results = query_vector_store(vector_store, user_query, k=top_k)
    
        # Display results
        for i, result in enumerate(results):
            print(f"Result {i + 1}:")
            print(f"Content:\n{result['content']}\n")
            print(f"Metadata:\n{result['metadata']}\n")
            print("=" * 50)
    
    if __name__ == "__main__":
        main()
    

    Notes on the Implementation

    • Logging and Error Handling: Comprehensive logging and error handling are implemented to ensure any issues are caught gracefully.

    • Type Hints: All functions include type hints for clarity and maintainability.

    • Performance Considerations:

      • ThreadPoolExecutor is used in process_files to process files concurrently, improving I/O-bound operation performance.

      • ProcessPoolExecutor can be used for CPU-bound tasks if needed (e.g., in text processing or embeddings), though care must be taken with large objects due to inter-process communication overhead.

    • Scalability: The code is designed to handle large datasets, with progress bars provided by tqdm to monitor processing.

    • Embedding Model: The thenlper/gte-large model is used for embeddings. Replace EMBEDDING_MODEL_NAME with any model suitable for your needs.

    • FAISS Index: The FAISS vector store provides efficient similarity search capabilities, essential for handling large-scale vector data.

    • Text Splitting: Documents are split into chunks to manage memory usage and improve retrieval relevance.

    • NLTK Stopwords and Tokenization: NLTK resources are prepared if further text processing is needed (e.g., stopword removal, tokenization).

    Requirements

    Ensure that you have the following packages installed:

    pip install nltk pandas PyMuPDF python-docx python-pptx tqdm transformers langchain datasets
    

    Additionally, install any other dependencies required by the imported modules.

    Adjustments

    • File Paths: Update /path/to/user_folder and /path/to/output_txt_folder with the actual paths to your input and output directories.

    • Embedding Device: If you have a GPU available, you can set model_kwargs={'device': 'cuda'} when initializing HuggingFaceEmbeddings to utilize GPU acceleration.

    • Custom Separators: Adjust the separators in RecursiveCharacterTextSplitter if needed to better suit your data's structure.

    • Chunk Size and Overlap: Modify CHUNK_SIZE and CHUNK_OVERLAP to fine-tune how the documents are split.


    By following these steps, the code provides a robust, scalable solution that adheres to best practices and your specified requirements.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.