Following this instruction : 1. Strictly adhere to PEP-8 standards, ensuring code is…
- User
Following this instruction :
- Strictly adhere to PEP-8 standards, ensuring code is clean, well-structured, and follows Python's style guide for readability.
- Use appropriate modules like
typingfor type hints, ensuring code clarity, maintainability, and leveraging Python's standard libraries effectively. - Focus on time, space, robustness, scalability; write optimized, efficient code that can handle large datasets and scale seamlessly.
- Implement comprehensive error handling, catching exceptions gracefully, providing meaningful error messages, and ensuring code robustness under unexpected conditions.
- Ensure the developer agent writes advanced, maintainable Python code, balancing performance and readability while adhering to best practices.
TASK : main problem statement user query , we should relevent content , meta_data including relevent file name also
- using user folder (reccusively in nature) we should convert into folder(with .txt files) write properly so that chucking will every easy next step import os import logging from typing import List, Optional from pathlib import Path from concurrent.futures import ThreadPoolExecutor import shutil import pandas as pd import json import yaml import docx import pptx import fitz
import nltk from nltk.tokenize import word_tokenize from nltk.corpus import stopwords
from raglogger import setup_logger
logger=setup_logger()
Download necessary NLTK resources quietly
nltk.download('punkt', quiet=True) nltk.download('stopwords', quiet=True)
- folder(with .txt files) , user query , k=selectinon , int
import datasets from typing import List, Dict, Optional from langchain.docstore.document import Document as LangchainDocument from langchain.text_splitter import RecursiveCharacterTextSplitter from transformers import AutoTokenizer from tqdm import tqdm import logging from concurrent.futures import ProcessPoolExecutor from langchain_community.vectorstores import FAISS from langchain_huggingface import HuggingFaceEmbeddings from langchain_community.vectorstores.utils import DistanceStrategy
from raglogger import setup_logger
logger = setup_logger()
Constants for embedding
EMBEDDING_MODEL_NAME = "thenlper/gte-large" # Replace with your preferred model
Define markdown separators
MARKDOWN_SEPARATORS = [ "\n#{1,6} ", "```\n", "\n\\\*+\n", "\n---+\n", "\n___+\n", "\n\n", "\n", " ", # Include space as separator "", # Empty string to finalize small chunks ]
- saving and loading vector data base Serialization save
Saves an object to a disk file.
load
Loads an object saved with torch.save() from a file.
Parallelism get_num_threads
Returns the number of threads used for parallelizing CPU operations
set_num_threads
Sets the number of threads used for intraop parallelism on CPU.
get_num_interop_threads
Returns the number of threads used for inter-op parallelism on CPU (e.g.
set_num_interop_threads
Sets the number of threads used for interop parallelism (e.g.
- Assistant
Here is a Python module that accomplishes your specified tasks:
-
Recursively process a user's folder, converting files into
.txtfiles for easy chunking. -
Process the folder with
.txtfiles, perform text embedding, and retrieve the topkrelevant documents based on a user query. -
Implement saving and loading of the vector database.
The code is structured for readability, maintainability, and adheres to PEP-8 standards.
import os import logging import shutil from typing import List, Optional, Dict from pathlib import Path from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor import json import yaml import pandas as pd import docx import pptx import fitz # PyMuPDF for PDF processing import nltk from nltk.tokenize import word_tokenize from nltk.corpus import stopwords from datasets import Dataset from transformers import AutoTokenizer from tqdm import tqdm from langchain.docstore.document import Document as LangchainDocument from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain_community.vectorstores import FAISS from langchain_huggingface import HuggingFaceEmbeddings from langchain_community.vectorstores.utils import DistanceStrategy from raglogger import setup_logger # Initialize logger logger = setup_logger() # Download necessary NLTK resources quietly nltk.download('punkt', quiet=True) nltk.download('stopwords', quiet=True) # Constants EMBEDDING_MODEL_NAME = "thenlper/gte-large" # Replace with your preferred model CHUNK_SIZE = 500 # Adjust based on your needs CHUNK_OVERLAP = 50 # Overlap between chunks for contextStep 1: Convert Files to Text
def extract_text_from_pdf(file_path: Path) -> str: """Extract text from a PDF file.""" try: with fitz.open(file_path) as doc: text = "" for page in doc: text += page.get_text() return text except Exception as e: logger.error(f"Error reading PDF file {file_path}: {e}") return "" def extract_text_from_docx(file_path: Path) -> str: """Extract text from a Word document.""" try: doc = docx.Document(file_path) text = "\n".join([para.text for para in doc.paragraphs]) return text except Exception as e: logger.error(f"Error reading DOCX file {file_path}: {e}") return "" def extract_text_from_pptx(file_path: Path) -> str: """Extract text from a PowerPoint presentation.""" try: prs = pptx.Presentation(file_path) text = "" for slide in prs.slides: for shape in slide.shapes: if hasattr(shape, "text"): text += shape.text + "\n" return text except Exception as e: logger.error(f"Error reading PPTX file {file_path}: {e}") return "" def extract_text_from_json(file_path: Path) -> str: """Extract text from a JSON file.""" try: with open(file_path, 'r', encoding='utf-8') as f: data = json.load(f) return json.dumps(data) except Exception as e: logger.error(f"Error reading JSON file {file_path}: {e}") return "" def extract_text_from_yaml(file_path: Path) -> str: """Extract text from a YAML file.""" try: with open(file_path, 'r', encoding='utf-8') as f: data = yaml.safe_load(f) return yaml.dump(data) except Exception as e: logger.error(f"Error reading YAML file {file_path}: {e}") return "" def extract_text_from_csv(file_path: Path) -> str: """Extract text from a CSV file.""" try: df = pd.read_csv(file_path) return df.to_csv(index=False) except Exception as e: logger.error(f"Error reading CSV file {file_path}: {e}") return "" def extract_text(file_path: Path) -> str: """Extract text from a file based on its extension.""" ext = file_path.suffix.lower() if ext == '.pdf': return extract_text_from_pdf(file_path) elif ext == '.docx': return extract_text_from_docx(file_path) elif ext == '.pptx': return extract_text_from_pptx(file_path) elif ext == '.json': return extract_text_from_json(file_path) elif ext in ['.yml', '.yaml']: return extract_text_from_yaml(file_path) elif ext == '.csv': return extract_text_from_csv(file_path) elif ext == '.txt': try: with open(file_path, 'r', encoding='utf-8') as f: return f.read() except Exception as e: logger.error(f"Error reading TXT file {file_path}: {e}") return "" else: logger.warning(f"Unsupported file type: {file_path}") return "" def process_files(input_dir: Path, output_dir: Path): """Recursively process files in input_dir and save text files to output_dir.""" if not output_dir.exists(): output_dir.mkdir(parents=True) def process_file(file_path: Path): text = extract_text(file_path) if text: relative_path = file_path.relative_to(input_dir) output_file = output_dir.joinpath(relative_path).with_suffix('.txt') output_file.parent.mkdir(parents=True, exist_ok=True) try: with open(output_file, 'w', encoding='utf-8') as f: f.write(text) logger.info(f"Processed and saved text from {file_path} to {output_file}") except Exception as e: logger.error(f"Error writing to file {output_file}: {e}") files = [p for p in input_dir.rglob('*') if p.is_file()] with ThreadPoolExecutor() as executor: executor.map(process_file, files)Step 2: Build Vector Database and Query
def load_documents(txt_folder: Path) -> List[LangchainDocument]: """Load text files and return as LangchainDocument objects.""" documents = [] files = list(txt_folder.rglob('*.txt')) for file_path in tqdm(files, desc="Loading documents"): try: with open(file_path, 'r', encoding='utf-8') as f: text = f.read() metadata = { 'source': str(file_path), 'file_name': file_path.name, 'file_path': str(file_path) } document = LangchainDocument(page_content=text, metadata=metadata) documents.append(document) except Exception as e: logger.error(f"Error loading document {file_path}: {e}") return documents def split_documents(documents: List[LangchainDocument]) -> List[LangchainDocument]: """Split documents into smaller chunks.""" text_splitter = RecursiveCharacterTextSplitter( chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP, separators=["\n\n", "\n", " ", ""] ) split_docs = [] for doc in tqdm(documents, desc="Splitting documents"): splits = text_splitter.split_text(doc.page_content) for i, content in enumerate(splits): metadata = doc.metadata.copy() metadata['chunk'] = i split_docs.append(LangchainDocument(page_content=content, metadata=metadata)) return split_docs def create_embeddings(documents: List[LangchainDocument]) -> FAISS: """Create embeddings and build FAISS index.""" embeddings = HuggingFaceEmbeddings( model_name=EMBEDDING_MODEL_NAME, model_kwargs={'device': 'cpu'} # Change to 'cuda' if using GPU ) vector_store = FAISS.from_documents( documents, embedding=embeddings, distance_strategy=DistanceStrategy.EUCLIDEAN ) return vector_store def save_vector_store(vector_store: FAISS, index_path: Path, metadata_path: Path): """Save the FAISS index and metadata.""" vector_store.save_local(str(index_path), str(metadata_path)) logger.info(f"Vector store saved to {index_path} and {metadata_path}") def load_vector_store(index_path: Path, metadata_path: Path) -> FAISS: """Load the FAISS index and metadata.""" embeddings = HuggingFaceEmbeddings(model_name=EMBEDDING_MODEL_NAME) vector_store = FAISS.load_local(str(index_path), embeddings, str(metadata_path)) logger.info(f"Vector store loaded from {index_path} and {metadata_path}") return vector_store def query_vector_store(vector_store: FAISS, query: str, k: int = 5) -> List[Dict]: """Query the vector store and get top k documents.""" docs = vector_store.similarity_search(query, k=k) results = [] for doc in docs: result = { 'content': doc.page_content, 'metadata': doc.metadata } results.append(result) return resultsStep 3: Main Execution Logic
def main(): # Define input and output directories input_dir = Path('/path/to/user_folder') # Replace with your input directory txt_output_dir = Path('/path/to/output_txt_folder') # Replace with your output directory # Step 1: Process files and convert to .txt process_files(input_dir, txt_output_dir) # Step 2: Load documents and build vector database documents = load_documents(txt_output_dir) split_docs = split_documents(documents) vector_store = create_embeddings(split_docs) # Save the vector store index_path = Path('faiss_index') metadata_path = Path('faiss_index_metadata') save_vector_store(vector_store, index_path, metadata_path) # Load the vector store (optional, to demonstrate loading) vector_store = load_vector_store(index_path, metadata_path) # Query the vector store user_query = "Enter your query here" # Replace with the user's query top_k = 5 # Replace with desired number of results results = query_vector_store(vector_store, user_query, k=top_k) # Display results for i, result in enumerate(results): print(f"Result {i + 1}:") print(f"Content:\n{result['content']}\n") print(f"Metadata:\n{result['metadata']}\n") print("=" * 50) if __name__ == "__main__": main()Notes on the Implementation
-
Logging and Error Handling: Comprehensive logging and error handling are implemented to ensure any issues are caught gracefully.
-
Type Hints: All functions include type hints for clarity and maintainability.
-
Performance Considerations:
-
ThreadPoolExecutor is used in
process_filesto process files concurrently, improving I/O-bound operation performance. -
ProcessPoolExecutor can be used for CPU-bound tasks if needed (e.g., in text processing or embeddings), though care must be taken with large objects due to inter-process communication overhead.
-
-
Scalability: The code is designed to handle large datasets, with progress bars provided by
tqdmto monitor processing. -
Embedding Model: The
thenlper/gte-largemodel is used for embeddings. ReplaceEMBEDDING_MODEL_NAMEwith any model suitable for your needs. -
FAISS Index: The FAISS vector store provides efficient similarity search capabilities, essential for handling large-scale vector data.
-
Text Splitting: Documents are split into chunks to manage memory usage and improve retrieval relevance.
-
NLTK Stopwords and Tokenization: NLTK resources are prepared if further text processing is needed (e.g., stopword removal, tokenization).
Requirements
Ensure that you have the following packages installed:
pip install nltk pandas PyMuPDF python-docx python-pptx tqdm transformers langchain datasetsAdditionally, install any other dependencies required by the imported modules.
Adjustments
-
File Paths: Update
/path/to/user_folderand/path/to/output_txt_folderwith the actual paths to your input and output directories. -
Embedding Device: If you have a GPU available, you can set
model_kwargs={'device': 'cuda'}when initializingHuggingFaceEmbeddingsto utilize GPU acceleration. -
Custom Separators: Adjust the
separatorsinRecursiveCharacterTextSplitterif needed to better suit your data's structure. -
Chunk Size and Overlap: Modify
CHUNK_SIZEandCHUNK_OVERLAPto fine-tune how the documents are split.
By following these steps, the code provides a robust, scalable solution that adheres to best practices and your specified requirements.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.