Analyze some sample web pages from www.statcan.gc.ca, such as the Daily pages, the IMDB…
- User
Analyze some sample web pages from www.statcan.gc.ca, such as the Daily pages, the IMDB pages, the NDM products page, the Census pages, use wget to download those pages into a folder in the local file system (the environment is cloud ubuntu linux vm with 64g of ram and 500g of space in the /mnt/ path) and use python codes to parse the saved files as sample pages for the html page, the pdf page, and the tables page, and use python codes to process them and index them into elasticsearch embeddings for AI RAG search that accept natural language queries such as "what is the population of Ottawa in 2025 and what is the estimate of population for Ottawa area in 2026". The fields information to put to the embeddings include the title, the description, the subject, the release date, the keywords (most of the information comes from the meta data section of the html page, such as dcterms, if not, try to get it from the body or contents part of the page ), the body content part which is properly chunked using different strategies such as the tags p, h1, h2, h3, list, div, table, tr, span etc, and if there is no such tags there use paragraphs with carriage returns to chunk them or use sentence boundary punctuations. The best chunks should be between 180 to 500 characters long except for those that cannot be chunked properly, use the 400 character chunk strategy. The python codes should be able to search the embeddings index, and give the response results in natural language, and also give reference to the source url, title, description, release_date. The response and results should be based only on the statcan contents, not the general answers from the llm. The llm used for inference and response should be open source llm easily downloadable without a key or access token, or passwords. In setting up the elasticsearch, try to make it not using the authentication, i.e. without a password to use it. Give me the scripts for the set up of the virtual environment for python, the requirements.txt file, the wget script file, the python codes where the open source llm is used, and also how to download and setup elasticsearch and the llm.
Assistant
This is a comprehensive request that covers data acquisition, parsing, chunking, embedding, indexing, and retrieval augmented generation (RAG) using open-source tools. Let's break it down step by step.
We will set up:
- Virtual Environment: For Python dependencies.
requirements.txt: List of Python packages.wgetScript: To download sample pages.- Elasticsearch: For indexing and vector search.
- Ollama & LLM: For local LLM inference.
- Python Scripts:
config.py: Configuration details.es_manager.py: Elasticsearch client setup and indexing.data_parser.py: Functions to parse HTML, PDF, and extract tables.chunker.py: Logic for intelligent text chunking.main_processor.py: Orchestrates data download, parsing, chunking, and indexing.rag_query.py: Performs RAG queries against the indexed data.
1. Initial Setup: Environment Preparation
Assuming you have a fresh Ubuntu VM.
1.1 Install System Dependencies
sudo apt update sudo apt install -y python3-pip python3-venv curl wget1.2 Create Python Virtual Environment
cd /mnt/ mkdir statcan_rag cd statcan_rag python3 -m venv venv_statcan_rag source venv_statcan_rag/bin/activate1.3
requirements.txtCreate a file named
requirements.txtinside/mnt/statcan_rag/:beautifulsoup4==4.12.3 lxml==5.1.0 pypdf==4.2.0 html2text==2020.1.16 pandas==2.2.2 numpy==1.26.4 sentence-transformers==2.7.0 elasticsearch==8.13.0 requests==2.32.3 nltk==3.8.1 tqdm==4.66.41.4 Install Python Dependencies
pip install -r requirements.txt # Download NLTK punkt tokenizer for sentence splitting python -c "import nltk; nltk.download('punkt')"
2. Elasticsearch Setup
We'll set up Elasticsearch without authentication for simplicity as requested, but BE AWARE: This is HIGHLY INSECURE for production environments. For production, you must enable and configure security.
2.1 Download and Extract Elasticsearch
cd /mnt/statcan_rag/ # Choose a stable version, e.g., 8.11.3 or 8.13.0 curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.13.0-linux-x86_64.tar.gz tar -xzf elasticsearch-8.13.0-linux-x86_64.tar.gz mv elasticsearch-8.13.0 elasticsearch2.2 Configure Elasticsearch (Disable Security)
# Open the configuration file nano /mnt/statcan_rag/elasticsearch/config/elasticsearch.ymlFind the
xpack.security.enabledline and change it tofalse. If it's commented out, uncomment it. Also, setxpack.security.enrollment.enabledtofalseandxpack.security.http.ssl.enabledtofalse.It should look like this (add if not present):
# ---------------------------------- Network ----------------------------------- # network.host: 0.0.0.0 # Listen on all interfaces http.port: 9200 # ---------------------------------- Security ---------------------------------- # xpack.security.enabled: false xpack.security.enrollment.enabled: false xpack.security.http.ssl.enabled: falseSave and exit (
Ctrl+X,Y,Enter).2.3 Start Elasticsearch
cd /mnt/statcan_rag/elasticsearch/ ./bin/elasticsearch &This will start Elasticsearch in the background. It might take a minute or two to fully initialize. You can check its status by visiting
http://localhost:9200in your browser or usingcurl http://localhost:9200from the VM.
3. Ollama & LLM Setup
Ollama makes it incredibly easy to run open-source LLMs locally.
3.1 Install Ollama
curl -fsSL https://ollama.com/install.sh | sh3.2 Download an LLM Model
We'll use
llama2as it's a good general-purpose open-source model.ollama run llama2This command will download the
llama2model (can be several GBs, depending on the variant) and then start an interactive chat session. Type/byeto exit the chat session once the download is complete. Ollama will automatically start a server process in the background.
4.
wgetScript for Data AcquisitionCreate a file named
download_pages.shinside/mnt/statcan_rag/:#!/bin/bash # Define the local directory to save files DATA_DIR="/mnt/statcan_rag/data" mkdir -p "$DATA_DIR" echo "Starting download of StatCan pages to $DATA_DIR..." # List of URLs to download # HTML general pages wget -P "$DATA_DIR" -r -l 1 -k -p -U "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" \ https://www.statcan.gc.ca/en/daily \ https://www.statcan.gc.ca/en/imdb \ https://www.statcan.gc.ca/en/ndm \ https://www.statcan.gc.ca/en/census \ https://www.statcan.gc.ca/en/topics/population-demography # Example HTML page with potential tables or more detailed content wget -P "$DATA_DIR" -r -l 1 -k -p -U "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" \ https://www150.statcan.gc.ca/n1/pub/71-607-x/71-607-x2023005-eng.htm # Example PDF document (Population Estimates) wget -P "$DATA_DIR" -U "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" \ https://www150.statcan.gc.ca/n1/pub/11-627-m/11-627-m2023001-eng.pdf echo "Download complete."Make the script executable and run it:
chmod +x /mnt/statcan_rag/download_pages.sh /mnt/statcan_rag/download_pages.shThis will create a
datadirectory insidestatcan_ragand download the specified pages. The-r -l 1options download the page and its immediate linked pages (one level deep) which might include CSS, images, and other HTML files. For PDFs, it downloads the PDF directly.
5. Python Codes for Parsing, Indexing, and RAG
Create these files in
/mnt/statcan_rag/.5.1
config.pyimport os # Base directory for the project BASE_DIR = "/mnt/statcan_rag" # Directory where downloaded data is stored DATA_DIR = os.path.join(BASE_DIR, "data") # Elasticsearch settings ELASTICSEARCH_HOST = "localhost" ELASTICSEARCH_PORT = 9200 ELASTICSEARCH_INDEX = "statcan_docs" # Ollama LLM settings OLLAMA_API_BASE_URL = "http://localhost:11434" OLLAMA_MODEL = "llama2" # or "mistral", "gemma", etc. # Embedding model settings EMBEDDING_MODEL_NAME = "all-MiniLM-L6-v2" # Good balance of size/performance # Chunking strategy CHUNK_SIZE = 400 CHUNK_OVERLAP = 50 # Not strictly used with current tag-based, but good practice MIN_CHUNK_LENGTH = 180 MAX_CHUNK_LENGTH = 5005.2
es_manager.pyimport os from elasticsearch import Elasticsearch, BadRequestError from sentence_transformers import SentenceTransformer from tqdm import tqdm import logging from config import ELASTICSEARCH_HOST, ELASTICSEARCH_PORT, ELASTICSEARCH_INDEX, EMBEDDING_MODEL_NAME logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') class ESManager: def __init__(self): self.es = Elasticsearch( hosts=[{"host": ELASTICSEARCH_HOST, "port": ELASTICSEARCH_PORT, "scheme": "http"}], timeout=30, max_retries=10, retry_on_timeout=True ) self.embedding_model = SentenceTransformer(EMBEDDING_MODEL_NAME) logging.info(f"Initialized ESManager. Connected to Elasticsearch: {self.es.ping()}") def create_index(self, index_name=ELASTICSEARCH_INDEX): """Creates the Elasticsearch index with the specified mapping for RAG.""" if self.es.indices.exists(index=index_name): logging.info(f"Index '{index_name}' already exists. Deleting and recreating.") self.es.indices.delete(index=index_name, ignore=[400, 404]) logging.info(f"Index '{index_name}' deleted.") # Determine embedding dimension sample_embedding = self.embedding_model.encode("test").shape[0] settings = { "analysis": { "analyzer": { "standard_analyzer": { "type": "standard" } } } } mappings = { "properties": { "content": {"type": "text", "analyzer": "standard_analyzer"}, "content_vector": {"type": "dense_vector", "dims": sample_embedding}, "url": {"type": "keyword"}, "title": {"type": "text"}, "description": {"type": "text"}, "subject": {"type": "keyword"}, "release_date": {"type": "date", "format": "yyyy-MM-dd||yyyy-MM||yyyy||date_optional_time"}, "keywords": {"type": "keyword"}, "chunk_id": {"type": "keyword"}, "original_filename": {"type": "keyword"} } } try: self.es.indices.create(index=index_name, settings=settings, mappings=mappings) logging.info(f"Index '{index_name}' created successfully with {sample_embedding} dimensions.") except BadRequestError as e: logging.error(f"Error creating index '{index_name}': {e.info}") raise def add_documents(self, documents, index_name=ELASTICSEARCH_INDEX): """Adds a list of documents (chunks) to the Elasticsearch index.""" if not documents: logging.warning("No documents to add.") return logging.info(f"Indexing {len(documents)} document chunks into '{index_name}'...") operations = [] for i, doc in enumerate(tqdm(documents, desc="Generating embeddings and preparing for bulk indexing")): content = doc.get("content", "") if not content.strip(): continue # Skip empty chunks # Generate embedding try: content_vector = self.embedding_model.encode(content).tolist() except Exception as e: logging.error(f"Error generating embedding for document {i}: {e}") continue operations.append({ "index": { "_index": index_name, "_id": f"{doc['original_filename']}_{doc['chunk_id']}" # Unique ID per chunk } }) operations.append({ "content": content, "content_vector": content_vector, "url": doc.get("url"), "title": doc.get("title"), "description": doc.get("description"), "subject": doc.get("subject"), "release_date": doc.get("release_date"), "keywords": doc.get("keywords"), "chunk_id": doc.get("chunk_id"), "original_filename": doc.get("original_filename") }) try: res = self.es.bulk(operations=operations, request_timeout=60) if res["errors"]: logging.error(f"Bulk indexing completed with errors: {res['errors']}") else: logging.info(f"Successfully indexed {len(documents)} document chunks.") except Exception as e: logging.error(f"Error during bulk indexing: {e}") def search_documents(self, query_text, index_name=ELASTICSEARCH_INDEX, k=5): """ Searches the Elasticsearch index for relevant documents using vector similarity. Combines vector search (kNN) with keyword search for better recall. """ query_vector = self.embedding_model.encode(query_text).tolist() search_body = { "query": { "bool": { "must": [ { "match": { "content": { "query": query_text, "analyzer": "standard_analyzer" # Use standard analyzer for keyword matching } } } ], "should": [ { "knn": { "field": "content_vector", "query_vector": query_vector, "k": k, "num_candidates": 100 # Adjust based on data size } } ], "minimum_should_match": 1 # At least one of must or should conditions must match } }, "_source": ["content", "url", "title", "description", "release_date"], "size": k } try: response = self.es.search(index=index_name, body=search_body) hits = response["hits"]["hits"] results = [] for hit in hits: results.append({ "score": hit["_score"], "content": hit["_source"]["content"], "url": hit["_source"].get("url"), "title": hit["_source"].get("title"), "description": hit["_source"].get("description"), "release_date": hit["_source"].get("release_date") }) logging.info(f"Found {len(results)} relevant documents.") return results except Exception as e: logging.error(f"Error during Elasticsearch search: {e}") return [] if __name__ == "__main__": # This block is for testing ES connection and index creation es_manager = ESManager() if es_manager.es.ping(): print("Successfully connected to Elasticsearch.") es_manager.create_index() print(f"Index '{ELASTICSEARCH_INDEX}' created/recreated.") else: print("Could not connect to Elasticsearch. Make sure it's running.")5.3
data_parser.pyimport os import re from bs4 import BeautifulSoup from pypdf import PdfReader import html2text import logging from datetime import datetime logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') def parse_html(file_path): """Parses an HTML file to extract metadata and main content.""" try: with open(file_path, 'r', encoding='utf-8') as f: html_content = f.read() soup = BeautifulSoup(html_content, 'lxml') # Extract Metadata title = soup.find('title').get_text(strip=True) if soup.find('title') else os.path.basename(file_path) description = soup.find('meta', attrs={'name': 'description'}) description = description.get('content', '').strip() if description else '' keywords = soup.find('meta', attrs={'name': 'keywords'}) keywords = [k.strip() for k in keywords.get('content', '').split(',') if k.strip()] if keywords else [] subject = soup.find('meta', attrs={'property': 'dcterms.subject'}) subject = subject.get('content', '').strip() if subject else '' release_date = soup.find('meta', attrs={'property': 'dcterms.issued'}) release_date = release_date.get('content', '').strip() if release_date else '' # Try to parse and format date if release_date: try: # Handle various date formats (e.g., YYYY-MM-DD, YYYY-MM, YYYY) if len(release_date) == 10: # YYYY-MM-DD release_date = datetime.strptime(release_date, '%Y-%m-%d').strftime('%Y-%m-%d') elif len(release_date) == 7: # YYYY-MM release_date = datetime.strptime(release_date, '%Y-%m').strftime('%Y-%m-01') # Default to 1st of month elif len(release_date) == 4: # YYYY release_date = datetime.strptime(release_date, '%Y').strftime('%Y-01-01') # Default to 1st Jan else: release_date = '' # Fallback if format is unexpected except ValueError: release_date = '' # Invalid date format # Extract URL from a canonical link or construct from filename/directory url_tag = soup.find('link', rel='canonical') url = url_tag.get('href', '').strip() if url_tag else '' if not url and 'statcan.gc.ca' in file_path: # Basic attempt to reconstruct URL if canonical is missing # This is tricky without knowing wget's exact mirroring structure # For a robust solution, you'd need the original URL list from wget # For now, let's just indicate the original file name url = os.path.basename(file_path) # Placeholder # Extract main content main_content_div = soup.find('main') # Common for StatCan if not main_content_div: main_content_div = soup.find('body') # Fallback if main_content_div: # Convert HTML to markdown/text for cleaner content extraction h = html2text.HTML2Text() h.ignore_links = True h.ignore_images = True h.body_width = 0 # No wrapping main_content = h.handle(str(main_content_div)) # Clean up extra newlines/spaces main_content = re.sub(r'\n\s*\n', '\n\n', main_content).strip() else: main_content = '' return { "title": title, "description": description, "subject": subject, "release_date": release_date, "keywords": keywords, "content": main_content, "url": url, "original_filename": os.path.basename(file_path) } except Exception as e: logging.error(f"Error parsing HTML file {file_path}: {e}") return None def parse_pdf(file_path): """Parses a PDF file to extract text content.""" try: reader = PdfReader(file_path) text_content = "" for page in reader.pages: text_content += page.extract_text() + "\n" # Simple cleanup text_content = re.sub(r'\s*\n\s*', '\n', text_content).strip() text_content = re.sub(r'\n{3,}', '\n\n', text_content).strip() # For PDFs, metadata is often harder to extract programmatically from headers. # Fallback to filename or general description. title = reader.metadata.get('/Title', os.path.basename(file_path)).replace('\x00', '') author = reader.metadata.get('/Author', '').replace('\x00', '') creation_date = reader.metadata.get('/CreationDate', '') # Basic date parsing from PDF metadata string (e.g., D:20230501103000Z) parsed_date = '' if creation_date and creation_date.startswith('D:'): try: date_str = creation_date[2:10] # YYYYMMDD parsed_date = datetime.strptime(date_str, '%Y%m%d').strftime('%Y-%m-%d') except ValueError: pass # Ignore if date format is unexpected return { "title": title, "description": f"PDF document by {author}" if author else "PDF document", "subject": "PDF document", "release_date": parsed_date, "keywords": [], # Can't reliably extract from PDF "content": text_content, "url": os.path.basename(file_path), # Will be original URL if it was a direct download "original_filename": os.path.basename(file_path) } except Exception as e: logging.error(f"Error parsing PDF file {file_path}: {e}") return None def extract_tables_from_html(soup): """ Extracts tables from BeautifulSoup object and converts them to a readable text format. This is a helper function to get text representation of tables for RAG. It doesn't parse into structured dataframes. """ tables_content = [] for i, table in enumerate(soup.find_all('table')): table_text = f"Table {i+1}:\n" headers = [th.get_text(strip=True) for th in table.find_all('th')] if headers: table_text += "| " + " | ".join(headers) + " |\n" table_text += "|-" + "-|-".join(['-' * len(h) for h in headers]) + "-|\n" for row in table.find_all('tr'): cols = [td.get_text(strip=True) for td in row.find_all('td')] if cols: table_text += "| " + " | ".join(cols) + " |\n" tables_content.append(table_text) return "\n\n".join(tables_content)5.4
chunker.pyimport re import nltk from config import MIN_CHUNK_LENGTH, MAX_CHUNK_LENGTH, CHUNK_SIZE class TextChunker: def __init__(self, min_len=MIN_CHUNK_LENGTH, max_len=MAX_CHUNK_LENGTH, target_len=CHUNK_SIZE): self.min_len = min_len self.max_len = max_len self.target_len = target_len def _split_into_paragraphs(self, text): """Splits text into paragraphs by multiple newlines.""" return [p.strip() for p in re.split(r'\n{2,}', text) if p.strip()] def _split_into_sentences(self, text): """Splits text into sentences using NLTK.""" return nltk.sent_tokenize(text) def chunk_text(self, text, metadata): """ Chunks text based on various strategies, prioritizing semantic units. Each chunk will inherit the metadata. """ chunks = [] current_chunk_id = 0 # Try paragraph-based chunking first paragraphs = self._split_into_paragraphs(text) temp_chunks = [] for para in paragraphs: if len(para) > self.max_len: # If paragraph is too long, split by sentences sentences = self._split_into_sentences(para) combined_sentence_chunk = "" for sentence in sentences: if len(combined_sentence_chunk) + len(sentence) + 1 <= self.max_len: combined_sentence_chunk += (sentence + " ").strip() else: if combined_sentence_chunk: temp_chunks.append(combined_sentence_chunk) combined_sentence_chunk = (sentence + " ").strip() if combined_sentence_chunk: temp_chunks.append(combined_sentence_chunk) else: temp_chunks.append(para) # Further combine or split to meet length targets final_chunks = [] combined_chunk = "" for tc in temp_chunks: if len(tc) >= self.min_len and len(tc) <= self.max_len: if combined_chunk: # Add any pending combined_chunk final_chunks.append(combined_chunk.strip()) combined_chunk = "" final_chunks.append(tc) elif len(tc) < self.min_len: if len(combined_chunk) + len(tc) + 1 <= self.target_len: combined_chunk += (tc + " ").strip() else: if combined_chunk: final_chunks.append(combined_chunk.strip()) combined_chunk = (tc + " ").strip() elif len(tc) > self.max_len: # Fallback for very long chunks (e.g., after sentence split) # Simple fixed-size split with overlap start = 0 while start < len(tc): end = min(start + self.target_len, len(tc)) sub_chunk = tc[start:end].strip() if sub_chunk: final_chunks.append(sub_chunk) start += self.target_len # No explicit overlap for simplicity here, can be added if combined_chunk: # Add any remaining combined_chunk final_chunks.append(combined_chunk.strip()) # Clean up and add metadata for chunk_content in final_chunks: if chunk_content.strip(): chunk_metadata = metadata.copy() chunk_metadata["content"] = chunk_content.strip() chunk_metadata["chunk_id"] = current_chunk_id chunks.append(chunk_metadata) current_chunk_id += 1 return chunks5.5
main_processor.pyimport os import glob import logging from tqdm import tqdm from config import DATA_DIR, ELASTICSEARCH_INDEX from data_parser import parse_html, parse_pdf from chunker import TextChunker from es_manager import ESManager logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') def process_and_index_documents(): """ Orchestrates the parsing, chunking, and indexing of documents. """ es_manager = ESManager() es_manager.create_index(ELASTICSEARCH_INDEX) chunker = TextChunker() all_chunks = [] # Process HTML files html_files = glob.glob(os.path.join(DATA_DIR, "**", "*.html"), recursive=True) html_files.extend(glob.glob(os.path.join(DATA_DIR, "**", "*.htm"), recursive=True)) logging.info(f"Found {len(html_files)} HTML files to process.") for file_path in tqdm(html_files, desc="Processing HTML files"): parsed_data = parse_html(file_path) if parsed_data and parsed_data["content"].strip(): chunks = chunker.chunk_text(parsed_data["content"], { "url": parsed_data["url"], "title": parsed_data["title"], "description": parsed_data["description"], "subject": parsed_data["subject"], "release_date": parsed_data["release_date"], "keywords": parsed_data["keywords"], "original_filename": parsed_data["original_filename"] }) all_chunks.extend(chunks) # Process PDF files pdf_files = glob.glob(os.path.join(DATA_DIR, "**", "*.pdf"), recursive=True) logging.info(f"Found {len(pdf_files)} PDF files to process.") for file_path in tqdm(pdf_files, desc="Processing PDF files"): parsed_data = parse_pdf(file_path) if parsed_data and parsed_data["content"].strip(): chunks = chunker.chunk_text(parsed_data["content"], { "url": parsed_data["url"], "title": parsed_data["title"], "description": parsed_data["description"], "subject": parsed_data["subject"], "release_date": parsed_data["release_date"], "keywords": parsed_data["keywords"], "original_filename": parsed_data["original_filename"] }) all_chunks.extend(chunks) logging.info(f"Total chunks prepared for indexing: {len(all_chunks)}") if all_chunks: es_manager.add_documents(all_chunks, ELASTICSEARCH_INDEX) else: logging.warning("No content chunks generated. Check data directory and parser output.") if __name__ == "__main__": logging.info("Starting document processing and indexing...") process_and_index_documents() logging.info("Document processing and indexing complete.")5.6
rag_query.pyimport requests import json import logging from es_manager import ESManager from config import OLLAMA_API_BASE_URL, OLLAMA_MODEL, ELASTICSEARCH_INDEX logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s') def get_llm_response(prompt): """Sends a prompt to the local Ollama LLM and returns the response.""" url = f"{OLLAMA_API_BASE_URL}/api/generate" headers = {"Content-Type": "application/json"} data = { "model": OLLAMA_MODEL, "prompt": prompt, "stream": False # Set to False for single response } try: response = requests.post(url, headers=headers, data=json.dumps(data)) response.raise_for_status() # Raise an exception for bad status codes return response.json()["response"] except requests.exceptions.RequestException as e: logging.error(f"Error connecting to Ollama: {e}") logging.error("Ensure Ollama is running and the specified model is downloaded.") return "Error: Could not get response from LLM." except KeyError: logging.error(f"Unexpected response format from Ollama: {response.json()}") return "Error: Unexpected LLM response format." def rag_search(query): """ Performs RAG search: retrieves relevant chunks from ES and generates a response using the LLM based on those chunks. """ es_manager = ESManager() # 1. Retrieve relevant documents/chunks from Elasticsearch retrieved_docs = es_manager.search_documents(query, index_name=ELASTICSEARCH_INDEX, k=5) # Retrieve top 5 if not retrieved_docs: return "No relevant information found in the StatCan content for your query.", [] # 2. Prepare context for the LLM context_parts = [] source_references = [] for i, doc in enumerate(retrieved_docs): context_parts.append(f"--- Document Source {i+1} ---\nTitle: {doc.get('title', 'N/A')}\nURL: {doc.get('url', 'N/A')}\nDescription: {doc.get('description', 'N/A')}\nRelease Date: {doc.get('release_date', 'N/A')}\nContent: {doc['content']}\n--- End Source {i+1} ---") source_references.append({ "Source": i+1, "Title": doc.get("title", "N/A"), "URL": doc.get("url", "N/A"), "Description": doc.get("description", "N/A"), "Release Date": doc.get("release_date", "N/A") }) context = "\n\n".join(context_parts) # 3. Formulate the prompt for the LLM prompt = f""" You are an AI assistant specialized in Canadian statistics, specifically using data from StatCan. Answer the following question based ONLY on the provided context from StatCan. If the information is not explicitly available in the context, state that you cannot answer based on the provided information. Do not use any external knowledge. Question: {query} Context: {context} Response (including referring to source document numbers if possible, e.g., "(Source 1)"): """ logging.info("Sending prompt to LLM...") # 4. Get response from LLM llm_response = get_llm_response(prompt) return llm_response, source_references if __name__ == "__main__": print("Welcome to the StatCan RAG Search System!") print("Type your query or 'exit' to quit.") while True: user_query = input("\nEnter your query: ") if user_query.lower() == 'exit': break response, references = rag_search(user_query) print("\n--- AI Response ---") print(response) if references: print("\n--- Sources ---") for ref in references: print(f"Source {ref['Source']}:") print(f" Title: {ref['Title']}") print(f" URL: {ref['URL']}") print(f" Description: {ref['Description']}") print(f" Release Date: {ref['Release Date']}") print("-" * 20) else: print("\nNo specific sources found for this query.")
6. Execution Steps
Make sure you are in the
statcan_ragdirectory:/mnt/statcan_rag/.6.1 Activate Virtual Environment
source venv_statcan_rag/bin/activate6.2 Start Elasticsearch (if not already running)
cd /mnt/statcan_rag/elasticsearch/ ./bin/elasticsearch &Wait a minute for it to start. You can check
curl http://localhost:9200.6.3 Start Ollama (if not already running)
ollama run llama2 # Wait for the model to download if it hasn't, then type /bye to exit chat. # Ollama server will continue running in background.You can check
curl http://localhost:11434.6.4 Download Sample Pages
cd /mnt/statcan_rag/ chmod +x download_pages.sh ./download_pages.sh6.5 Process and Index Documents into Elasticsearch
python main_processor.pyThis script will parse all downloaded HTML and PDF files, chunk their content, generate embeddings, and index them into Elasticsearch. This step might take some time depending on the number and size of files.
6.6 Run the RAG Query System
python rag_query.pyNow, you can type your natural language queries, such as:
what is the population of Ottawa in 2025 and what is the estimate of population for Ottawa area in 2026What is the purpose of the IMDB on StatCan?Summarize the key findings from the latest population estimates.
The system will search Elasticsearch, retrieve relevant chunks, send them to the local
llama2LLM, and provide a natural language answer with references.
Important Considerations and Limitations:
- Elasticsearch Security: As explicitly noted, disabling security is for demonstration only. For any real-world use, configure proper authentication (e.g., API keys, username/password) and TLS/SSL.
wgetData Mirroring:wget -r -l 1might download a complex directory structure for mirrored sites. Thedata_parser.pyattempts to reconstruct URLs but for precise original URLs, you'd need a more sophisticated scraper that logs original URLs. For this example,urlfield might sometimes contain the local file path or a generic placeholder.- HTML Parsing Robustness: Web pages are dynamic.
BeautifulSoupis good, but complex JavaScript-rendered content or highly inconsistent HTML structures can be challenging. The current parser targets common StatCan patterns (<main>,metatags). - Table Parsing: The
extract_tables_from_htmlindata_parser.pysimply converts tables to a Markdown-like text representation. For deeper analysis or structured querying of tables, you'd need more advanced table parsing libraries (e.g.,camelot-pyfor PDFs orpandas.read_htmlfor very clean HTML tables) and different indexing strategies. For RAG, a textual representation is often sufficient for the LLM to understand. - Chunking Strategy: The
TextChunkeraims for semantic chunks (paragraphs, sentences) and then adjusts for length. Perfect chunking is an active research area. The400 charactertarget is a heuristic. Very long documents might require hierarchical chunking or more sophisticated summarization before indexing. - LLM Choice:
llama2is a good starting point. You can experiment with other Ollama models likemistral,gemma,phi3etc., which might offer different performance or quality trade-offs. Remember toollama run <model_name>first. - RAG Prompt Engineering: The prompt used in
rag_query.pyis crucial. Refining it can significantly improve the quality and relevance of the LLM's responses, especially in adhering to "only use provided context." - Scalability: For large datasets, consider distributed Elasticsearch clusters, more powerful embedding models, and potentially fine-tuning an LLM.
This setup provides a robust foundation for building an AI RAG system with StatCan data on your Ubuntu VM.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.