Write the script in Python that reads all the files "htm" in the directory…
- User
Write the script in Python that reads all the files "htm" in the directory "D:\Progr\Auto", extracts the fivegrams from them and writes the unique fivegrams and their count (if exceeds 5) to the respective cells "fivegram" and "count" of the table "fivegrams" in the newly created SQLite database "fivegrams.db".
Assistant
Python Script to Extract Fivegrams from HTML Files and Store in SQLite Database
Here's a Python script that will:
- Read all
.htmfiles from the specified directory - Extract fivegrams (sequences of 5 words)
- Count occurrences of each fivegram
- Store unique fivegrams with counts > 5 in an SQLite database
import os import re import sqlite3 from collections import defaultdict def extract_text_from_html(html_content): """Extract text from HTML content by removing tags.""" # Remove script and style elements clean_content = re.sub(r'<(script|style).*?>.*?</\1>', '', html_content, flags=re.DOTALL | re.IGNORECASE) # Remove all HTML tags clean_content = re.sub(r'<.*?>', ' ', clean_content) # Replace multiple whitespace characters with single space clean_content = re.sub(r'\s+', ' ', clean_content) return clean_content.strip() def extract_fivegrams(text): """Extract fivegrams from text.""" # Tokenize into words (split on whitespace and filter out empty strings) words = [word.lower() for word in re.split(r'\s+', text) if word] fivegrams = [] # Generate fivegrams for i in range(len(words) - 4): fivegram = ' '.join(words[i:i+5]) fivegrams.append(fivegram) return fivegrams def process_html_files(directory): """Process all .htm files in the directory and extract fivegrams.""" fivegram_counts = defaultdict(int) for filename in os.listdir(directory): if filename.lower().endswith('.htm'): filepath = os.path.join(directory, filename) try: with open(filepath, 'r', encoding='utf-8') as file: html_content = file.read() text = extract_text_from_html(html_content) fivegrams = extract_fivegrams(text) for fivegram in fivegrams: fivegram_counts[fivegram] += 1 except Exception as e: print(f"Error processing {filename}: {e}") return fivegram_counts def create_database(fivegram_counts): """Create SQLite database and store fivegrams with count > 5.""" # Connect to the database (creates it if it doesn't exist) conn = sqlite3.connect('fivegrams.db') cursor = conn.cursor() # Create the table cursor.execute(''' CREATE TABLE IF NOT EXISTS fivegrams ( id INTEGER PRIMARY KEY AUTOINCREMENT, fivegram TEXT UNIQUE, count INTEGER ) ''') # Prepare data to insert (only fivegrams with count > 5) data_to_insert = [(fg, count) for fg, count in fivegram_counts.items() if count > 5] # Insert data using parameterized queries to prevent SQL injection cursor.executemany(''' INSERT OR IGNORE INTO fivegrams (fivegram, count) VALUES (?, ?) ''', data_to_insert) # Commit changes and close connection conn.commit() conn.close() print(f"Database created with {len(data_to_insert)} fivegrams (count > 5)") def main(): directory = r'D:\Progr\Auto' if not os.path.exists(directory): print(f"Directory not found: {directory}") return print("Processing HTML files...") fivegram_counts = process_html_files(directory) print("Creating database...") create_database(fivegram_counts) print("Done!") if __name__ == "__main__": main()How This Script Works:
-
HTML Processing:
- Reads all
.htmfiles from the specified directory - Extracts text content by removing HTML tags
- Cleans the text by normalizing whitespace
- Reads all
-
Fivegram Extraction:
- Splits text into words
- Creates sequences of 5 consecutive words (fivegrams)
- Converts words to lowercase for case-insensitive comparison
-
Database Operations:
- Creates a new SQLite database file
fivegrams.db - Creates a table with columns for fivegram text and count
- Inserts only fivegrams that appear more than 5 times
- Uses
INSERT OR IGNOREto handle potential duplicates
- Creates a new SQLite database file
-
Error Handling:
- Handles potential file reading errors
- Checks if the directory exists before processing
Requirements:
- Python 3.x
- No additional libraries needed (uses built-in modules)
You can run this script directly, and it will create the database file in the same directory where the script is executed.
- Read all
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.