Write the script in Python that reads all the files "htm" in the directory…
- User
Write the script in Python that reads all the files "htm" in the directory "D:\Progr\Auto", extracts the bigrams from them and writes the unique bigrams and their count (if exceeds 10) to the respective cells "bigram" and "count" of the table "bigrams" in the SQLite database "bigrams.db".
Assistant
Here's a complete Python script that:
- Scans the directory
D:\Progr\Autofor.htmfiles. - Reads each file and extracts bigrams (pairs of consecutive words).
- Counts the frequency of each bigram.
- Stores only the unique bigrams with counts greater than 10 into an SQLite database (
bigrams.db) in a table calledbigramswith columnsbigramandcount.
✅ Requirements:
Make sure you have Python installed, and the
nltklibrary for tokenization (optional but useful). You can install it via:pip install nltk
📜 Python Script
import os import re import sqlite3 import nltk from nltk.tokenize import word_tokenize # Download required NLTK data (only once) nltk.download('punkt') # Directory containing .htm files directory = r'D:\Progr\Auto' # Initialize dictionary to store bigram counts bigram_counts = {} # Function to clean and tokenize text def tokenize(text): # Replace non-word characters and lowercase text = re.sub(r'[^\w\s]', '', text.lower()) tokens = word_tokenize(text) return tokens # Function to extract bigrams def extract_bigrams(tokens): return [f"{tokens[i]} {tokens[i+1]}" for i in range(len(tokens) - 1)] # Loop through all .htm files in the directory for filename in os.listdir(directory): if filename.endswith(".htm"): filepath = os.path.join(directory, filename) try: with open(filepath, 'r', encoding='utf-8', errors='ignore') as file: content = file.read() tokens = tokenize(content) bigrams = extract_bigrams(tokens) for bg in bigrams: bigram_counts[bg] = bigram_counts.get(bg, 0) + 1 except Exception as e: print(f"Error reading {filename}: {e}") # Connect to SQLite database (or create it) conn = sqlite3.connect('bigrams.db') cursor = conn.cursor() # Create table if not exists cursor.execute(''' CREATE TABLE IF NOT EXISTS bigrams ( id INTEGER PRIMARY KEY AUTOINCREMENT, bigram TEXT NOT NULL UNIQUE, count INTEGER NOT NULL ) ''') # Insert or update bigrams where count > 10 for bigram, count in bigram_counts.items(): if count > 10: cursor.execute(''' INSERT OR IGNORE INTO bigrams (bigram, count) VALUES (?, ?) ''', (bigram, count)) # If the bigram already exists, update its count (optional behavior) cursor.execute(''' UPDATE bigrams SET count = ? WHERE bigram = ? AND count < ? ''', (count, bigram, count)) # Commit and close conn.commit() conn.close() print("Bigrams extraction completed and saved to bigrams.db")
📝 Notes:
- The script uses
nltkfor better tokenization. You can replace it with simple splits if preferred. - It uses
INSERT OR IGNOREto avoid duplicate entries. - It optionally updates existing rows if the new count is higher.
- The regex removes punctuation; adjust this based on your needs.
Let me know if you'd like to include phrases across HTML tags or filter specific parts of the HTML (like ignoring scripts or styles).
- Scans the directory
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.