Write the script in Python that reads all the files "htm" in the directory…
- User
Write the script in Python that reads all the files "htm" in the directory "D:\Progr\Auto", extracts the bigrams from them and writes the unique bigrams and their count (if exceeds 10) to the respective cells "bigram" and "count" of the table "bigrams" in the SQLite database "bigrams.db".
- Assistant
Below is a compact, self-contained script that will
- walk through every .htm file in
D:\Progr\Auto\, - strip the HTML,
- build (word₁, word₂) bigrams,
- keep only those appearing more than 10 ×, and
- store / update them in the SQLite database bigrams.db (table bigrams, columns bigram and count).
#!/usr/bin/env python # -*- coding: utf-8 -*- """ Collect bigrams from every .htm file in D:\Progr\Auto and store them (in bulk) in bigrams.db -> bigrams(bigram TEXT PRIMARY KEY, count INTEGER). Only bigrams whose frequency > 10 are persisted. """ import glob import os import re import sqlite3 from collections import Counter from contextlib import closing try: # If BeautifulSoup is available we use it, else we fall back to regex stripping. from bs4 import BeautifulSoup HAVE_BS = True except ImportError: HAVE_BS = False import html # --------------------------------------------------------------------------- # # Settings # --------------------------------------------------------------------------- # SOURCE_DIR = r"D:\Progr\Auto" GLOB_MASK = os.path.join(SOURCE_DIR, "*.htm") DB_PATH = r"D:\Progr\Auto\bigrams.db" # or wherever you want it TABLE_NAME = "bigrams" MIN_COUNT = 10 # --------------------------------------------------------------------------- # # Helpers # --------------------------------------------------------------------------- # WORD_RE = re.compile(r"[a-zA-ZÀ-ÿ]+") def strip_html(text: str) -> str: """Return plain text from HTML string.""" if HAVE_BS: return BeautifulSoup(text, "lxml").get_text(" ") # Fallback: very naive – remove tags then unescape entities return html.unescape(re.sub(r"<[^>]+>", " ", text)) def tokenize(text: str): """Lower-cases and returns an iterable of alphabetic tokens.""" return WORD_RE.findall(text.lower()) def extract_bigrams(tokens): """Return generator of bigram strings 'w1 w2'.""" return (f"{w1} {w2}" for w1, w2 in zip(tokens, tokens[1:])) # --------------------------------------------------------------------------- # # 1. Read & accumulate bigram counts # --------------------------------------------------------------------------- # counter = Counter() for filepath in glob.iglob(GLOB_MASK): try: with open(filepath, "r", encoding="utf-8", errors="ignore") as f: plain = strip_html(f.read()) toks = tokenize(plain) counter.update(extract_bigrams(toks)) except Exception as exc: print(f"❗ Skipped {filepath}: {exc}") # --------------------------------------------------------------------------- # # 2. Filter & prepare for DB # --------------------------------------------------------------------------- # rows = [(bg, cnt) for bg, cnt in counter.items() if cnt > MIN_COUNT] if not rows: print("No bigrams exceeded the threshold.") raise SystemExit(0) # --------------------------------------------------------------------------- # # 3. Store in SQLite # --------------------------------------------------------------------------- # with closing(sqlite3.connect(DB_PATH)) as conn, conn: # 'conn' is its own context manager cur = conn.cursor() # Create table if it doesn't exist cur.execute(f""" CREATE TABLE IF NOT EXISTS {TABLE_NAME} ( bigram TEXT PRIMARY KEY, count INTEGER NOT NULL ) """) # Insert or update cur.executemany(f""" INSERT INTO {TABLE_NAME} (bigram, count) VALUES (?, ?) ON CONFLICT(bigram) DO UPDATE SET count = excluded.count """, rows) print(f"✅ Stored {len(rows)} bigrams (freq > {MIN_COUNT}) in {DB_PATH}.")How it works, at a glance • Every file ending with
.htmis opened, HTML tags are stripped, and simple alphabetic tokenisation is applied.
• Bigrams are counted across the whole corpus viacollections.Counter.
• Only bigrams whose overall frequency is > 10 survive the filter.
• Usingsqlite3the script creates the table bigrams if needed andINSERT … ON CONFLICThandles both new and already-seen bigrams gracefully (updating the existing count).Run it as a standalone script (
python path\to\script.py). If you prefer a different threshold, directory, or DB path, tweak the settings block on top. - walk through every .htm file in
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.