Skip to content
TheChatArchiveThe Chat Archive
Qwenqwen3-235b-a22b-no-thinking· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

Write the script in Python that reads all the files "htm" in the directory…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Googlegemma-3-27b-it’s answer.
  1. User

    Write the script in Python that reads all the files "htm" in the directory "D:\Progr\Auto", extracts the bigrams from them and writes the unique bigrams and their count (if exceeds 10) to the respective cells "bigram" and "count" of the table "bigrams" in the SQLite database "bigrams.db".

  2. Qwen Assistant

    Here's a complete Python script that:

    1. Scans the directory D:\Progr\Auto for .htm files.
    2. Reads each file and extracts bigrams (pairs of consecutive words).
    3. Counts the frequency of each bigram.
    4. Stores only the unique bigrams with counts greater than 10 into an SQLite database (bigrams.db) in a table called bigrams with columns bigram and count.

    ✅ Requirements:

    Make sure you have Python installed, and the nltk library for tokenization (optional but useful). You can install it via:

    pip install nltk
    

    📜 Python Script

    import os
    import re
    import sqlite3
    import nltk
    from nltk.tokenize import word_tokenize
    
    # Download required NLTK data (only once)
    nltk.download('punkt')
    
    # Directory containing .htm files
    directory = r'D:\Progr\Auto'
    
    # Initialize dictionary to store bigram counts
    bigram_counts = {}
    
    # Function to clean and tokenize text
    def tokenize(text):
        # Replace non-word characters and lowercase
        text = re.sub(r'[^\w\s]', '', text.lower())
        tokens = word_tokenize(text)
        return tokens
    
    # Function to extract bigrams
    def extract_bigrams(tokens):
        return [f"{tokens[i]} {tokens[i+1]}" for i in range(len(tokens) - 1)]
    
    # Loop through all .htm files in the directory
    for filename in os.listdir(directory):
        if filename.endswith(".htm"):
            filepath = os.path.join(directory, filename)
            try:
                with open(filepath, 'r', encoding='utf-8', errors='ignore') as file:
                    content = file.read()
                    tokens = tokenize(content)
                    bigrams = extract_bigrams(tokens)
    
                    for bg in bigrams:
                        bigram_counts[bg] = bigram_counts.get(bg, 0) + 1
            except Exception as e:
                print(f"Error reading {filename}: {e}")
    
    # Connect to SQLite database (or create it)
    conn = sqlite3.connect('bigrams.db')
    cursor = conn.cursor()
    
    # Create table if not exists
    cursor.execute('''
    CREATE TABLE IF NOT EXISTS bigrams (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        bigram TEXT NOT NULL UNIQUE,
        count INTEGER NOT NULL
    )
    ''')
    
    # Insert or update bigrams where count > 10
    for bigram, count in bigram_counts.items():
        if count > 10:
            cursor.execute('''
            INSERT OR IGNORE INTO bigrams (bigram, count)
            VALUES (?, ?)
            ''', (bigram, count))
            # If the bigram already exists, update its count (optional behavior)
            cursor.execute('''
            UPDATE bigrams SET count = ?
            WHERE bigram = ? AND count < ?
            ''', (count, bigram, count))
    
    # Commit and close
    conn.commit()
    conn.close()
    
    print("Bigrams extraction completed and saved to bigrams.db")
    

    📝 Notes:

    • The script uses nltk for better tokenization. You can replace it with simple splits if preferred.
    • It uses INSERT OR IGNORE to avoid duplicate entries.
    • It optionally updates existing rows if the new count is higher.
    • The regex removes punctuation; adjust this based on your needs.

    Let me know if you'd like to include phrases across HTML tags or filter specific parts of the HTML (like ignoring scripts or styles).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.