Skip to content
TheChatArchiveThe Chat Archive
Mistralmistral-medium-2505· Mistral AIHugging Face LMArena Human Preference 140K

Write the script in Python that reads all the files "htm" in the directory…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Coherecommand-a-03-2025’s answer.
  1. User

    Write the script in Python that reads all the files "htm" in the directory "D:\Progr\Auto", extracts the fivegrams from them and writes the unique fivegrams and their count (if exceeds 5) to the respective cells "fivegram" and "count" of the table "fivegrams" in the newly created SQLite database "fivegrams.db".

  2. Mistral Assistant

    Python Script to Extract Fivegrams from HTML Files and Store in SQLite Database

    Here's a Python script that will:

    1. Read all .htm files from the specified directory
    2. Extract fivegrams (sequences of 5 words)
    3. Count occurrences of each fivegram
    4. Store unique fivegrams with counts > 5 in an SQLite database
    import os
    import re
    import sqlite3
    from collections import defaultdict
    
    def extract_text_from_html(html_content):
        """Extract text from HTML content by removing tags."""
        # Remove script and style elements
        clean_content = re.sub(r'<(script|style).*?>.*?</\1>', '', html_content, flags=re.DOTALL | re.IGNORECASE)
        # Remove all HTML tags
        clean_content = re.sub(r'<.*?>', ' ', clean_content)
        # Replace multiple whitespace characters with single space
        clean_content = re.sub(r'\s+', ' ', clean_content)
        return clean_content.strip()
    
    def extract_fivegrams(text):
        """Extract fivegrams from text."""
        # Tokenize into words (split on whitespace and filter out empty strings)
        words = [word.lower() for word in re.split(r'\s+', text) if word]
        fivegrams = []
    
        # Generate fivegrams
        for i in range(len(words) - 4):
            fivegram = ' '.join(words[i:i+5])
            fivegrams.append(fivegram)
    
        return fivegrams
    
    def process_html_files(directory):
        """Process all .htm files in the directory and extract fivegrams."""
        fivegram_counts = defaultdict(int)
    
        for filename in os.listdir(directory):
            if filename.lower().endswith('.htm'):
                filepath = os.path.join(directory, filename)
                try:
                    with open(filepath, 'r', encoding='utf-8') as file:
                        html_content = file.read()
                        text = extract_text_from_html(html_content)
                        fivegrams = extract_fivegrams(text)
    
                        for fivegram in fivegrams:
                            fivegram_counts[fivegram] += 1
                except Exception as e:
                    print(f"Error processing {filename}: {e}")
    
        return fivegram_counts
    
    def create_database(fivegram_counts):
        """Create SQLite database and store fivegrams with count > 5."""
        # Connect to the database (creates it if it doesn't exist)
        conn = sqlite3.connect('fivegrams.db')
        cursor = conn.cursor()
    
        # Create the table
        cursor.execute('''
        CREATE TABLE IF NOT EXISTS fivegrams (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            fivegram TEXT UNIQUE,
            count INTEGER
        )
        ''')
    
        # Prepare data to insert (only fivegrams with count > 5)
        data_to_insert = [(fg, count) for fg, count in fivegram_counts.items() if count > 5]
    
        # Insert data using parameterized queries to prevent SQL injection
        cursor.executemany('''
        INSERT OR IGNORE INTO fivegrams (fivegram, count)
        VALUES (?, ?)
        ''', data_to_insert)
    
        # Commit changes and close connection
        conn.commit()
        conn.close()
    
        print(f"Database created with {len(data_to_insert)} fivegrams (count > 5)")
    
    def main():
        directory = r'D:\Progr\Auto'
    
        if not os.path.exists(directory):
            print(f"Directory not found: {directory}")
            return
    
        print("Processing HTML files...")
        fivegram_counts = process_html_files(directory)
    
        print("Creating database...")
        create_database(fivegram_counts)
    
        print("Done!")
    
    if __name__ == "__main__":
        main()
    

    How This Script Works:

    1. HTML Processing:

      • Reads all .htm files from the specified directory
      • Extracts text content by removing HTML tags
      • Cleans the text by normalizing whitespace
    2. Fivegram Extraction:

      • Splits text into words
      • Creates sequences of 5 consecutive words (fivegrams)
      • Converts words to lowercase for case-insensitive comparison
    3. Database Operations:

      • Creates a new SQLite database file fivegrams.db
      • Creates a table with columns for fivegram text and count
      • Inserts only fivegrams that appear more than 5 times
      • Uses INSERT OR IGNORE to handle potential duplicates
    4. Error Handling:

      • Handles potential file reading errors
      • Checks if the directory exists before processing

    Requirements:

    • Python 3.x
    • No additional libraries needed (uses built-in modules)

    You can run this script directly, and it will create the database file in the same directory where the script is executed.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.