Skip to content
TheChatArchiveThe Chat Archive
Metallama-4-maverick-03-26-experimental· MetaHugging Face LMArena Human Preference 140K

Make it handle newlines better, it should look the way it looks on the webpage

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwen-max-2025-01-25’s answer.
  1. User

    Make it handle newlines better, it should look the way it looks on the webpage

  2. Meta Assistant

    Below is an updated Python script that improves newline handling to better preserve the visual structure of the webpage content. It uses additional techniques to:

    1. Collapse multiple newlines into single newlines (like browsers do).
    2. Preserve meaningful spacing (e.g., paragraphs, lists, headings).
    3. Remove excessive whitespace while keeping the natural flow.

    We'll enhance the script by:

    • Using prettify() to structure the HTML content before extracting text.
    • Applying regex to clean up redundant newlines and spaces.
    • Handling <pre>, <code>, and other tags where whitespace matters.

    Here's the updated code:

    import os
    from bs4 import BeautifulSoup
    import re
    
    # Define folder paths
    input_folder = "justify"
    output_folder = "clean"
    
    # Create the output folder if it doesn't exist.
    if not os.path.exists(output_folder):
        os.makedirs(output_folder)
    
    # Function to clean up text (normalize newlines and spaces)
    def normalize_text(text):
        # Replace multiple newlines with a single newline
        text = re.sub(r'\n{3,}', '\n\n', text)
        # Replace multiple spaces with a single space
        text = re.sub(r' {2,}', ' ', text)
        # Remove spaces before newlines
        text = re.sub(r' +\n', '\n', text)
        # Remove leading/trailing whitespace
        text = text.strip()
        return text
    
    # Loop over every file in the input folder.
    for filename in os.listdir(input_folder):
        # Process only .aspx files.
        if filename.lower().endswith(".aspx"):
            input_path = os.path.join(input_folder, filename)
            
            with open(input_path, "r", encoding="utf-8") as file:
                html_content = file.read()
    
            # Parse the HTML content with BeautifulSoup.
            soup = BeautifulSoup(html_content, "html.parser")
            
            # Extract text while preserving structure (use prettify for better spacing).
            text_content = soup.prettify()
            
            # Now get the text content (newline-separated).
            text_content = soup.get_text(separator="\n")
            
            # Normalize the text (fix newlines, spaces).
            text_content = normalize_text(text_content)
            
            # Define the output .txt filename.
            base_name = os.path.splitext(filename)[0]
            output_filename = base_name + ".txt"
            output_path = os.path.join(output_folder, output_filename)
            
            # Write the cleaned text to the new file.
            with open(output_path, "w", encoding="utf-8") as out_file:
                out_file.write(text_content)
            
            print(f"Converted '{filename}' to '{output_filename}'.")
    

    Key Improvements:

    1. normalize_text() function:

      • Collapses multiple newlines (\n\n\n → \n\n) to avoid large gaps.
      • Removes excessive spaces ("Hello World" → "Hello World").
      • Trims trailing spaces before newlines (e.g., "text \n" → "text\n").
    2. soup.prettify() (optional but useful):

      • Adds proper indentation to HTML before extraction (helps in debugging if needed).
    3. get_text(separator="\n"):

      • Ensures block-level elements (like <p>, <div>, <li>) are separated by newlines.
    4. Regex Cleanup:

      • Fixes over-spacing and redundant line breaks, making the output readable like a rendered webpage.

    Example Output Improvements

    Before (raw extraction):

    Title
     
    
     
    
    Hello world this is a test 
    
     
    
     
    
    Some more text here.
    

    After (normalized):

    Title
    
    Hello world this is a test
    
    Some more text here.
    

    This version should give you plaintext files that resemble the visual layout of the original webpages, with:

    ✅ Proper paragraphs
    ✅ No random gaps
    ✅ Clean spacing

    Run the script in the folder containing justify, and check the clean folder for the results! Let me know if further tweaks are needed. 😊

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.