Skip to content
TheChatArchiveThe Chat Archive
o4-mini-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

On a webpage of a website I have a table with class="tabletable_8D7gp". Below that I…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4.1-2025-04-14’s answer.
  1. User

    On a webpage of a website I have a table with class="table_table__8D7gp". Below that I have a button with class="MuiButtonBase-root MuiButton-root MuiButton-contained MuiButton-containedSecondary MuiButton-sizeSmall MuiButton-containedSizeSmall MuiButton-colorSecondary MuiButtonGroup-grouped MuiButtonGroup-groupedHorizontal MuiButtonGroup-groupedContained MuiButtonGroup-groupedContainedHorizontal MuiButtonGroup-groupedContainedSecondary MuiButton-root MuiButton-contained MuiButton-containedSecondary MuiButton-sizeSmall MuiButton-containedSizeSmall MuiButton-colorSecondary MuiButtonGroup-middleButton css-cbvs2h" This button switches to the next part of the table.

    How can I scrape this in web browser? I want each of the table pages saved in new file.

  2. Assistant

    Here’s one way you can “click” through each page of a JS-powered table and dump its HTML into separate files. I’ll show you two approaches:

    1. Reverse-engineer the XHR/Fetch and pull the pages directly
    2. Browser-automation with Python + Selenium

    —

    1) Hit the same API the “Next” button is using

    Open your DevTools → Network → click “Next” once. Look for an XHR/fetch that returns JSON or HTML for your table. Often it will be a URL like:

    https://www.example.com/api/table?page=1&size=50
    

    If you can find that, you can skip the browser entirely and do something like:

    import requests
    
    BASE = "https://www.example.com/api/table"
    headers = {
        "User-Agent": "Mozilla/5.0",
        # … any cookies or auth headers you need …
    }
    
    for page in range(1, 11):   # or while True, break on empty result
        params = {"page": page, "size": 50}
        r = requests.get(BASE, headers=headers, params=params)
        if not r.ok or not r.text.strip():
            break
        open(f"table-page-{page}.html", "w", encoding="utf8").write(r.text)
        print("Saved page", page)
    

    If the response is JSON, you can either
    • Render it into HTML yourself, or
    • Save it as JSON (table-page-1.json)

    This is by far the most robust.

    —

    2) Drive a real browser with Selenium

    If you can’t find a nice API endpoint, you can literally click the “Next” button in a headless Chrome/Firefox with Selenium.

    a) Install prerequisites

    pip install selenium webdriver-manager
    

    b) Example script (scrape_with_selenium.py)

    from selenium import webdriver
    from selenium.webdriver.chrome.service import Service
    from webdriver_manager.chrome import ChromeDriverManager
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    import time
    
    URL = "https://www.example.com/your-page"
    driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
    
    try:
        driver.get(URL)
        wait = WebDriverWait(driver, 10)
    
        page = 1
        while True:
            # 1) Wait for your table to load
            table = wait.until(EC.presence_of_element_located(
                (By.CSS_SELECTOR, ".table_table__8D7gp")
            ))
    
            # 2) Dump table HTML
            html = table.get_attribute("outerHTML")
            with open(f"table-page-{page}.html", "w", encoding="utf8") as f:
                f.write(html)
            print(f"Saved table page {page}")
    
            # 3) Find and click the Next button
            #    Adjust this selector to match only the "Next" button
            try:
                next_btn = driver.find_element(
                    By.XPATH,
                    "//button[contains(@class,'MuiButton-containedSecondary') and normalize-space(.)='Next']"
                )
            except:
                print("No Next button found → done.")
                break
    
            # If "Next" button is disabled, stop
            if next_btn.get_attribute("aria-disabled") == "true":
                print("Next button is disabled → done.")
                break
    
            next_btn.click()
            page += 1
    
            # 4) Give React/JS a moment to re-render the table
            time.sleep(1)
    
    finally:
        driver.quit()
    

    c) Run it

    python scrape_with_selenium.py
    

    Each iteration saves table-page-1.html, table-page-2.html, …

    —

    Tips

    • If you know exactly how many pages there are, you can loop for page in range(1, N+1) instead and stop when you’ve hit N.
    • If your “Next” button has no text, use only class-based selectors. e.g.:

    driver.find_element(By.CSS_SELECTOR,
      "button.MuiButton-containedSecondary")
    

    • If the button is in a div that only appears once pages are loaded, you might need a second WebDriverWait.

    —
    Choose approach 1 when you can. It’s faster, simpler and won’t break if Facebook/Google changes their CSS. If you’re forced to emulate a real browser, approach 2 works reliably for any JS-heavy site. Good luck!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.