On a webpage of a website I have a table with class="tabletable_8D7gp". Below that I…
- User
On a webpage of a website I have a table with
class="table_table__8D7gp". Below that I have a button withclass="MuiButtonBase-root MuiButton-root MuiButton-contained MuiButton-containedSecondary MuiButton-sizeSmall MuiButton-containedSizeSmall MuiButton-colorSecondary MuiButtonGroup-grouped MuiButtonGroup-groupedHorizontal MuiButtonGroup-groupedContained MuiButtonGroup-groupedContainedHorizontal MuiButtonGroup-groupedContainedSecondary MuiButton-root MuiButton-contained MuiButton-containedSecondary MuiButton-sizeSmall MuiButton-containedSizeSmall MuiButton-colorSecondary MuiButtonGroup-middleButton css-cbvs2h"This button switches to the next part of the table.How can I scrape this in web browser? I want each of the table pages saved in new file.
- Assistant
Here’s one way you can “click” through each page of a JS-powered table and dump its HTML into separate files. I’ll show you two approaches:
- Reverse-engineer the XHR/Fetch and pull the pages directly
- Browser-automation with Python + Selenium
—
1) Hit the same API the “Next” button is using
Open your DevTools → Network → click “Next” once. Look for an XHR/fetch that returns JSON or HTML for your table. Often it will be a URL like:
https://www.example.com/api/table?page=1&size=50If you can find that, you can skip the browser entirely and do something like:
import requests BASE = "https://www.example.com/api/table" headers = { "User-Agent": "Mozilla/5.0", # … any cookies or auth headers you need … } for page in range(1, 11): # or while True, break on empty result params = {"page": page, "size": 50} r = requests.get(BASE, headers=headers, params=params) if not r.ok or not r.text.strip(): break open(f"table-page-{page}.html", "w", encoding="utf8").write(r.text) print("Saved page", page)If the response is JSON, you can either
• Render it into HTML yourself, or
• Save it as JSON (table-page-1.json)This is by far the most robust.
—
2) Drive a real browser with Selenium
If you can’t find a nice API endpoint, you can literally click the “Next” button in a headless Chrome/Firefox with Selenium.
a) Install prerequisites
pip install selenium webdriver-managerb) Example script (
scrape_with_selenium.py)from selenium import webdriver from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC import time URL = "https://www.example.com/your-page" driver = webdriver.Chrome(service=Service(ChromeDriverManager().install())) try: driver.get(URL) wait = WebDriverWait(driver, 10) page = 1 while True: # 1) Wait for your table to load table = wait.until(EC.presence_of_element_located( (By.CSS_SELECTOR, ".table_table__8D7gp") )) # 2) Dump table HTML html = table.get_attribute("outerHTML") with open(f"table-page-{page}.html", "w", encoding="utf8") as f: f.write(html) print(f"Saved table page {page}") # 3) Find and click the Next button # Adjust this selector to match only the "Next" button try: next_btn = driver.find_element( By.XPATH, "//button[contains(@class,'MuiButton-containedSecondary') and normalize-space(.)='Next']" ) except: print("No Next button found → done.") break # If "Next" button is disabled, stop if next_btn.get_attribute("aria-disabled") == "true": print("Next button is disabled → done.") break next_btn.click() page += 1 # 4) Give React/JS a moment to re-render the table time.sleep(1) finally: driver.quit()c) Run it
python scrape_with_selenium.pyEach iteration saves
table-page-1.html,table-page-2.html, …—
Tips
• If you know exactly how many pages there are, you can loop
for page in range(1, N+1)instead and stop when you’ve hitN.
• If your “Next” button has no text, use only class-based selectors. e.g.:driver.find_element(By.CSS_SELECTOR, "button.MuiButton-containedSecondary")• If the button is in a
divthat only appears once pages are loaded, you might need a secondWebDriverWait.—
Choose approach 1 when you can. It’s faster, simpler and won’t break if Facebook/Google changes their CSS. If you’re forced to emulate a real browser, approach 2 works reliably for any JS-heavy site. Good luck!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.