Skip to content
TheChatArchiveThe Chat Archive
GPT 5.1· openaiHugging Face Frontier Reasoning Traces

Write a Python script to scrape data from a dynamic website using Selenium.

  1. User

    Write a Python script to scrape data from a dynamic website using Selenium.

  2. Assistant

    Creating a Python script for web scraping

    The user wants a Python script to scrape data from a dynamic website using Selenium. I need to provide a generic template that sets up the Selenium WebDriver, waits for elements, and potentially handles pagination or headless browsing. It's important to include a note on the website's terms of service and legal/ethical considerations. I might use examples from a demo site like "quotes.toscrape.com/js" for scraping product titles and prices, ensuring the script includes comments for clarity.Building a Selenium web scraping script

    I'm deciding between using Chrome or Edge for this task, with Chrome being the popular choice. The user might not have the chromedriver, so I’ll use webdriver-manager instead. I’ll start by importing necessary libraries and setting up the basics.

    Next, I’ll configure options, start the driver, and open the URL "https://quotes.toscrape.com/js/". I need to wait for dynamic content to load, scrape quotes and authors, and possibly handle pagination by clicking "Next" until it's disabled. I’ll keep the script minimal yet complete, using the appropriate method for newer Selenium versions.Crafting a concise script response

    I need to keep in mind the guidelines, avoiding phrases like "Great question" and getting straight to the point. I’ll focus on providing a single, effective script along with quick usage steps. Since the verbosity level is set to 2, I'll make sure to keep it concise while including essential code snippets and some helpful comments for clarity. My goal is to balance being helpful without overwhelming the user. Let's get started on that! Below is a minimal but complete example of using Selenium to scrape data from a dynamic website (one that loads content with JavaScript).

    This example uses https://quotes.toscrape.com/js/, a site specifically made for scraping demos.


    1. Install dependencies

    pip install selenium webdriver-manager beautifulsoup4
    

    2. Python script (save as scrape_dynamic.py)

    """
    Scrape data from a dynamic website using Selenium.
    
    Example target: https://quotes.toscrape.com/js/
    This page loads quotes via JavaScript, so normal requests/BeautifulSoup
    won't see the quotes without executing JS.
    
    Usage:
        python scrape_dynamic.py
    """
    
    from selenium import webdriver
    from selenium.webdriver.chrome.service import Service
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    
    from webdriver_manager.chrome import ChromeDriverManager
    from bs4 import BeautifulSoup
    import time
    
    
    URL = "https://quotes.toscrape.com/js/"
    
    
    def create_driver(headless: bool = True) -> webdriver.Chrome:
        """Create and return a configured Selenium Chrome WebDriver."""
        chrome_options = webdriver.ChromeOptions()
        if headless:
            chrome_options.add_argument("--headless=new")
        chrome_options.add_argument("--no-sandbox")
        chrome_options.add_argument("--disable-dev-shm-usage")
    
        service = Service(ChromeDriverManager().install())
        driver = webdriver.Chrome(service=service, options=chrome_options)
        return driver
    
    
    def scrape_quotes(driver: webdriver.Chrome):
        """Scrape all quotes (with pagination) from the demo site."""
    
        driver.get(URL)
        wait = WebDriverWait(driver, 10)
    
        all_results = []
    
        while True:
            # Wait until the quotes are loaded on the page
            wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.quote")))
    
            # You can either parse via Selenium directly or via BeautifulSoup
            page_html = driver.page_source
            soup = BeautifulSoup(page_html, "html.parser")
    
            quote_blocks = soup.select("div.quote")
            for qb in quote_blocks:
                text = qb.select_one("span.text").get_text(strip=True)
                author = qb.select_one("small.author").get_text(strip=True)
                tags = [t.get_text(strip=True) for t in qb.select("div.tags a.tag")]
    
                all_results.append(
                    {
                        "text": text,
                        "author": author,
                        "tags": tags,
                    }
                )
    
            # Try to click the "Next" button if it exists
            try:
                next_btn = driver.find_element(By.CSS_SELECTOR, "li.next > a")
            except Exception:
                # No next button -> reached last page
                break
    
            next_btn.click()
            time.sleep(1)  # small delay to allow next page to load
    
        return all_results
    
    
    def main():
        driver = create_driver(headless=True)
        try:
            quotes = scrape_quotes(driver)
        finally:
            driver.quit()
    
        # Print results
        for i, q in enumerate(quotes, start=1):
            print(f"{i}. {q['text']} — {q['author']}  (tags: {', '.join(q['tags'])})")
    
    
    if __name__ == "__main__":
        main()
    

    How to adapt this to your own dynamic site

    1. Change URL to your target page.
    2. Inspect the page in your browser (DevTools) and update selectors:
      • Replace div.quote, span.text, small.author, etc. with the actual CSS selectors or XPath for the elements you want.
    3. Adjust the wait condition:
      • E.g., presence_of_all_elements_located((By.CSS_SELECTOR, "your-selector"))
    4. Remove the pagination loop or change the logic if your site uses infinite scroll, “Load more” buttons, etc.

    Important note

    Always check the website’s Terms of Service and robots.txt, and respect rate limits and legal restrictions when scraping.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.