Write a Python script to scrape data from a dynamic website using Selenium.
- User
Write a Python script to scrape data from a dynamic website using Selenium.
- Assistant
Creating a Python script for web scraping
The user wants a Python script to scrape data from a dynamic website using Selenium. I need to provide a generic template that sets up the Selenium WebDriver, waits for elements, and potentially handles pagination or headless browsing. It's important to include a note on the website's terms of service and legal/ethical considerations. I might use examples from a demo site like "quotes.toscrape.com/js" for scraping product titles and prices, ensuring the script includes comments for clarity.Building a Selenium web scraping script
I'm deciding between using Chrome or Edge for this task, with Chrome being the popular choice. The user might not have the chromedriver, so I’ll use webdriver-manager instead. I’ll start by importing necessary libraries and setting up the basics.
Next, I’ll configure options, start the driver, and open the URL "https://quotes.toscrape.com/js/". I need to wait for dynamic content to load, scrape quotes and authors, and possibly handle pagination by clicking "Next" until it's disabled. I’ll keep the script minimal yet complete, using the appropriate method for newer Selenium versions.Crafting a concise script response
I need to keep in mind the guidelines, avoiding phrases like "Great question" and getting straight to the point. I’ll focus on providing a single, effective script along with quick usage steps. Since the verbosity level is set to 2, I'll make sure to keep it concise while including essential code snippets and some helpful comments for clarity. My goal is to balance being helpful without overwhelming the user. Let's get started on that! Below is a minimal but complete example of using Selenium to scrape data from a dynamic website (one that loads content with JavaScript).
This example uses
https://quotes.toscrape.com/js/, a site specifically made for scraping demos.
1. Install dependencies
pip install selenium webdriver-manager beautifulsoup4
2. Python script (save as
scrape_dynamic.py)""" Scrape data from a dynamic website using Selenium. Example target: https://quotes.toscrape.com/js/ This page loads quotes via JavaScript, so normal requests/BeautifulSoup won't see the quotes without executing JS. Usage: python scrape_dynamic.py """ from selenium import webdriver from selenium.webdriver.chrome.service import Service from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from webdriver_manager.chrome import ChromeDriverManager from bs4 import BeautifulSoup import time URL = "https://quotes.toscrape.com/js/" def create_driver(headless: bool = True) -> webdriver.Chrome: """Create and return a configured Selenium Chrome WebDriver.""" chrome_options = webdriver.ChromeOptions() if headless: chrome_options.add_argument("--headless=new") chrome_options.add_argument("--no-sandbox") chrome_options.add_argument("--disable-dev-shm-usage") service = Service(ChromeDriverManager().install()) driver = webdriver.Chrome(service=service, options=chrome_options) return driver def scrape_quotes(driver: webdriver.Chrome): """Scrape all quotes (with pagination) from the demo site.""" driver.get(URL) wait = WebDriverWait(driver, 10) all_results = [] while True: # Wait until the quotes are loaded on the page wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.quote"))) # You can either parse via Selenium directly or via BeautifulSoup page_html = driver.page_source soup = BeautifulSoup(page_html, "html.parser") quote_blocks = soup.select("div.quote") for qb in quote_blocks: text = qb.select_one("span.text").get_text(strip=True) author = qb.select_one("small.author").get_text(strip=True) tags = [t.get_text(strip=True) for t in qb.select("div.tags a.tag")] all_results.append( { "text": text, "author": author, "tags": tags, } ) # Try to click the "Next" button if it exists try: next_btn = driver.find_element(By.CSS_SELECTOR, "li.next > a") except Exception: # No next button -> reached last page break next_btn.click() time.sleep(1) # small delay to allow next page to load return all_results def main(): driver = create_driver(headless=True) try: quotes = scrape_quotes(driver) finally: driver.quit() # Print results for i, q in enumerate(quotes, start=1): print(f"{i}. {q['text']} — {q['author']} (tags: {', '.join(q['tags'])})") if __name__ == "__main__": main()
How to adapt this to your own dynamic site
- Change
URLto your target page. - Inspect the page in your browser (DevTools) and update selectors:
- Replace
div.quote,span.text,small.author, etc. with the actual CSS selectors or XPath for the elements you want.
- Replace
- Adjust the wait condition:
- E.g.,
presence_of_all_elements_located((By.CSS_SELECTOR, "your-selector"))
- E.g.,
- Remove the pagination loop or change the logic if your site uses infinite scroll, “Load more” buttons, etc.
Important note
Always check the website’s Terms of Service and robots.txt, and respect rate limits and legal restrictions when scraping.
- Change
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.