Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro· GoogleHugging Face LMArena Human Preference 140K

Write a small Python process for web scrapping.

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-sonnet-4-20250514-thinking-32k’s answer.
  1. User

    Write a small Python process for web scrapping.

  2. Gemini Assistant

    Of course! Here is a small but complete Python process for web scraping.

    This example will be simple, easy to understand, and will perform a common scraping task: extracting specific data from a web page.

    The Goal

    We will write a script to scrape all the quotes from the website http://quotes.toscrape.com, which is a site specifically designed for practicing web scraping. For each quote, we will extract the text, the author, and the tags.

    The Tools

    We'll use two of the most popular Python libraries for this task:

    1. requests: To fetch the HTML content of the web page.
    2. BeautifulSoup4: To parse the HTML and easily extract the data we need.

    Step 1: Setting up your Environment

    First, you need to install the required libraries. If you don't have them, open your terminal or command prompt and run:

    pip install requests beautifulsoup4
    

    Step 2: The Python Script

    Create a new Python file, for example scraper.py, and paste the following code into it. The comments in the code explain what each part does.

    # scraper.py
    
    import requests
    from bs4 import BeautifulSoup
    import csv
    
    # The URL of the website we want to scrape
    URL = "http://quotes.toscrape.com"
    
    def scrape_quotes(url):
        """
        Fetches the webpage and scrapes the quotes, authors, and tags.
        Returns a list of dictionaries, where each dictionary represents a quote.
        """
        try:
            # 1. Make an HTTP GET request to the URL
            print(f"Fetching {url}...")
            response = requests.get(url)
            
            # Raise an exception if the request was unsuccessful (e.g., 404 Not Found)
            response.raise_for_status() 
            print("Successfully fetched the page.")
    
        except requests.exceptions.RequestException as e:
            print(f"Error fetching URL: {e}")
            return []
    
        # 2. Parse the HTML content using BeautifulSoup
        # 'html.parser' is the built-in Python parser
        soup = BeautifulSoup(response.text, 'html.parser')
    
        # 3. Find all the quote containers
        # By inspecting the website, we find each quote is in a <div> with class="quote"
        quotes_html = soup.find_all('div', class_='quote')
        
        scraped_data = []
    
        # 4. Loop through each quote container to extract the data
        for quote in quotes_html:
            # Extract the text of the quote (found in a <span> with class="text")
            text = quote.find('span', class_='text').text.strip()
            
            # Extract the author (found in a <small> with class="author")
            author = quote.find('small', class_='author').text.strip()
            
            # Extract the tags (each tag is an <a> with class="tag" inside a <div> with class="tags")
            tags_container = quote.find('div', class_='tags')
            tags = [tag.text for tag in tags_container.find_all('a', class_='tag')]
    
            # Store the extracted data in our list
            scraped_data.append({
                'text': text,
                'author': author,
                'tags': tags
            })
            
        return scraped_data
    
    def save_to_csv(data, filename="quotes.csv"):
        """Saves the scraped data to a CSV file."""
        if not data:
            print("No data to save.")
            return
            
        # The keys of the first dictionary will be our CSV headers
        headers = data[0].keys()
        
        with open(filename, 'w', newline='', encoding='utf-8') as csvfile:
            writer = csv.DictWriter(csvfile, fieldnames=headers)
            writer.writeheader()
            writer.writerows(data)
        
        print(f"Data successfully saved to {filename}")
    
    
    # --- Main execution block ---
    if __name__ == "__main__":
        # Scrape the data
        quotes_data = scrape_quotes(URL)
        
        if quotes_data:
            # Print the first quote to verify it's working
            print("\n--- First Scraped Quote ---")
            print(quotes_data[0])
            print("--------------------------\n")
    
            # Save all the data to a CSV file
            save_to_csv(quotes_data)
    

    Step 3: How to Run the Script

    1. Save the code above as scraper.py.
    2. Open your terminal or command prompt.
    3. Navigate to the directory where you saved the file.
    4. Run the script using Python:
    python scraper.py
    

    What Happens When You Run It?

    1. Output in Terminal: The script will first print messages indicating it's fetching the URL and then show you the first quote it scraped, like this:

      Fetching http://quotes.toscrape.com...
      Successfully fetched the page.
      
      --- First Scraped Quote ---
      {'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}
      --------------------------
      
      Data successfully saved to quotes.csv
      
    2. A New File: A new file named quotes.csv will be created in the same directory. If you open it with a spreadsheet program (like Excel, Google Sheets, or LibreOffice Calc), you'll see the neatly organized data:

    textauthortags
    “The world as we have created it is a process of our thinking. It cannot be changed without..."Albert Einstein['change', 'deep-thoughts', ... ]
    “It is our choices, Harry, that show what we truly are, far more than our abilities.”J.K. Rowling['abilities', 'choices']
    ...and so on for all quotes on the page.

    Important Considerations for Real-World Scraping

    • Be Respectful: Don't send too many requests in a short time. This can overload the server. Use time.sleep() between requests if you are scraping multiple pages.
    • Check robots.txt: Many websites have a robots.txt file (e.g., http://example.com/robots.txt) that specifies which parts of the site bots are not allowed to access. Always respect these rules.
    • User-Agent: Some websites block scripts with the default requests user-agent. You can mimic a real browser by setting a custom user-agent in your request headers.
    • Dynamic Content: This script works for static HTML. If a website loads content using JavaScript, you may need more advanced tools like Selenium or Playwright that can control a real web browser.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.