Write a small Python process for web scrapping.
- User
Write a small Python process for web scrapping.
Assistant
Of course! Here is a small but complete Python process for web scraping.
This example will be simple, easy to understand, and will perform a common scraping task: extracting specific data from a web page.
The Goal
We will write a script to scrape all the quotes from the website http://quotes.toscrape.com, which is a site specifically designed for practicing web scraping. For each quote, we will extract the text, the author, and the tags.
The Tools
We'll use two of the most popular Python libraries for this task:
requests: To fetch the HTML content of the web page.BeautifulSoup4: To parse the HTML and easily extract the data we need.
Step 1: Setting up your Environment
First, you need to install the required libraries. If you don't have them, open your terminal or command prompt and run:
pip install requests beautifulsoup4Step 2: The Python Script
Create a new Python file, for example
scraper.py, and paste the following code into it. The comments in the code explain what each part does.# scraper.py import requests from bs4 import BeautifulSoup import csv # The URL of the website we want to scrape URL = "http://quotes.toscrape.com" def scrape_quotes(url): """ Fetches the webpage and scrapes the quotes, authors, and tags. Returns a list of dictionaries, where each dictionary represents a quote. """ try: # 1. Make an HTTP GET request to the URL print(f"Fetching {url}...") response = requests.get(url) # Raise an exception if the request was unsuccessful (e.g., 404 Not Found) response.raise_for_status() print("Successfully fetched the page.") except requests.exceptions.RequestException as e: print(f"Error fetching URL: {e}") return [] # 2. Parse the HTML content using BeautifulSoup # 'html.parser' is the built-in Python parser soup = BeautifulSoup(response.text, 'html.parser') # 3. Find all the quote containers # By inspecting the website, we find each quote is in a <div> with class="quote" quotes_html = soup.find_all('div', class_='quote') scraped_data = [] # 4. Loop through each quote container to extract the data for quote in quotes_html: # Extract the text of the quote (found in a <span> with class="text") text = quote.find('span', class_='text').text.strip() # Extract the author (found in a <small> with class="author") author = quote.find('small', class_='author').text.strip() # Extract the tags (each tag is an <a> with class="tag" inside a <div> with class="tags") tags_container = quote.find('div', class_='tags') tags = [tag.text for tag in tags_container.find_all('a', class_='tag')] # Store the extracted data in our list scraped_data.append({ 'text': text, 'author': author, 'tags': tags }) return scraped_data def save_to_csv(data, filename="quotes.csv"): """Saves the scraped data to a CSV file.""" if not data: print("No data to save.") return # The keys of the first dictionary will be our CSV headers headers = data[0].keys() with open(filename, 'w', newline='', encoding='utf-8') as csvfile: writer = csv.DictWriter(csvfile, fieldnames=headers) writer.writeheader() writer.writerows(data) print(f"Data successfully saved to {filename}") # --- Main execution block --- if __name__ == "__main__": # Scrape the data quotes_data = scrape_quotes(URL) if quotes_data: # Print the first quote to verify it's working print("\n--- First Scraped Quote ---") print(quotes_data[0]) print("--------------------------\n") # Save all the data to a CSV file save_to_csv(quotes_data)Step 3: How to Run the Script
- Save the code above as
scraper.py. - Open your terminal or command prompt.
- Navigate to the directory where you saved the file.
- Run the script using Python:
python scraper.pyWhat Happens When You Run It?
-
Output in Terminal: The script will first print messages indicating it's fetching the URL and then show you the first quote it scraped, like this:
Fetching http://quotes.toscrape.com... Successfully fetched the page. --- First Scraped Quote --- {'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']} -------------------------- Data successfully saved to quotes.csv -
A New File: A new file named
quotes.csvwill be created in the same directory. If you open it with a spreadsheet program (like Excel, Google Sheets, or LibreOffice Calc), you'll see the neatly organized data:
text author tags “The world as we have created it is a process of our thinking. It cannot be changed without..." Albert Einstein ['change', 'deep-thoughts', ... ] “It is our choices, Harry, that show what we truly are, far more than our abilities.” J.K. Rowling ['abilities', 'choices'] ...and so on for all quotes on the page. Important Considerations for Real-World Scraping
- Be Respectful: Don't send too many requests in a short time. This can overload the server. Use
time.sleep()between requests if you are scraping multiple pages. - Check
robots.txt: Many websites have arobots.txtfile (e.g.,http://example.com/robots.txt) that specifies which parts of the site bots are not allowed to access. Always respect these rules. - User-Agent: Some websites block scripts with the default
requestsuser-agent. You can mimic a real browser by setting a custom user-agent in your request headers. - Dynamic Content: This script works for static HTML. If a website loads content using JavaScript, you may need more advanced tools like Selenium or Playwright that can control a real web browser.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.