Automation python Automation workflow solution: scrap a news site daily for updating a…
- User
Automation python
Automation workflow solution: scrap a news site daily for updating a file that stores content
I need this app self execute once at day and update store files
Title, date, content
I need you give me some url for testing
The urls’s topic must be related to Ai in education
- Assistant
Creating an automated Python application to scrape news related to AI in education daily and update stored files involves several steps:
-
Selecting the Source Websites: Choose reliable news websites that frequently publish articles on AI in education.
-
Web Scraping: Use libraries like
requestsandBeautifulSoupto extract titles, dates, and content from the chosen websites. -
Data Storage: Store the extracted data in a structured format such as CSV, JSON, or a database like SQLite.
-
Scheduling the Task: Use scheduling libraries like
schedule,APScheduler, or system tools likecron(Unix) or Task Scheduler (Windows) to run the script daily. -
Execution and Maintenance: Ensure the script handles exceptions, logs activities, and updates the data without duplication.
Below is a step-by-step guide along with sample code to help you build this automation workflow.
1. Prerequisites
Before you begin, ensure you have the following Python libraries installed:
pip install requests pip install beautifulsoup4 pip install schedule2. Choose Source Websites
Here are some URLs that focus on AI in education, suitable for testing and scraping:
-
EdTech Magazine - AI in Education
-
Education Week - Artificial Intelligence
-
AI in Education News on Forbes
-
TechCrunch - AI Education
-
The Hechinger Report - Teaching and Technology
Note: Ensure that you have permission to scrape these websites by reviewing their
robots.txtfiles and terms of service. Alternatively, consider using official APIs if available.3. Sample Python Script
Below is a sample Python script that scrapes articles from EdTech Magazine related to AI in education and stores the data in a CSV file. You can adapt this script to other websites by modifying the scraping logic.
import requests from bs4 import BeautifulSoup import csv import os from datetime import datetime import schedule import time # Function to scrape a single website def scrape_edtech_magazine(): url = 'https://www.edtechmagazine.com/k12/article/topic/artificial-intelligence' headers = {'User-Agent': 'Mozilla/5.0'} response = requests.get(url, headers=headers) if response.status_code != 200: print(f"Failed to fetch {url}") return soup = BeautifulSoup(response.content, 'html.parser') # Find all article blocks (this may vary based on website structure) articles = soup.find_all('div', class_='river-well') data = [] for article in articles: # Extract title title_tag = article.find('h3', class_='headline') title = title_tag.get_text(strip=True) if title_tag else 'No Title' # Extract date date_tag = article.find('time') date = date_tag['datetime'] if date_tag and date_tag.has_attr('datetime') else 'No Date' # Extract content/summary content_tag = article.find('div', class_='summary') content = content_tag.get_text(strip=True) if content_tag else 'No Content' data.append({ 'Title': title, 'Date': date, 'Content': content }) # Define the CSV file path file_exists = os.path.isfile('ai_in_education_news.csv') with open('ai_in_education_news.csv', 'a', newline='', encoding='utf-8') as csvfile: fieldnames = ['Title', 'Date', 'Content'] writer = csv.DictWriter(csvfile, fieldnames=fieldnames) # Write header only if the file does not exist if not file_exists: writer.writeheader() for entry in data: writer.writerow(entry) print(f"{datetime.now()}: Scraped {len(data)} articles from EdTech Magazine.") # Function to run the scraper daily def job(): scrape_edtech_magazine() # You can add more scraper functions for other websites here # Schedule the job every day at a specific time, e.g., 8:00 AM schedule.every().day.at("08:00").do(job) print("Scheduler started. Waiting to run the daily job...") while True: schedule.run_pending() time.sleep(60) # wait one minute4. Explanation of the Script
-
Imports:
requestsandBeautifulSoupfor web scraping.csvandosfor file handling.datetimefor timestamping.scheduleandtimefor scheduling the daily task.
-
scrape_edtech_magazine():
- Sends a GET request to the EdTech Magazine AI in Education page.
- Parses the HTML content to find article blocks.
- Extracts the title, date, and content from each article.
- Appends the data to a CSV file named
ai_in_education_news.csv.
-
Scheduling:
- The
jobfunction calls the scraper. schedule.every().day.at("08:00").do(job)schedules the job to run daily at 8:00 AM.- The
whileloop keeps the script running, checking every minute if the scheduled time has arrived.
- The
5. Running the Script
To ensure the script runs continuously and executes daily, you can:
-
Run as a Background Process:
- On Unix systems, use
nohuporscreento run the script in the background. - Example:
nohup python scraper.py &
- On Unix systems, use
-
Use System Services:
- Create a systemd service (Linux) or a Windows Service to manage the script.
-
Alternative Scheduling:
- Instead of using the
schedulelibrary, you can use system-level schedulers likecron(Linux/macOS) or Task Scheduler (Windows) to execute the script daily.
- Instead of using the
6. Adapting to Other Websites
Each website has a unique HTML structure. To scrape different websites:
-
Inspect the Website:
- Use browser developer tools to understand the HTML structure of the articles.
-
Modify the Scraping Logic:
- Adjust the
find_alland other BeautifulSoup methods to target the correct HTML tags and classes.
- Adjust the
-
Example for Forbes:
Here's an example function to scrape Forbes search results for "AI in Education":
def scrape_forbes_ai_education(): query = 'AI in Education' url = f'https://www.forbes.com/search/?q={query.replace(" ", "+")}' headers = {'User-Agent': 'Mozilla/5.0'} response = requests.get(url, headers=headers) if response.status_code != 200: print(f"Failed to fetch {url}") return soup = BeautifulSoup(response.content, 'html.parser') articles = soup.find_all('div', class_='stream-item__text') data = [] for article in articles: title_tag = article.find('h3', class_='stream-item__title') title = title_tag.get_text(strip=True) if title_tag else 'No Title' date_tag = article.find('span', class_='stream-item__date') date = date_tag.get_text(strip=True) if date_tag else 'No Date' content_tag = article.find('p', class_='stream-item__excerpt') content = content_tag.get_text(strip=True) if content_tag else 'No Content' data.append({ 'Title': title, 'Date': date, 'Content': content }) # Append to the same CSV file_exists = os.path.isfile('ai_in_education_news.csv') with open('ai_in_education_news.csv', 'a', newline='', encoding='utf-8') as csvfile: fieldnames = ['Title', 'Date', 'Content'] writer = csv.DictWriter(csvfile, fieldnames=fieldnames) if not file_exists: writer.writeheader() for entry in data: writer.writerow(entry) print(f"{datetime.now()}: Scraped {len(data)} articles from Forbes.")- Add
scrape_forbes_ai_education()to thejob()function to include Forbes in the scraping process.
- Add
7. Enhancements and Best Practices
-
Avoid Duplication:
- Implement checks to prevent duplicate entries, such as tracking article URLs or unique identifiers.
-
Use a Database:
- For scalability, consider using a database (e.g., SQLite, PostgreSQL) instead of CSV files.
-
Error Handling and Logging:
- Add comprehensive error handling to manage exceptions.
- Implement logging to monitor the scraper’s activities and issues.
-
Respectful Scraping:
- Adhere to the website’s
robots.txt. - Include delays between requests if scraping multiple pages to avoid overloading servers.
- Adhere to the website’s
-
Dynamic Content:
- If a website loads content dynamically using JavaScript, consider using Selenium or headless browsers like Playwright.
8. Conclusion
By following the steps above, you can create a Python-based automation workflow that scrapes news articles related to AI in education daily and stores them for further analysis or usage. Customize the scraper to fit the HTML structure of your chosen websites and enhance the script with additional functionalities as needed.
If you encounter specific challenges or have questions about particular aspects of the implementation, feel free to ask!
-