Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

Automation python Automation workflow solution: scrap a news site daily for updating a…

  1. User

    Automation python

    Automation workflow solution: scrap a news site daily for updating a file that stores content

    I need this app self execute once at day and update store files

    Title, date, content

    I need you give me some url for testing

    The urls’s topic must be related to Ai in education

  2. Assistant

    Creating an automated Python application to scrape news related to AI in education daily and update stored files involves several steps:

    1. Selecting the Source Websites: Choose reliable news websites that frequently publish articles on AI in education.

    2. Web Scraping: Use libraries like requests and BeautifulSoup to extract titles, dates, and content from the chosen websites.

    3. Data Storage: Store the extracted data in a structured format such as CSV, JSON, or a database like SQLite.

    4. Scheduling the Task: Use scheduling libraries like schedule, APScheduler, or system tools like cron (Unix) or Task Scheduler (Windows) to run the script daily.

    5. Execution and Maintenance: Ensure the script handles exceptions, logs activities, and updates the data without duplication.

    Below is a step-by-step guide along with sample code to help you build this automation workflow.


    1. Prerequisites

    Before you begin, ensure you have the following Python libraries installed:

    pip install requests
    pip install beautifulsoup4
    pip install schedule
    

    2. Choose Source Websites

    Here are some URLs that focus on AI in education, suitable for testing and scraping:

    1. EdTech Magazine - AI in Education

    2. Education Week - Artificial Intelligence

    3. AI in Education News on Forbes

    4. TechCrunch - AI Education

    5. The Hechinger Report - Teaching and Technology

    Note: Ensure that you have permission to scrape these websites by reviewing their robots.txt files and terms of service. Alternatively, consider using official APIs if available.

    3. Sample Python Script

    Below is a sample Python script that scrapes articles from EdTech Magazine related to AI in education and stores the data in a CSV file. You can adapt this script to other websites by modifying the scraping logic.

    import requests
    from bs4 import BeautifulSoup
    import csv
    import os
    from datetime import datetime
    import schedule
    import time
    
    # Function to scrape a single website
    def scrape_edtech_magazine():
        url = 'https://www.edtechmagazine.com/k12/article/topic/artificial-intelligence'
        headers = {'User-Agent': 'Mozilla/5.0'}
        response = requests.get(url, headers=headers)
    
        if response.status_code != 200:
            print(f"Failed to fetch {url}")
            return
    
        soup = BeautifulSoup(response.content, 'html.parser')
    
        # Find all article blocks (this may vary based on website structure)
        articles = soup.find_all('div', class_='river-well')
    
        data = []
        for article in articles:
            # Extract title
            title_tag = article.find('h3', class_='headline')
            title = title_tag.get_text(strip=True) if title_tag else 'No Title'
    
            # Extract date
            date_tag = article.find('time')
            date = date_tag['datetime'] if date_tag and date_tag.has_attr('datetime') else 'No Date'
    
            # Extract content/summary
            content_tag = article.find('div', class_='summary')
            content = content_tag.get_text(strip=True) if content_tag else 'No Content'
    
            data.append({
                'Title': title,
                'Date': date,
                'Content': content
            })
    
        # Define the CSV file path
        file_exists = os.path.isfile('ai_in_education_news.csv')
        with open('ai_in_education_news.csv', 'a', newline='', encoding='utf-8') as csvfile:
            fieldnames = ['Title', 'Date', 'Content']
            writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    
            # Write header only if the file does not exist
            if not file_exists:
                writer.writeheader()
    
            for entry in data:
                writer.writerow(entry)
        
        print(f"{datetime.now()}: Scraped {len(data)} articles from EdTech Magazine.")
    
    # Function to run the scraper daily
    def job():
        scrape_edtech_magazine()
        # You can add more scraper functions for other websites here
    
    # Schedule the job every day at a specific time, e.g., 8:00 AM
    schedule.every().day.at("08:00").do(job)
    
    print("Scheduler started. Waiting to run the daily job...")
    
    while True:
        schedule.run_pending()
        time.sleep(60)  # wait one minute
    

    4. Explanation of the Script

    • Imports:

      • requests and BeautifulSoup for web scraping.
      • csv and os for file handling.
      • datetime for timestamping.
      • schedule and time for scheduling the daily task.
    • scrape_edtech_magazine():

      • Sends a GET request to the EdTech Magazine AI in Education page.
      • Parses the HTML content to find article blocks.
      • Extracts the title, date, and content from each article.
      • Appends the data to a CSV file named ai_in_education_news.csv.
    • Scheduling:

      • The job function calls the scraper.
      • schedule.every().day.at("08:00").do(job) schedules the job to run daily at 8:00 AM.
      • The while loop keeps the script running, checking every minute if the scheduled time has arrived.

    5. Running the Script

    To ensure the script runs continuously and executes daily, you can:

    • Run as a Background Process:

      • On Unix systems, use nohup or screen to run the script in the background.
      • Example:
        nohup python scraper.py &
        
    • Use System Services:

      • Create a systemd service (Linux) or a Windows Service to manage the script.
    • Alternative Scheduling:

      • Instead of using the schedule library, you can use system-level schedulers like cron (Linux/macOS) or Task Scheduler (Windows) to execute the script daily.

    6. Adapting to Other Websites

    Each website has a unique HTML structure. To scrape different websites:

    1. Inspect the Website:

      • Use browser developer tools to understand the HTML structure of the articles.
    2. Modify the Scraping Logic:

      • Adjust the find_all and other BeautifulSoup methods to target the correct HTML tags and classes.
    3. Example for Forbes:

      Here's an example function to scrape Forbes search results for "AI in Education":

      def scrape_forbes_ai_education():
          query = 'AI in Education'
          url = f'https://www.forbes.com/search/?q={query.replace(" ", "+")}'
          headers = {'User-Agent': 'Mozilla/5.0'}
          response = requests.get(url, headers=headers)
      
          if response.status_code != 200:
              print(f"Failed to fetch {url}")
              return
      
          soup = BeautifulSoup(response.content, 'html.parser')
          articles = soup.find_all('div', class_='stream-item__text')
      
          data = []
          for article in articles:
              title_tag = article.find('h3', class_='stream-item__title')
              title = title_tag.get_text(strip=True) if title_tag else 'No Title'
      
              date_tag = article.find('span', class_='stream-item__date')
              date = date_tag.get_text(strip=True) if date_tag else 'No Date'
      
              content_tag = article.find('p', class_='stream-item__excerpt')
              content = content_tag.get_text(strip=True) if content_tag else 'No Content'
      
              data.append({
                  'Title': title,
                  'Date': date,
                  'Content': content
              })
      
          # Append to the same CSV
          file_exists = os.path.isfile('ai_in_education_news.csv')
          with open('ai_in_education_news.csv', 'a', newline='', encoding='utf-8') as csvfile:
              fieldnames = ['Title', 'Date', 'Content']
              writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
      
              if not file_exists:
                  writer.writeheader()
      
              for entry in data:
                  writer.writerow(entry)
          
          print(f"{datetime.now()}: Scraped {len(data)} articles from Forbes.")
      
      • Add scrape_forbes_ai_education() to the job() function to include Forbes in the scraping process.

    7. Enhancements and Best Practices

    • Avoid Duplication:

      • Implement checks to prevent duplicate entries, such as tracking article URLs or unique identifiers.
    • Use a Database:

      • For scalability, consider using a database (e.g., SQLite, PostgreSQL) instead of CSV files.
    • Error Handling and Logging:

      • Add comprehensive error handling to manage exceptions.
      • Implement logging to monitor the scraper’s activities and issues.
    • Respectful Scraping:

      • Adhere to the website’s robots.txt.
      • Include delays between requests if scraping multiple pages to avoid overloading servers.
    • Dynamic Content:

      • If a website loads content dynamically using JavaScript, consider using Selenium or headless browsers like Playwright.

    8. Conclusion

    By following the steps above, you can create a Python-based automation workflow that scrapes news articles related to AI in education daily and stores them for further analysis or usage. Customize the scraper to fit the HTML structure of your chosen websites and enhance the script with additional functionalities as needed.

    If you encounter specific challenges or have questions about particular aspects of the implementation, feel free to ask!