Skip to content
The Chat Archive
gpt-3.5-turbo-0301· OpenAI WildChat-4.8M (AI2)

Write a program in Python that takes the last 10 sports news from the yjc.ir website and…

  1. User

    Write a program in Python that takes the last 10 sports news from the yjc.ir website and stores it in the mongodb database. Then, every 8 hours, it receives the last 10 news of the site and saves it in the database. If the messages are duplicates, do not save them. Please explain the steps in order.

  2. Assistant

    Sure, the steps to achieve this task are as follows:

    1. Install the required libraries: pymongo, requests, and Beautiful Soup 4 (bs4) using pip.

    2. Import the required libraries in your Python code:

      from datetime import datetime, timedelta
      import time
      import requests
      from bs4 import BeautifulSoup
      from pymongo import MongoClient
      
    3. Connect to your MongoDB database with a MongoClient instance, using the connect() method. You need to replace username, password, host, and port with your own values.

      client = MongoClient('mongodb://username:password@host:port')
      
    4. Select the database and the collection you want to store the news in. If the collection does not exist yet, it will be automatically created.

      db = client['sports_news']
      news_collection = db['news']
      
    5. Define a function named get_last_news() that extracts the last 10 sports news from the yjc.ir website and returns them as a list of dictionaries. You can use the requests library to get the HTML content of the website and the BeautifulSoup library to parse it and extract the news.

      def get_last_news():
          url = 'https://www.yjc.ir/fa/list/12/%D9%88%D8%B1%D8%B2%D8%B4%DB%8C'
          r = requests.get(url)
          soup = BeautifulSoup(r.content, 'html.parser')
          news = []
          for item in soup.find_all('div', class_='news'):
              title = item.find('a').get_text().strip()
              link = item.find('a').get('href')
              date_str = item.find('span', class_='date').get_text()
              date = datetime.strptime(date_str, '%Y-%m-%d %H:%M:%S')
              news.append({'title': title, 'link': link, 'date': date})
          return news[:10]
      
    6. Define a function named save_news(news) that inserts the news in the news_collection collection, but only if they are not duplicates. In order to check for duplicates, you can query the database for the last 10 news and compare their titles and links with the new ones. If a news is already in the database, you should not insert it.

      def save_news(news):
          last_news = news_collection.find().sort('date', -1).limit(10)
          last_titles = set(map(lambda n: n['title'], last_news))
          last_links = set(map(lambda n: n['link'], last_news))
          for item in news:
              if item['title'] not in last_titles and item['link'] not in last_links:
                  news_collection.insert_one(item)
      
    7. Define the main loop of your program that executes every 8 hours and calls the get_last_news() and save_news() functions. In order to run this loop every 8 hours, you can use the time.sleep() function together with Python's datetime and timedelta classes, like this:

      while True:
          news = get_last_news()
          save_news(news)
          time.sleep(timedelta(hours=8).seconds)
      

      This loop will run indefinitely, getting the last 10 news and storing them in the database every 8 hours.

    Here's the full code:

    from datetime import datetime, timedelta
    import time
    import requests
    from bs4 import BeautifulSoup
    from pymongo import MongoClient
    
    client = MongoClient('mongodb://username:password@host:port')
    db = client['sports_news']
    news_collection = db['news']
    
    def get_last_news():
        url = 'https://www.yjc.ir/fa/list/12/%D9%88%D8%B1%D8%B2%D8%B4%DB%8C'
        r = requests.get(url)
        soup = BeautifulSoup(r.content, 'html.parser')
        news = []
        for item in soup.find_all('div', class_='news'):
            title = item.find('a').get_text().strip()
            link = item.find('a').get('href')
            date_str = item.find('span', class_='date').get_text()
            date = datetime.strptime(date_str, '%Y-%m-%d %H:%M:%S')
            news.append({'title': title, 'link': link, 'date': date})
        return news[:10]
    
    def save_news(news):
        last_news = news_collection.find().sort('date', -1).limit(10)
        last_titles = set(map(lambda n: n['title'], last_news))
        last_links = set(map(lambda n: n['link'], last_news))
        for item in news:
            if item['title'] not in last_titles and item['link'] not in last_links:
                news_collection.insert_one(item)
    
    while True:
        news = get_last_news()
        save_news(news)
        time.sleep(timedelta(hours=8).seconds)
    

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.