Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0613· OpenAIAi2 WildChat-4.8M (AI2)

Write a program in Python that takes the last 10 sports news from the site yjc.ir and…

  1. User

    Write a program in Python that takes the last 10 sports news from the site yjc.ir and stores it in the mongodb database. Then, every 3 hours, it receives the last 10 news of the site and saves it in the database. It checks the messages and does not save the messages that are repeated. Please explain the steps in order. This program stores the content of each news page in the database. For example, link to photo, news text, news URL, news date, etc.

  2. Assistant

    To write a program in Python that achieves the mentioned task, you can follow these steps:

    1. Import the required libraries:
    from urllib.request import urlopen
    from bs4 import BeautifulSoup
    from pymongo import MongoClient
    from datetime import datetime
    import time
    
    1. Set up a connection with the MongoDB database:
    client = MongoClient("mongodb://localhost:27017/")  # Update with your MongoDB connection string
    database = client["news_database"]
    collection = database["sports_news"]
    
    1. Create a function to scrape and save the news data:
    def save_news():
        # Fetch the news page URL
        url = "http://www.yjc.ir/fa/list/9/سرگرمی"
    
        # Open and read the news page
        response = urlopen(url)
        html = response.read()
    
        # Parse the HTML content using BeautifulSoup
        soup = BeautifulSoup(html, 'html.parser')
    
        # Find the last 10 news articles
        news_articles = soup.find_all('div', {'class': 'item-list'})
    
        # Iterate over each news article
        for article in news_articles[:10]:
            # Extract relevant information from the article
            link = article.find('a')['href']
            text = article.find('h2').text
            date = article.find('span', {'class': 'date-time'}).text
    
            # Check if the news article already exists in the database
            existing_news = collection.find_one({'link': link})
            if existing_news:
                continue
    
            # Create a news document to insert into the database
            news_data = {
                'link': link,
                'text': text,
                'date': date,
                'timestamp': datetime.now()
            }
    
            # Insert the news data into the collection
            collection.insert_one(news_data)
    
        print("News saved successfully!")
    
    1. Create a loop that executes the save_news() function every 3 hours:
    while True:
        save_news()
        time.sleep(3 * 60 * 60)  # Sleep for 3 hours before fetching the news again
    

    By following these steps, you can periodically scrape the last 10 sports news from the given website and store them in a MongoDB database. The program also checks for duplicate news articles and avoids saving them.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.