Write a program in Python that takes the last 10 sports news from the site yjc.ir and…
- User
Write a program in Python that takes the last 10 sports news from the site yjc.ir and stores it in the mongodb database. Then, every 3 hours, it receives the last 10 news of the site and saves it in the database. It checks the messages and does not save the messages that are repeated. Please explain the steps in order. This program stores the content of each news page in the database. For example, link to photo, news text, news URL, news date, etc.
- Assistant
To write a program in Python that achieves the mentioned task, you can follow these steps:
- Import the required libraries:
from urllib.request import urlopen from bs4 import BeautifulSoup from pymongo import MongoClient from datetime import datetime import time- Set up a connection with the MongoDB database:
client = MongoClient("mongodb://localhost:27017/") # Update with your MongoDB connection string database = client["news_database"] collection = database["sports_news"]- Create a function to scrape and save the news data:
def save_news(): # Fetch the news page URL url = "http://www.yjc.ir/fa/list/9/سرگرمی" # Open and read the news page response = urlopen(url) html = response.read() # Parse the HTML content using BeautifulSoup soup = BeautifulSoup(html, 'html.parser') # Find the last 10 news articles news_articles = soup.find_all('div', {'class': 'item-list'}) # Iterate over each news article for article in news_articles[:10]: # Extract relevant information from the article link = article.find('a')['href'] text = article.find('h2').text date = article.find('span', {'class': 'date-time'}).text # Check if the news article already exists in the database existing_news = collection.find_one({'link': link}) if existing_news: continue # Create a news document to insert into the database news_data = { 'link': link, 'text': text, 'date': date, 'timestamp': datetime.now() } # Insert the news data into the collection collection.insert_one(news_data) print("News saved successfully!")- Create a loop that executes the save_news() function every 3 hours:
while True: save_news() time.sleep(3 * 60 * 60) # Sleep for 3 hours before fetching the news againBy following these steps, you can periodically scrape the last 10 sports news from the given website and store them in a MongoDB database. The program also checks for duplicate news articles and avoids saving them.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.