Write a program in Python that takes the last 10 sports news from the site yjc.ir and…
- User
Write a program in Python that takes the last 10 sports news from the site yjc.ir and stores it in the mongodb database. Then, every 3 hours, it receives the last 10 news of the site and saves it in the database. It checks the messages and does not save the messages that are repeated. Please explain the steps in order. This program stores the content of each news page in the database. For example, link to photo, news text, news URL, news date, etc.
- Assistant
To implement this program in Python, you can follow the steps below:
-
Install required packages:
- pymongo: Allows interacting with MongoDB from Python.
- beautifulsoup4: A library for web scraping.
You can install these packages using pip:
pip install pymongo beautifulsoup4 -
Import necessary modules:
from datetime import datetime from time import sleep from bs4 import BeautifulSoup import requests from pymongo import MongoClient- Connect to MongoDB:
client = MongoClient() db = client['news_database'] # Creating or accessing the database news_collection = db['news_collection'] # Creating or accessing the collection- Define a function to scrape and store the news:
def scrape_and_store_news(): # Fetching the latest 10 news from yjc.ir url = 'https://www.yjc.ir/fa/rss/18' response = requests.get(url) soup = BeautifulSoup(response.content, 'xml') news_items = soup.find_all('item')[:10] # Fetching only the latest 10 news # Adding unique news items to the MongoDB collection for item in news_items: # Check if news already exists in the collection if news_collection.count_documents({'title': item.title.text}) > 0: continue # Extracting required data from the news item news_data = { 'title': item.title.text, 'link': item.link.text, 'description': item.description.text, 'pubDate': datetime.strptime(item.pubDate.text, '%a, %d %b %Y %H:%M:%S %z') } # Inserting the news data into the MongoDB collection news_collection.insert_one(news_data)- Call the
scrape_and_store_news()function once to store the initial 10 news from yjc.ir:
scrape_and_store_news()- Use a while loop to periodically fetch and store the latest 10 news every 3 hours:
while True: sleep(3 * 60 * 60) # Sleep for 3 hours scrape_and_store_news()By following these steps, you should be able to write a Python program that periodically fetches the last 10 sports news from yjc.ir and stores them in a MongoDB database. The program also checks for repeated news items before saving them in the collection.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.