Write a program in Python that takes the last 10 sports news from the site yjc.ir and…
- User
Write a program in Python that takes the last 10 sports news from the site yjc.ir and stores it in the mongodb database. Then, every n hours, it receives the last 10 news of the site and stores it in the database. If the messages are duplicates, do not save them. Get n from the user. Please explain the steps in order. This program stores the content of each news page in the database. For example, links to photos and videos, news text, news URL, news date, etc.
- Assistant
To write a program in Python that scrapes sports news from the website yjc.ir and stores it in a MongoDB database, you can follow these steps:
-
Install the necessary libraries:
- pymongo: for interacting with MongoDB
- requests: for sending HTTP requests
- BeautifulSoup: for parsing HTML content
You can install these libraries using pip with the following command:
pip install pymongo requests beautifulsoup4 -
Import the required modules:
import pymongo import requests from bs4 import BeautifulSoup -
Connect to MongoDB:
# Create a MongoDB client client = pymongo.MongoClient() # Connect to the desired database db = client['news_database'] # Access the collection for news articles collection = db['sports_news'] -
Scrape the news articles from yjc.ir:
url = 'https://www.yjc.ir/fa/subcategory/9' response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser') # Extract the required information for each news article news_articles = [] articles_div = soup.find(class_='articles') for article_div in articles_div.find_all('article'): # Extract the news title, URL, date, and other required information title = article_div.find('h2').text.strip() news_url = article_div.find('a')['href'] news_date = article_div.find('time')['datetime'] # Store the extracted information in a dictionary news_article = { 'title': title, 'url': news_url, 'date': news_date } # Add the news article to the list news_articles.append(news_article) -
Store the news articles in the MongoDB database:
# Insert the news articles into the collection collection.insert_many(news_articles) -
Ask the user for the refresh interval 'n':
refresh_interval = int(input("Enter the refresh interval in hours: ")) -
Implement a loop that periodically scrapes and stores news articles:
import time while True: # Scrape the news articles (similar to step 4) # Store the news articles in the database (similar to step 5) # Sleep for 'n' hours before scraping again time.sleep(refresh_interval * 3600) -
Check for duplicate news articles before storing them:
for news_article in news_articles: # Check if the news article already exists in the collection if collection.count_documents({'url': news_article['url']}) == 0: # Insert the news article into the collection collection.insert_one(news_article)
That's it! You now have a program that scrapes and stores sports news from yjc.ir into a MongoDB database, refreshing every 'n' hours and avoiding duplicate articles based on their URLs. Note that you may need to modify the scraping logic based on the specific structure of the website's HTML.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.