Write a program in Python that takes the last 10 sports news from the yjc.ir website and…
- User
Write a program in Python that takes the last 10 sports news from the yjc.ir website and stores it in the mongodb database. Then, every 8 hours, it receives the last 10 news of the site and saves it in the database. If the messages are duplicates, do not save them. Please explain the steps in order.
- Assistant
Sure, the steps to achieve this task are as follows:
-
Install the required libraries: pymongo, requests, and Beautiful Soup 4 (bs4) using pip.
-
Import the required libraries in your Python code:
from datetime import datetime, timedelta import time import requests from bs4 import BeautifulSoup from pymongo import MongoClient -
Connect to your MongoDB database with a MongoClient instance, using the
connect()method. You need to replaceusername,password,host, andportwith your own values.client = MongoClient('mongodb://username:password@host:port') -
Select the database and the collection you want to store the news in. If the collection does not exist yet, it will be automatically created.
db = client['sports_news'] news_collection = db['news'] -
Define a function named
get_last_news()that extracts the last 10 sports news from the yjc.ir website and returns them as a list of dictionaries. You can use therequestslibrary to get the HTML content of the website and theBeautifulSouplibrary to parse it and extract the news.def get_last_news(): url = 'https://www.yjc.ir/fa/list/12/%D9%88%D8%B1%D8%B2%D8%B4%DB%8C' r = requests.get(url) soup = BeautifulSoup(r.content, 'html.parser') news = [] for item in soup.find_all('div', class_='news'): title = item.find('a').get_text().strip() link = item.find('a').get('href') date_str = item.find('span', class_='date').get_text() date = datetime.strptime(date_str, '%Y-%m-%d %H:%M:%S') news.append({'title': title, 'link': link, 'date': date}) return news[:10] -
Define a function named
save_news(news)that inserts the news in thenews_collectioncollection, but only if they are not duplicates. In order to check for duplicates, you can query the database for the last 10 news and compare their titles and links with the new ones. If a news is already in the database, you should not insert it.def save_news(news): last_news = news_collection.find().sort('date', -1).limit(10) last_titles = set(map(lambda n: n['title'], last_news)) last_links = set(map(lambda n: n['link'], last_news)) for item in news: if item['title'] not in last_titles and item['link'] not in last_links: news_collection.insert_one(item) -
Define the main loop of your program that executes every 8 hours and calls the
get_last_news()andsave_news()functions. In order to run this loop every 8 hours, you can use thetime.sleep()function together with Python's datetime and timedelta classes, like this:while True: news = get_last_news() save_news(news) time.sleep(timedelta(hours=8).seconds)This loop will run indefinitely, getting the last 10 news and storing them in the database every 8 hours.
Here's the full code:
from datetime import datetime, timedelta import time import requests from bs4 import BeautifulSoup from pymongo import MongoClient client = MongoClient('mongodb://username:password@host:port') db = client['sports_news'] news_collection = db['news'] def get_last_news(): url = 'https://www.yjc.ir/fa/list/12/%D9%88%D8%B1%D8%B2%D8%B4%DB%8C' r = requests.get(url) soup = BeautifulSoup(r.content, 'html.parser') news = [] for item in soup.find_all('div', class_='news'): title = item.find('a').get_text().strip() link = item.find('a').get('href') date_str = item.find('span', class_='date').get_text() date = datetime.strptime(date_str, '%Y-%m-%d %H:%M:%S') news.append({'title': title, 'link': link, 'date': date}) return news[:10] def save_news(news): last_news = news_collection.find().sort('date', -1).limit(10) last_titles = set(map(lambda n: n['title'], last_news)) last_links = set(map(lambda n: n['link'], last_news)) for item in news: if item['title'] not in last_titles and item['link'] not in last_links: news_collection.insert_one(item) while True: news = get_last_news() save_news(news) time.sleep(timedelta(hours=8).seconds) -
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.