Write a program in Python that takes the last 10 sports news from the site yjc.ir and…
- User
Write a program in Python that takes the last 10 sports news from the site yjc.ir and stores it in the mongodb database. Then, every 3 hours, it receives the last 10 news of the site and saves it in the database. It checks the messages and does not save the messages that are repeated. Please explain the steps in order. This program stores the content of each news page in the database. For example, link to photo, news text, news URL, news date, etc.
- Assistant
Sure! Here's a step-by-step explanation of how you can achieve this:
Step 1: Install Required Libraries First, you need to install the required libraries. You'll need the pymongo library to interact with the MongoDB database, and the requests and BeautifulSoup libraries to scrape the website's HTML content.
You can install these libraries using pip:
pip install pymongo requests beautifulsoup4Step 2: Import Required Libraries Now, import the required libraries in your Python script:
import requests from bs4 import BeautifulSoup from pymongo import MongoClient import datetime import timeStep 3: Connect to MongoDB Connect to your MongoDB database using the MongoClient class:
client = MongoClient("mongodb://localhost:27017/") # Replace with your MongoDB connection URL db = client["newsdb"] # Replace with your database name collection = db["news"] # Replace with your collection nameStep 4: Scrape and Store Initial News Write a function to scrape the initial 10 news from the desired website and store them in the MongoDB collection.
def scrape_and_store_news(): url = "http://yjc.ir" response = requests.get(url) soup = BeautifulSoup(response.content, "html.parser") news_elements = soup.select(".news-list ul li")[:10] # Adjust the selector as per the website's structure for news_element in news_elements: # Extract necessary information from the news element link = news_element.select_one("a")["href"] photo = news_element.select_one("img")["src"] text = news_element.select_one(".text").get_text(strip=True) news_date = datetime.datetime.now() # Check if the news already exists in the collection if collection.count_documents({"text": text}, limit=1) == 0: news = {"photo": photo, "text": text, "link": link, "date": news_date} collection.insert_one(news)Call this function once to scrape and store the initial 10 news:
scrape_and_store_news()Step 5: Periodically Scrape and Store News Now, write a loop that runs every 3 hours and scrapes the last 10 news from the website, excluding any already stored news.
while True: scrape_and_store_news() time.sleep(10800) # Sleep for 3 hours before running againThat's it! This program will scrape the last 10 news from the yjc.ir website, store them in the MongoDB database, and run every 3 hours to update the database with new news while ignoring duplicates. You can further modify the logic and database structure based on your requirements.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.