Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0613· openaiHugging Face Suzhen/CodeChat

Write a Python program that crawls data from yjc.ir based on tags or titles or topics.…

  1. User

    Write a Python program that crawls data from yjc.ir based on tags or titles or topics. The program receives input from the user and crawls the data as requested by the user.

  2. Assistant

    To crawl data from yjc.ir based on tags, titles, or topics, you can use the requests and BeautifulSoup libraries in Python. Here's an example program that receives user input and fetches data from the website accordingly.

    import requests
    from bs4 import BeautifulSoup
    
    def crawl_data(tag=None, title=None, topic=None):
        base_url = "https://www.yjc.ir"
    
        # Creating the search query based on user input
        if tag:
            url = f"{base_url}/fa/tags/{tag}"
        elif title:
            url = f"{base_url}/fa/news/{title}"
        elif topic:
            url = f"{base_url}/fa/news/{topic}"
        else:
            print("No search parameter provided.")
            return
    
        # Sending a GET request to the website
        response = requests.get(url)
        if response.status_code == 200:
            soup = BeautifulSoup(response.content, "html.parser")
    
            # Parsing the crawled data based on the website structure
            if tag or title:
                news_items = soup.find_all("div", class_="title")
                for item in news_items:
                    print(item.text.strip())
            elif topic:
                news_items = soup.find_all("div", class_="news-title")
                for item in news_items:
                    print(item.text.strip())
        else:
            print("Failed to fetch data from the website.")
    
    # Taking user input to specify the search
    search_type = input("Enter the search type (tag, title, topic): ")
    if search_type == "tag":
        tag = input("Enter the tag: ")
        crawl_data(tag=tag)
    elif search_type == "title":
        title = input("Enter the title: ")
        crawl_data(title=title)
    elif search_type == "topic":
        topic = input("Enter the topic: ")
        crawl_data(topic=topic)
    else:
        print("Invalid search type.")
    

    In this program, the crawl_data() function takes user input (tag, title, or topic) to construct the search query and fetch the data from yjc.ir accordingly. The function uses the BeautifulSoup library to parse the HTML content of the website and extract the desired information. Finally, it prints the extracted data.

    The program prompts the user to specify the search type and required parameter, then calls the crawl_data() function accordingly.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.