Skip to content
TheChatArchiveThe Chat Archive
gpt-4-1106-preview· OpenAIAi2 WildChat-4.8M (AI2)

Help me build a Wikipedia webscraper with python to extract articles as text

  1. User

    Help me build a Wikipedia webscraper with python to extract articles as text

  2. Assistant

    Building a Wikipedia scraper with Python to extract articles as text involves several steps:

    1. Choose a scraping library: To scrape websites, you can use libraries like requests to make HTTP requests and BeautifulSoup or lxml to parse HTML.

    2. Abide by Wikipedia's scraping rules: Wikipedia has a User-Agent policy and an API. Instead of scraping the website directly, it's better to use the API to fetch articles.

    3. Get the right URLs or page titles: To fetch articles, you'll need a list of page titles or URLs you want to scrape.

    Below, I will guide you through building a simple scraper using Wikipedia's API. For educational purposes, the example focuses on fetching text for a specific article:

    import requests
    
    def get_wikipedia_article(page_title):
        """
        Fetches the text content of a Wikipedia article.
        
        Args:
            page_title (str): title of the Wikipedia page to fetch.
        
        Returns:
            str: text content of the page.
        """
        # URL Endpoint for the Wikipedia API
        ENDPOINT = 'https://en.wikipedia.org/w/api.php'
        
        # Parameters for the API request
        params = {
            'action': 'query',
            'format': 'json',
            'titles': page_title,
            'prop': 'extracts',
            'exlimit': 1,
            'explaintext': True,
            'formatversion': 2
        }
        
        response = requests.get(ENDPOINT, params=params)
        data = response.json()
        
        # Extracting the page content
        pages = data.get('query', {}).get('pages', [])
        if pages:
            page_text = pages[0].get('extract', '')
            return page_text
        else:
            return 'Article not found.'
    
    # Example usage:
    page_title = 'Python (programming language)'
    article_text = get_wikipedia_article(page_title)
    
    # Output the article text
    print(article_text)
    

    The script uses requests to fetch data and utilizes Wikipedia's API endpoints. We are requesting the extract of a specific page in plain text. The explaintext parameter ensures that we get just the text of the page without HTML or wiki markup.

    Remember: When you are using APIs or scrapers, you must respect the website's terms of service, rate limits, and legal considerations. Wikipedia provides an API with endpoints designed to be accessed programmatically, thus making it unnecessary and discouraged to scrape the website directly.

    For more complex scraping jobs or a large number of articles, consider using the pywikibot library, which is a tool specifically designed for interacting with MediaWiki sites (like Wikipedia).

    Before you deploy any scraper, make sure to check the latest rules on Wikipedia. Wikipedia's terms of use and API guidelines are available at:

    Always use official APIs when possible to get data rather than scraping, as it is more reliable and respectful to the service providers.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.