Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-opus-4-20250514-thinking-16k· AnthropicHugging Face LMArena Human Preference 140K

how to build a own web crawller

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwen3-235b-a22b’s answer.
  1. User

    how to build a own web crawller

  2. Claude Assistant

    Building a web crawler involves several key components. Here's a step-by-step guide to create a basic one:

    Core Components

    1. HTTP Client You need a way to fetch web pages. Popular libraries include:

    • Python: requests, urllib
    • JavaScript: axios, node-fetch
    • Java: HttpClient, OkHttp

    2. HTML Parser To extract data and links from pages:

    • Python: BeautifulSoup, lxml
    • JavaScript: cheerio, jsdom
    • Java: jsoup

    3. URL Management

    • Queue for URLs to visit
    • Set of visited URLs to avoid duplicates
    • URL normalization (handling relative links, parameters, etc.)

    Basic Python Example

    Here's a simple crawler using Python:

    import requests
    from bs4 import BeautifulSoup
    from urllib.parse import urljoin, urlparse
    from collections import deque
    
    class WebCrawler:
        def __init__(self, start_url, max_depth=2):
            self.start_url = start_url
            self.max_depth = max_depth
            self.visited = set()
            self.to_visit = deque([(start_url, 0)])
            
        def is_valid_url(self, url):
            parsed = urlparse(url)
            return bool(parsed.netloc) and bool(parsed.scheme)
        
        def crawl(self):
            while self.to_visit:
                url, depth = self.to_visit.popleft()
                
                if url in self.visited or depth > self.max_depth:
                    continue
                    
                try:
                    response = requests.get(url, timeout=5)
                    if response.status_code == 200:
                        self.visited.add(url)
                        print(f"Crawled: {url} (depth: {depth})")
                        
                        # Parse the page
                        soup = BeautifulSoup(response.text, 'html.parser')
                        
                        # Extract links
                        for link in soup.find_all('a', href=True):
                            absolute_url = urljoin(url, link['href'])
                            if self.is_valid_url(absolute_url):
                                self.to_visit.append((absolute_url, depth + 1))
                                
                except Exception as e:
                    print(f"Error crawling {url}: {e}")
    
    # Usage
    crawler = WebCrawler("https://example.com", max_depth=2)
    crawler.crawl()
    

    Important Considerations

    1. Respect robots.txt Check and follow the site's crawling rules:

    from urllib.robotparser import RobotFileParser
    
    rp = RobotFileParser()
    rp.set_url("https://example.com/robots.txt")
    rp.read()
    if rp.can_fetch("*", url):
        # Safe to crawl
    

    2. Rate Limiting Add delays between requests to avoid overwhelming servers:

    import time
    time.sleep(1)  # Wait 1 second between requests
    

    3. User Agent Identify your crawler properly:

    headers = {'User-Agent': 'YourBot/1.0 (your@email.com)'}
    response = requests.get(url, headers=headers)
    

    4. Handle Edge Cases

    • JavaScript-rendered pages (use Selenium or Playwright)
    • Authentication requirements
    • Different content types (PDFs, images)
    • Redirects and error pages
    • Character encoding issues

    Advanced Features

    For production crawlers, consider:

    • Multithreading/Async: Speed up crawling with concurrent requests
    • Database Storage: Store crawled data efficiently
    • Distributed Crawling: Scale across multiple machines
    • Duplicate Detection: Use content hashing to avoid storing duplicates
    • Link Analysis: Implement PageRank or similar algorithms

    Example with Async (Python)

    import asyncio
    import aiohttp
    from bs4 import BeautifulSoup
    
    async def fetch_page(session, url):
        try:
            async with session.get(url) as response:
                return await response.text()
        except Exception as e:
            print(f"Error fetching {url}: {e}")
            return None
    
    async def crawl_async(urls):
        async with aiohttp.ClientSession() as session:
            tasks = [fetch_page(session, url) for url in urls]
            pages = await asyncio.gather(*tasks)
            return pages
    

    Remember to always crawl ethically - respect rate limits, follow robots.txt, and don't overload servers. Many websites also provide APIs which are often better alternatives to crawling.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.