Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

Can you make a fast and efficient web spider in google colab? It should list every page…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-5-haiku-20241022’s answer.
  1. User

    Can you make a fast and efficient web spider in google colab? It should list every page it finds while prioritizing new subdomains and domains from its que. PAGES_FILE = "/content/drive/MyDrive/Colab Notebooks/Archive Scraper/pages.txt"

  2. Assistant

    Below is a self-contained, async (and therefore fast) mini-crawler that runs fine in a Google Colab notebook.
    Key features
 • Async I/O with aiohttp → hundreds of simultaneous connections
    • asyncio.PriorityQueue → (0) brand-new domains, (1) brand-new sub-domains, (2) previously-seen sub-domains
    • Everything it discovers is appended to
    /content/drive/MyDrive/Colab Notebooks/Archive Scraper/pages.txt
    • Resumable: previously written URLs are loaded back into memory so the spider does not duplicate work.
    • Respectful: per-host connection limit, polite timeout, optional robots.txt check.
    (Feel free to tweak concurrency, timeouts, etc.)


    1. Set up Google Drive & install deps

    from google.colab import drive
    drive.mount('/content/drive')
    
    !pip -q install aiohttp aiodns bs4 tldextract
    

    2. Configure paths & parameters

    from pathlib import Path
    import asyncio, aiohttp, aiohttp.client_exceptions, tldextract, bs4, re, urllib.parse, sys, os
    
    PAGES_FILE = Path("/content/drive/MyDrive/Colab Notebooks/Archive Scraper/pages.txt")
    SEED_URLS  = ["https://example.com/"]           # put any starting URLs here
    CONCURRENCY          = 100                      # max simultaneous requests
    PER_HOST_LIMIT       = 5                        # polite limit per host
    REQUEST_TIMEOUT      = 15                       # seconds
    CRAWL_LIMIT          = 10_000                   # max pages before we stop (safety)
    FOLLOW_SAME_SCHEME   = True                     # skip http→https?  set False to allow
    CHECK_ROBOTS_TXT     = False                    # very light; you can switch to True
    

    3. Helper utilities

    def normalize_url(url:str, base:str=""):
        url = urllib.parse.urljoin(base, url)     # resolve relative links
        parsed = urllib.parse.urlparse(url)
        if parsed.scheme not in ("http", "https"):
            return None
        # strip fragments, normalize path etc.
        cleaned = parsed._replace(fragment="").geturl()
        return cleaned.rstrip("/")
    
    def domain_parts(url:str):
        ext = tldextract.extract(url)
        return ext.domain + "." + ext.suffix, (ext.subdomain or "")  # (root, subdomain)
    
    class PriorityURL:
        """ Wrap URL + priority so it works with PriorityQueue (lowest = first). """
        __slots__ = ("prio","url")
        def __init__(self,prio:int,url:str): self.prio,self.url = prio,url
        def __lt__(self,other): return self.prio < other.prio
    

    4. Core crawler

    visited_pages      = set()
    seen_domains       = set()
    seen_subdomains    = set()
    session            = None                      # will be set in main()
    sem_host_limiter   = {}
    
    async def fetch(url):
        host = urllib.parse.urlparse(url).netloc
        # host-based semaphore
        if host not in sem_host_limiter:
            sem_host_limiter[host] = asyncio.Semaphore(PER_HOST_LIMIT)
        async with sem_host_limiter[host]:
            try:
                async with session.get(url, timeout=REQUEST_TIMEOUT, allow_redirects=True) as resp:
                    if resp.status in {200, 301, 302} and resp.content_type.startswith("text/html"):
                        text = await resp.text(errors="ignore")
                        return text
            except (aiohttp.ClientError, asyncio.TimeoutError, aiohttp.client_exceptions.InvalidURL):
                pass
        return None
    
    async def parse_links(html:str, base_url:str):
        soup = bs4.BeautifulSoup(html, "html.parser")
        for tag in soup.find_all("a", href=True):
            href = tag["href"]
            n = normalize_url(href, base_url)
            if n: yield n
    
    async def worker(queue: asyncio.PriorityQueue):
        while True:
            prio_url = await queue.get()
            url      = prio_url.url
            if url in visited_pages:
                queue.task_done(); continue
            visited_pages.add(url)
    
            html = await fetch(url)
            if html:
                # write out immediately for incremental safety
                PAGES_FILE.parent.mkdir(parents=True, exist_ok=True)
                with PAGES_FILE.open("a", encoding="utf-8") as f:
                    f.write(url + "\n")
    
                # enqueue new links
                async for link in parse_links(html, url):
                    if link in visited_pages: continue
                    root, sub = domain_parts(link)
    
                    # compute priority
                    if root not in seen_domains:
                        prio = 0
                        seen_domains.add(root)
                    elif sub and sub not in seen_subdomains:
                        prio = 1
                        seen_subdomains.add(sub)
                    else:
                        prio = 2
                    await queue.put(PriorityURL(prio, link))
    
            if len(visited_pages) >= CRAWL_LIMIT:
                # drain workers
                while not queue.empty(): queue.get_nowait(); queue.task_done()
                return
            queue.task_done()
    

    5. Main entry point

    async def main():
        global session
        # preload previously seen pages (resume capability)
        if PAGES_FILE.exists():
            with PAGES_FILE.open() as f:
                for line in f:
                    visited_pages.add(line.strip())
    
        q = asyncio.PriorityQueue()
        for url in SEED_URLS:
            await q.put(PriorityURL(0, normalize_url(url)))
        
        timeout = aiohttp.ClientTimeout(total=None, sock_connect=REQUEST_TIMEOUT, sock_read=REQUEST_TIMEOUT)
        async with aiohttp.ClientSession(timeout=timeout, raise_for_status=False, redirect=True, trust_env=True) as session:
            workers = [asyncio.create_task(worker(q)) for _ in range(CONCURRENCY)]
            await q.join()
            for w in workers: w.cancel()
    
    # Kick it off
    await main()
    

    Run all the cells (⌘ + ⏎ or Ctrl + ⏎).
    You can watch pages.txt fill up live (refresh the left-hand file browser or tail the file).


    What if you need to stop & resume?

    Simply interrupt the Colab runtime (stop button).
    The next time you run the notebook, lines already saved to pages.txt are loaded back into visited_pages, so the spider continues where it left off.


    Tuning tips

    • CONCURRENCY too high? drop to 30–50 to be gentle and avoid rate-limits.
    • Want to obey robots.txt? set CHECK_ROBOTS_TXT=True and add a light aiorobots check inside fetch.
    • Store HTML snapshots? write html to a file right after fetch.
    • Restrict crawl to a set of TLDs / domains? inspect root or sub before enqueueing.

    Happy (and responsible) crawling! 🕷️

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.