Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro-preview-05-06· GoogleHugging Face LMArena Human Preference 140K

Can you make a fast and efficient web spider in google colab? It should list every page…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with grok-3-mini-beta’s answer.
  1. User

    Can you make a fast and efficient web spider in google colab? It should list every page it finds while prioritizing new subdomains and domains from its que. PAGES_FILE = "/content/drive/MyDrive/Colab Notebooks/Archive Scraper/pages.txt"

  2. Gemini Assistant

    Okay, let's build a fast and efficient web spider in Colab. This spider will:

    1. Use aiohttp for asynchronous HTTP requests (fast!).
    2. Use BeautifulSoup for parsing HTML.
    3. Use a collections.deque for the URL queue.
    4. Prioritize new domains and subdomains by adding them to the front of the queue.
    5. Store all discovered, valid, and unique URLs to the specified PAGES_FILE.
    6. Include basic politeness (timeout, user-agent).
    7. Handle common errors gracefully.
    8. Allow limiting the number of pages to crawl.

    First, ensure your Google Drive is mounted:

    from google.colab import drive
    drive.mount('/content/drive')
    

    Now, the spider code:

    import asyncio
    import aiohttp
    from bs4 import BeautifulSoup
    from urllib.parse import urlparse, urljoin
    from collections import deque
    import os
    import re
    import time
    
    # --- Configuration ---
    PAGES_FILE = "/content/drive/MyDrive/Colab Notebooks/Archive Scraper/pages.txt"
    START_URLS = [
        "http://example.com",
        "https://quotes.toscrape.com/",
        # Add more diverse starting points for better discovery
    ]
    MAX_PAGES_TO_CRAWL = 1000  # Safety limit
    MAX_CONCURRENT_REQUESTS = 10 # How many requests to make at once
    REQUEST_TIMEOUT = 10  # Seconds
    USER_AGENT = "ColabSpider/1.0 (+http://your-contact-info.com)" # Be a good bot!
    
    # Create directory for PAGES_FILE if it doesn't exist
    os.makedirs(os.path.dirname(PAGES_FILE), exist_ok=True)
    
    # --- Helper Functions ---
    
    def get_domain(url):
        """Extracts the main domain (e.g., example.com) from a URL."""
        try:
            parsed_url = urlparse(url)
            domain_parts = parsed_url.netloc.split('.')
            if len(domain_parts) > 1:
                # Handles cases like example.com, co.uk
                return ".".join(domain_parts[-2 if domain_parts[-2] != 'co' else -3:])
            return parsed_url.netloc
        except Exception:
            return None
    
    def get_netloc(url):
        """Extracts the network location (e.g., sub.example.com) from a URL."""
        try:
            return urlparse(url).netloc
        except Exception:
            return None
    
    def normalize_url(url, base_url):
        """
        Normalizes a URL:
        1. Makes it absolute.
        2. Removes fragments (#).
        3. Lowercases scheme and netloc.
        """
        joined_url = urljoin(base_url, url)
        parsed = urlparse(joined_url)
        # Reconstruct without fragment, with lowercase scheme and netloc
        normalized = parsed._replace(scheme=parsed.scheme.lower(),
                                     netloc=parsed.netloc.lower(),
                                     fragment="").geturl()
        return normalized
    
    # --- Core Spider Logic ---
    
    async def fetch_page(session, url):
        """Fetches a single page."""
        try:
            async with session.get(url, timeout=REQUEST_TIMEOUT, headers={'User-Agent': USER_AGENT}) as response:
                if response.status == 200 and "text/html" in response.headers.get('Content-Type', '').lower():
                    return await response.text(), response.url # Return final URL after redirects
                else:
                    print(f"[{response.status}] Non-HTML or error for {url}")
                    return None, url
        except aiohttp.ClientError as e:
            print(f"Network error fetching {url}: {e}")
            return None, url
        except asyncio.TimeoutError:
            print(f"Timeout fetching {url}")
            return None, url
        except Exception as e:
            print(f"Unexpected error fetching {url}: {e}")
            return None, url
    
    async def process_url(url_to_crawl, session, queue, visited_urls, seen_domains, seen_subdomains, pages_file_handle, crawled_count_lock, crawled_count):
        """Processes a single URL: fetches, parses, adds new links to queue."""
        if url_to_crawl in visited_urls:
            return
    
        async with crawled_count_lock:
            if crawled_count['value'] >= MAX_PAGES_TO_CRAWL:
                return
            crawled_count['value'] += 1
            current_crawl_num = crawled_count['value']
    
        print(f"[{current_crawl_num}/{MAX_PAGES_TO_CRAWL}] Crawling: {url_to_crawl}")
        visited_urls.add(url_to_crawl)
    
        html_content, final_url = await fetch_page(session, url_to_crawl)
        final_url_str = str(final_url) # Convert CIMultiDictProxy to string if needed
    
        if final_url_str != url_to_crawl: # Handle redirects
            if final_url_str in visited_urls:
                return # Already visited the redirected target
            print(f"Redirected: {url_to_crawl} -> {final_url_str}")
            visited_urls.add(final_url_str) # Mark redirected URL as visited
            url_to_crawl = final_url_str   # Process the final URL
    
        # Add to pages file (even if fetching fails, it was a URL we tried)
        # We add the *original* URL we intended to crawl, or the final one if redirected.
        # Filter out non-http/https URLs here
        if url_to_crawl.startswith("http"):
            pages_file_handle.write(f"{url_to_crawl}\n")
            pages_file_handle.flush() # Ensure it's written immediately
    
        if not html_content:
            return
    
        soup = BeautifulSoup(html_content, 'html.parser')
        current_domain = get_domain(url_to_crawl)
        current_netloc = get_netloc(url_to_crawl)
    
        if current_domain and current_domain not in seen_domains:
            seen_domains.add(current_domain)
            print(f"Discovered new domain: {current_domain}")
        if current_netloc and current_netloc not in seen_subdomains:
            seen_subdomains.add(current_netloc)
            if current_netloc != current_domain: # Avoid double printing for root domain
                 print(f"Discovered new subdomain: {current_netloc}")
    
    
        for link in soup.find_all('a', href=True):
            raw_href = link['href']
            abs_url = normalize_url(raw_href, url_to_crawl)
    
            if not abs_url.startswith("http"): # Only interested in http/https
                continue
    
            if abs_url not in visited_urls and abs_url not in queue:
                new_url_domain = get_domain(abs_url)
                new_url_netloc = get_netloc(abs_url)
    
                if not new_url_domain or not new_url_netloc: # Skip malformed URLs
                    continue
    
                # Prioritization logic
                if new_url_domain not in seen_domains:
                    queue.appendleft(abs_url) # New domain, highest priority
                elif new_url_netloc not in seen_subdomains:
                    queue.appendleft(abs_url) # New subdomain of known domain, high priority
                else:
                    queue.append(abs_url) # Same domain/subdomain, normal priority
    
    
    async def worker(name, queue, session, visited_urls, seen_domains, seen_subdomains, pages_file_handle, crawled_count_lock, crawled_count, semaphore):
        """Worker task that continuously fetches URLs from the queue."""
        while True:
            async with crawled_count_lock:
                if crawled_count['value'] >= MAX_PAGES_TO_CRAWL and not queue: # Stop if limit reached and queue empty
                    break
            
            try:
                # Wait for a URL to be available, with a timeout to allow checking crawl limit
                url_to_crawl = await asyncio.wait_for(queue.popleft_async(), timeout=1.0)
            except (asyncio.TimeoutError, IndexError): # IndexError if deque is empty (popleft_async might not exist, handle manually)
                if not queue: # Check if queue is actually empty
                     async with crawled_count_lock: # Re-check limit if queue is empty
                        if crawled_count['value'] >= MAX_PAGES_TO_CRAWL:
                            break # Exit worker
                     await asyncio.sleep(0.1) # Small sleep if queue is empty but limit not reached
                     continue
                else: # Queue not empty, but popleft_async timed out or other issue (should be rare)
                    continue
    
    
            # Acquire semaphore before processing
            async with semaphore:
                await process_url(url_to_crawl, session, queue, visited_urls, seen_domains, seen_subdomains, pages_file_handle, crawled_count_lock, crawled_count)
                # A small politeness delay, can be adjusted or removed if rate limiting is handled by semaphore alone
                await asyncio.sleep(0.1) # Small delay between requests from the same worker
    
    
    # Custom Deque with async pop
    class AsyncDeque(deque):
        def __init__(self, *args, **kwargs):
            super().__init__(*args, **kwargs)
            self._has_items = asyncio.Condition()
    
        async def append_async(self, item):
            super().append(item)
            async with self._has_items:
                self._has_items.notify()
    
        async def appendleft_async(self, item):
            super().appendleft(item)
            async with self._has_items:
                self._has_items.notify()
    
        async def popleft_async(self):
            async with self._has_items:
                while not self:
                    await self._has_items.wait()
                return self.popleft()
    
        # Add other methods like append, appendleft if you need them to also notify
        def append(self, item):
            super().append(item)
            # No async notification needed for sync append used internally by process_url
            # If external code were to use append and expect workers to pick up, it would need notify.
    
        def appendleft(self, item):
            super().appendleft(item)
            # Same as above.
    
    async def main():
        start_time = time.time()
    
        # Initialize data structures
        # queue = deque() # Standard deque
        queue = AsyncDeque() # Use our async-aware deque
        visited_urls = set()
        seen_domains = set()
        seen_subdomains = set()
    
        # For managing crawled count across tasks
        crawled_count = {'value': 0}
        crawled_count_lock = asyncio.Lock() # Lock for updating crawled_count
    
        # Add initial URLs to the queue and seen sets
        for url in START_URLS:
            normalized_start_url = normalize_url(url, url) # Normalize start URLs too
            if normalized_start_url:
                queue.append(normalized_start_url) # Use normal append for initial setup
                # visited_urls.add(normalized_start_url) # Don't add to visited yet, let processor do it
                
                # Pre-populate seen domains/subdomains from start URLs for correct initial prioritization
                s_domain = get_domain(normalized_start_url)
                s_netloc = get_netloc(normalized_start_url)
                if s_domain: seen_domains.add(s_domain)
                if s_netloc: seen_subdomains.add(s_netloc)
    
    
        # Semaphore to limit concurrent requests
        semaphore = asyncio.Semaphore(MAX_CONCURRENT_REQUESTS)
    
        print(f"Starting crawl. Outputting to: {PAGES_FILE}")
        print(f"Max pages to crawl: {MAX_PAGES_TO_CRAWL}")
        print(f"Max concurrent requests: {MAX_CONCURRENT_REQUESTS}")
        print(f"Initial queue size: {len(queue)}")
    
        try:
            with open(PAGES_FILE, "a", encoding="utf-8") as pages_file_handle:
                async with aiohttp.ClientSession() as session:
                    workers = [
                        asyncio.create_task(
                            worker(f"Worker-{i}", queue, session, visited_urls, seen_domains, seen_subdomains, pages_file_handle, crawled_count_lock, crawled_count, semaphore)
                        )
                        for i in range(MAX_CONCURRENT_REQUESTS)
                    ]
    
                    # Wait for all workers to complete
                    # This will happen when queue is empty and MAX_PAGES_TO_CRAWL is reached
                    # Or if an unhandled exception occurs in a worker (which it shouldn't with current try/excepts)
                    await asyncio.gather(*workers, return_exceptions=True) # return_exceptions=True helps debug worker issues
    
        except Exception as e:
            print(f"An error occurred in main: {e}")
        finally:
            duration = time.time() - start_time
            print(f"\n--- Crawl Finished ---")
            print(f"Total URLs visited (approx): {len(visited_urls)}")
            print(f"Total unique domains seen: {len(seen_domains)}")
            print(f"Total unique subdomains seen: {len(seen_subdomains)}")
            print(f"Total URLs written to {PAGES_FILE}")
            print(f"Duration: {duration:.2f} seconds")
    
            # Count lines in file for verification
            try:
                with open(PAGES_FILE, "r", encoding="utf-8") as pf:
                    line_count = sum(1 for _ in pf)
                print(f"Lines in {PAGES_FILE}: {line_count}")
            except FileNotFoundError:
                print(f"{PAGES_FILE} not found (no pages written or error).")
    
    if __name__ == "__main__":
        # In Colab, you run this with:
        # await main()
        # For local .py file:
        # asyncio.run(main())
        pass
    

    How to run in Google Colab:

    1. Make sure you've run the Drive mount cell.
    2. Create a new code cell.
    3. Paste the entire spider code into this cell.
    4. Add the following line at the very end of that cell to execute the main coroutine:
      await main()
      
    5. Run the cell.

    Explanation and Key Features:

    1. Asynchronous Operations (aiohttp, asyncio):

      • aiohttp.ClientSession(): Manages a pool of connections for efficient HTTP requests.
      • async def / await: Allows multiple network operations to happen concurrently without blocking the entire program.
      • asyncio.Semaphore(MAX_CONCURRENT_REQUESTS): Limits how many process_url coroutines can actively make network requests at the same time, preventing you from overwhelming servers or your own network.
      • AsyncDeque: A custom deque subclass that uses asyncio.Condition to allow workers to await new items if the queue is empty. This is more efficient than busy-waiting.
    2. URL Queue (AsyncDeque):

      • Stores URLs yet to be crawled.
      • appendleft(url): Adds high-priority URLs (new domains/subdomains) to the front.
      • append(url): Adds normal-priority URLs (same domain/subdomain) to the back.
      • popleft_async(): Workers get URLs from the front.
    3. Visited Tracking:

      • visited_urls (a set): Stores all URLs that have been initiated for processing (or successfully redirected to). This prevents re-crawling and getting into loops. O(1) average time complexity for lookups.
    4. Domain/Subdomain Tracking:

      • seen_domains (a set): Tracks unique parent domains encountered (e.g., example.com).
      • seen_subdomains (a set): Tracks unique network locations / subdomains (e.g., blog.example.com, www.example.com).
      • These are used for the prioritization logic.
    5. Prioritization Logic (in process_url):

      • When a new link is found:
        • If its domain is not in seen_domains, it's a brand new domain: highest priority (queue.appendleft()).
        • Else, if its netloc (subdomain) is not in seen_subdomains, it's a new subdomain on an existing domain: high priority (queue.appendleft()).
        • Otherwise, it's a page on an already seen subdomain/domain: normal priority (queue.append()).
    6. URL Normalization (normalize_url):

      • Converts relative links (/about.html) to absolute links (http://example.com/about.html).
      • Removes URL fragments (#section-name) as they typically point to the same page content.
      • Lowercases scheme and netloc for consistency, as HTTP://Example.COM is the same as http://example.com.
    7. Output (PAGES_FILE):

      • All unique, valid (starts with http/https) URLs that are attempted for crawling (or their final redirect targets) are written to this file, one URL per line.
      • The file is opened in append mode ("a") so previous crawl results are kept if the script is run multiple times.
      • pages_file_handle.flush() is used to ensure data is written to disk promptly.
    8. Error Handling & Politeness:

      • try-except blocks handle network errors, timeouts, and parsing issues.
      • USER_AGENT is set to identify your bot. Change your-contact-info.com to something relevant.
      • REQUEST_TIMEOUT prevents hanging indefinitely on slow servers.
      • A small await asyncio.sleep(0.1) is added after each page processing within a worker. This, combined with the semaphore, helps in being gentle.
    9. Configuration:

      • START_URLS: Where the spider begins its journey.
      • MAX_PAGES_TO_CRAWL: A crucial safety net to prevent accidental infinite crawls or crawling too much.
      • MAX_CONCURRENT_REQUESTS: Balances speed with resource usage and server load.
      • REQUEST_TIMEOUT: How long to wait for a server response.

    This spider is significantly faster than a synchronous one and implements the requested prioritization. Remember that web scraping should be done responsibly, respecting robots.txt (not implemented here for brevity but important for large-scale or public-facing spiders) and server load.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.