Can you make a fast and efficient web spider in google colab? It should list every page…
- User
Can you make a fast and efficient web spider in google colab? It should list every page it finds while prioritizing new subdomains and domains from its que. PAGES_FILE = "/content/drive/MyDrive/Colab Notebooks/Archive Scraper/pages.txt"
Assistant
Okay, let's build a fast and efficient web spider in Colab. This spider will:
- Use
aiohttpfor asynchronous HTTP requests (fast!). - Use
BeautifulSoupfor parsing HTML. - Use a
collections.dequefor the URL queue. - Prioritize new domains and subdomains by adding them to the front of the queue.
- Store all discovered, valid, and unique URLs to the specified
PAGES_FILE. - Include basic politeness (timeout, user-agent).
- Handle common errors gracefully.
- Allow limiting the number of pages to crawl.
First, ensure your Google Drive is mounted:
from google.colab import drive drive.mount('/content/drive')Now, the spider code:
import asyncio import aiohttp from bs4 import BeautifulSoup from urllib.parse import urlparse, urljoin from collections import deque import os import re import time # --- Configuration --- PAGES_FILE = "/content/drive/MyDrive/Colab Notebooks/Archive Scraper/pages.txt" START_URLS = [ "http://example.com", "https://quotes.toscrape.com/", # Add more diverse starting points for better discovery ] MAX_PAGES_TO_CRAWL = 1000 # Safety limit MAX_CONCURRENT_REQUESTS = 10 # How many requests to make at once REQUEST_TIMEOUT = 10 # Seconds USER_AGENT = "ColabSpider/1.0 (+http://your-contact-info.com)" # Be a good bot! # Create directory for PAGES_FILE if it doesn't exist os.makedirs(os.path.dirname(PAGES_FILE), exist_ok=True) # --- Helper Functions --- def get_domain(url): """Extracts the main domain (e.g., example.com) from a URL.""" try: parsed_url = urlparse(url) domain_parts = parsed_url.netloc.split('.') if len(domain_parts) > 1: # Handles cases like example.com, co.uk return ".".join(domain_parts[-2 if domain_parts[-2] != 'co' else -3:]) return parsed_url.netloc except Exception: return None def get_netloc(url): """Extracts the network location (e.g., sub.example.com) from a URL.""" try: return urlparse(url).netloc except Exception: return None def normalize_url(url, base_url): """ Normalizes a URL: 1. Makes it absolute. 2. Removes fragments (#). 3. Lowercases scheme and netloc. """ joined_url = urljoin(base_url, url) parsed = urlparse(joined_url) # Reconstruct without fragment, with lowercase scheme and netloc normalized = parsed._replace(scheme=parsed.scheme.lower(), netloc=parsed.netloc.lower(), fragment="").geturl() return normalized # --- Core Spider Logic --- async def fetch_page(session, url): """Fetches a single page.""" try: async with session.get(url, timeout=REQUEST_TIMEOUT, headers={'User-Agent': USER_AGENT}) as response: if response.status == 200 and "text/html" in response.headers.get('Content-Type', '').lower(): return await response.text(), response.url # Return final URL after redirects else: print(f"[{response.status}] Non-HTML or error for {url}") return None, url except aiohttp.ClientError as e: print(f"Network error fetching {url}: {e}") return None, url except asyncio.TimeoutError: print(f"Timeout fetching {url}") return None, url except Exception as e: print(f"Unexpected error fetching {url}: {e}") return None, url async def process_url(url_to_crawl, session, queue, visited_urls, seen_domains, seen_subdomains, pages_file_handle, crawled_count_lock, crawled_count): """Processes a single URL: fetches, parses, adds new links to queue.""" if url_to_crawl in visited_urls: return async with crawled_count_lock: if crawled_count['value'] >= MAX_PAGES_TO_CRAWL: return crawled_count['value'] += 1 current_crawl_num = crawled_count['value'] print(f"[{current_crawl_num}/{MAX_PAGES_TO_CRAWL}] Crawling: {url_to_crawl}") visited_urls.add(url_to_crawl) html_content, final_url = await fetch_page(session, url_to_crawl) final_url_str = str(final_url) # Convert CIMultiDictProxy to string if needed if final_url_str != url_to_crawl: # Handle redirects if final_url_str in visited_urls: return # Already visited the redirected target print(f"Redirected: {url_to_crawl} -> {final_url_str}") visited_urls.add(final_url_str) # Mark redirected URL as visited url_to_crawl = final_url_str # Process the final URL # Add to pages file (even if fetching fails, it was a URL we tried) # We add the *original* URL we intended to crawl, or the final one if redirected. # Filter out non-http/https URLs here if url_to_crawl.startswith("http"): pages_file_handle.write(f"{url_to_crawl}\n") pages_file_handle.flush() # Ensure it's written immediately if not html_content: return soup = BeautifulSoup(html_content, 'html.parser') current_domain = get_domain(url_to_crawl) current_netloc = get_netloc(url_to_crawl) if current_domain and current_domain not in seen_domains: seen_domains.add(current_domain) print(f"Discovered new domain: {current_domain}") if current_netloc and current_netloc not in seen_subdomains: seen_subdomains.add(current_netloc) if current_netloc != current_domain: # Avoid double printing for root domain print(f"Discovered new subdomain: {current_netloc}") for link in soup.find_all('a', href=True): raw_href = link['href'] abs_url = normalize_url(raw_href, url_to_crawl) if not abs_url.startswith("http"): # Only interested in http/https continue if abs_url not in visited_urls and abs_url not in queue: new_url_domain = get_domain(abs_url) new_url_netloc = get_netloc(abs_url) if not new_url_domain or not new_url_netloc: # Skip malformed URLs continue # Prioritization logic if new_url_domain not in seen_domains: queue.appendleft(abs_url) # New domain, highest priority elif new_url_netloc not in seen_subdomains: queue.appendleft(abs_url) # New subdomain of known domain, high priority else: queue.append(abs_url) # Same domain/subdomain, normal priority async def worker(name, queue, session, visited_urls, seen_domains, seen_subdomains, pages_file_handle, crawled_count_lock, crawled_count, semaphore): """Worker task that continuously fetches URLs from the queue.""" while True: async with crawled_count_lock: if crawled_count['value'] >= MAX_PAGES_TO_CRAWL and not queue: # Stop if limit reached and queue empty break try: # Wait for a URL to be available, with a timeout to allow checking crawl limit url_to_crawl = await asyncio.wait_for(queue.popleft_async(), timeout=1.0) except (asyncio.TimeoutError, IndexError): # IndexError if deque is empty (popleft_async might not exist, handle manually) if not queue: # Check if queue is actually empty async with crawled_count_lock: # Re-check limit if queue is empty if crawled_count['value'] >= MAX_PAGES_TO_CRAWL: break # Exit worker await asyncio.sleep(0.1) # Small sleep if queue is empty but limit not reached continue else: # Queue not empty, but popleft_async timed out or other issue (should be rare) continue # Acquire semaphore before processing async with semaphore: await process_url(url_to_crawl, session, queue, visited_urls, seen_domains, seen_subdomains, pages_file_handle, crawled_count_lock, crawled_count) # A small politeness delay, can be adjusted or removed if rate limiting is handled by semaphore alone await asyncio.sleep(0.1) # Small delay between requests from the same worker # Custom Deque with async pop class AsyncDeque(deque): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self._has_items = asyncio.Condition() async def append_async(self, item): super().append(item) async with self._has_items: self._has_items.notify() async def appendleft_async(self, item): super().appendleft(item) async with self._has_items: self._has_items.notify() async def popleft_async(self): async with self._has_items: while not self: await self._has_items.wait() return self.popleft() # Add other methods like append, appendleft if you need them to also notify def append(self, item): super().append(item) # No async notification needed for sync append used internally by process_url # If external code were to use append and expect workers to pick up, it would need notify. def appendleft(self, item): super().appendleft(item) # Same as above. async def main(): start_time = time.time() # Initialize data structures # queue = deque() # Standard deque queue = AsyncDeque() # Use our async-aware deque visited_urls = set() seen_domains = set() seen_subdomains = set() # For managing crawled count across tasks crawled_count = {'value': 0} crawled_count_lock = asyncio.Lock() # Lock for updating crawled_count # Add initial URLs to the queue and seen sets for url in START_URLS: normalized_start_url = normalize_url(url, url) # Normalize start URLs too if normalized_start_url: queue.append(normalized_start_url) # Use normal append for initial setup # visited_urls.add(normalized_start_url) # Don't add to visited yet, let processor do it # Pre-populate seen domains/subdomains from start URLs for correct initial prioritization s_domain = get_domain(normalized_start_url) s_netloc = get_netloc(normalized_start_url) if s_domain: seen_domains.add(s_domain) if s_netloc: seen_subdomains.add(s_netloc) # Semaphore to limit concurrent requests semaphore = asyncio.Semaphore(MAX_CONCURRENT_REQUESTS) print(f"Starting crawl. Outputting to: {PAGES_FILE}") print(f"Max pages to crawl: {MAX_PAGES_TO_CRAWL}") print(f"Max concurrent requests: {MAX_CONCURRENT_REQUESTS}") print(f"Initial queue size: {len(queue)}") try: with open(PAGES_FILE, "a", encoding="utf-8") as pages_file_handle: async with aiohttp.ClientSession() as session: workers = [ asyncio.create_task( worker(f"Worker-{i}", queue, session, visited_urls, seen_domains, seen_subdomains, pages_file_handle, crawled_count_lock, crawled_count, semaphore) ) for i in range(MAX_CONCURRENT_REQUESTS) ] # Wait for all workers to complete # This will happen when queue is empty and MAX_PAGES_TO_CRAWL is reached # Or if an unhandled exception occurs in a worker (which it shouldn't with current try/excepts) await asyncio.gather(*workers, return_exceptions=True) # return_exceptions=True helps debug worker issues except Exception as e: print(f"An error occurred in main: {e}") finally: duration = time.time() - start_time print(f"\n--- Crawl Finished ---") print(f"Total URLs visited (approx): {len(visited_urls)}") print(f"Total unique domains seen: {len(seen_domains)}") print(f"Total unique subdomains seen: {len(seen_subdomains)}") print(f"Total URLs written to {PAGES_FILE}") print(f"Duration: {duration:.2f} seconds") # Count lines in file for verification try: with open(PAGES_FILE, "r", encoding="utf-8") as pf: line_count = sum(1 for _ in pf) print(f"Lines in {PAGES_FILE}: {line_count}") except FileNotFoundError: print(f"{PAGES_FILE} not found (no pages written or error).") if __name__ == "__main__": # In Colab, you run this with: # await main() # For local .py file: # asyncio.run(main()) passHow to run in Google Colab:
- Make sure you've run the Drive mount cell.
- Create a new code cell.
- Paste the entire spider code into this cell.
- Add the following line at the very end of that cell to execute the
maincoroutine:await main() - Run the cell.
Explanation and Key Features:
-
Asynchronous Operations (
aiohttp,asyncio):aiohttp.ClientSession(): Manages a pool of connections for efficient HTTP requests.async def/await: Allows multiple network operations to happen concurrently without blocking the entire program.asyncio.Semaphore(MAX_CONCURRENT_REQUESTS): Limits how manyprocess_urlcoroutines can actively make network requests at the same time, preventing you from overwhelming servers or your own network.AsyncDeque: A customdequesubclass that usesasyncio.Conditionto allow workers toawaitnew items if the queue is empty. This is more efficient than busy-waiting.
-
URL Queue (
AsyncDeque):- Stores URLs yet to be crawled.
appendleft(url): Adds high-priority URLs (new domains/subdomains) to the front.append(url): Adds normal-priority URLs (same domain/subdomain) to the back.popleft_async(): Workers get URLs from the front.
-
Visited Tracking:
visited_urls(aset): Stores all URLs that have been initiated for processing (or successfully redirected to). This prevents re-crawling and getting into loops. O(1) average time complexity for lookups.
-
Domain/Subdomain Tracking:
seen_domains(aset): Tracks unique parent domains encountered (e.g.,example.com).seen_subdomains(aset): Tracks unique network locations / subdomains (e.g.,blog.example.com,www.example.com).- These are used for the prioritization logic.
-
Prioritization Logic (in
process_url):- When a new link is found:
- If its domain is not in
seen_domains, it's a brand new domain: highest priority (queue.appendleft()). - Else, if its netloc (subdomain) is not in
seen_subdomains, it's a new subdomain on an existing domain: high priority (queue.appendleft()). - Otherwise, it's a page on an already seen subdomain/domain: normal priority (
queue.append()).
- If its domain is not in
- When a new link is found:
-
URL Normalization (
normalize_url):- Converts relative links (
/about.html) to absolute links (http://example.com/about.html). - Removes URL fragments (
#section-name) as they typically point to the same page content. - Lowercases scheme and netloc for consistency, as
HTTP://Example.COMis the same ashttp://example.com.
- Converts relative links (
-
Output (
PAGES_FILE):- All unique, valid (starts with
http/https) URLs that are attempted for crawling (or their final redirect targets) are written to this file, one URL per line. - The file is opened in append mode (
"a") so previous crawl results are kept if the script is run multiple times. pages_file_handle.flush()is used to ensure data is written to disk promptly.
- All unique, valid (starts with
-
Error Handling & Politeness:
try-exceptblocks handle network errors, timeouts, and parsing issues.USER_AGENTis set to identify your bot. Changeyour-contact-info.comto something relevant.REQUEST_TIMEOUTprevents hanging indefinitely on slow servers.- A small
await asyncio.sleep(0.1)is added after each page processing within a worker. This, combined with the semaphore, helps in being gentle.
-
Configuration:
START_URLS: Where the spider begins its journey.MAX_PAGES_TO_CRAWL: A crucial safety net to prevent accidental infinite crawls or crawling too much.MAX_CONCURRENT_REQUESTS: Balances speed with resource usage and server load.REQUEST_TIMEOUT: How long to wait for a server response.
This spider is significantly faster than a synchronous one and implements the requested prioritization. Remember that web scraping should be done responsibly, respecting
robots.txt(not implemented here for brevity but important for large-scale or public-facing spiders) and server load.- Use
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.