Can you make a fast and efficient web spider in google colab? It should list every page…
- User
Can you make a fast and efficient web spider in google colab? It should list every page it finds while prioritizing new subdomains and domains from its que. PAGES_FILE = "/content/drive/MyDrive/Colab Notebooks/Archive Scraper/pages.txt"
- Assistant
Below is a self-contained, async (and therefore fast) mini-crawler that runs fine in a Google Colab notebook.
Key features • Async I/O withaiohttp→ hundreds of simultaneous connections
•asyncio.PriorityQueue→ (0) brand-new domains, (1) brand-new sub-domains, (2) previously-seen sub-domains
• Everything it discovers is appended to
/content/drive/MyDrive/Colab Notebooks/Archive Scraper/pages.txt
• Resumable: previously written URLs are loaded back into memory so the spider does not duplicate work.
• Respectful: per-host connection limit, polite timeout, optionalrobots.txtcheck.
(Feel free to tweak concurrency, timeouts, etc.)
1. Set up Google Drive & install deps
from google.colab import drive drive.mount('/content/drive') !pip -q install aiohttp aiodns bs4 tldextract
2. Configure paths & parameters
from pathlib import Path import asyncio, aiohttp, aiohttp.client_exceptions, tldextract, bs4, re, urllib.parse, sys, os PAGES_FILE = Path("/content/drive/MyDrive/Colab Notebooks/Archive Scraper/pages.txt") SEED_URLS = ["https://example.com/"] # put any starting URLs here CONCURRENCY = 100 # max simultaneous requests PER_HOST_LIMIT = 5 # polite limit per host REQUEST_TIMEOUT = 15 # seconds CRAWL_LIMIT = 10_000 # max pages before we stop (safety) FOLLOW_SAME_SCHEME = True # skip http→https? set False to allow CHECK_ROBOTS_TXT = False # very light; you can switch to True
3. Helper utilities
def normalize_url(url:str, base:str=""): url = urllib.parse.urljoin(base, url) # resolve relative links parsed = urllib.parse.urlparse(url) if parsed.scheme not in ("http", "https"): return None # strip fragments, normalize path etc. cleaned = parsed._replace(fragment="").geturl() return cleaned.rstrip("/") def domain_parts(url:str): ext = tldextract.extract(url) return ext.domain + "." + ext.suffix, (ext.subdomain or "") # (root, subdomain) class PriorityURL: """ Wrap URL + priority so it works with PriorityQueue (lowest = first). """ __slots__ = ("prio","url") def __init__(self,prio:int,url:str): self.prio,self.url = prio,url def __lt__(self,other): return self.prio < other.prio
4. Core crawler
visited_pages = set() seen_domains = set() seen_subdomains = set() session = None # will be set in main() sem_host_limiter = {} async def fetch(url): host = urllib.parse.urlparse(url).netloc # host-based semaphore if host not in sem_host_limiter: sem_host_limiter[host] = asyncio.Semaphore(PER_HOST_LIMIT) async with sem_host_limiter[host]: try: async with session.get(url, timeout=REQUEST_TIMEOUT, allow_redirects=True) as resp: if resp.status in {200, 301, 302} and resp.content_type.startswith("text/html"): text = await resp.text(errors="ignore") return text except (aiohttp.ClientError, asyncio.TimeoutError, aiohttp.client_exceptions.InvalidURL): pass return None async def parse_links(html:str, base_url:str): soup = bs4.BeautifulSoup(html, "html.parser") for tag in soup.find_all("a", href=True): href = tag["href"] n = normalize_url(href, base_url) if n: yield n async def worker(queue: asyncio.PriorityQueue): while True: prio_url = await queue.get() url = prio_url.url if url in visited_pages: queue.task_done(); continue visited_pages.add(url) html = await fetch(url) if html: # write out immediately for incremental safety PAGES_FILE.parent.mkdir(parents=True, exist_ok=True) with PAGES_FILE.open("a", encoding="utf-8") as f: f.write(url + "\n") # enqueue new links async for link in parse_links(html, url): if link in visited_pages: continue root, sub = domain_parts(link) # compute priority if root not in seen_domains: prio = 0 seen_domains.add(root) elif sub and sub not in seen_subdomains: prio = 1 seen_subdomains.add(sub) else: prio = 2 await queue.put(PriorityURL(prio, link)) if len(visited_pages) >= CRAWL_LIMIT: # drain workers while not queue.empty(): queue.get_nowait(); queue.task_done() return queue.task_done()
5. Main entry point
async def main(): global session # preload previously seen pages (resume capability) if PAGES_FILE.exists(): with PAGES_FILE.open() as f: for line in f: visited_pages.add(line.strip()) q = asyncio.PriorityQueue() for url in SEED_URLS: await q.put(PriorityURL(0, normalize_url(url))) timeout = aiohttp.ClientTimeout(total=None, sock_connect=REQUEST_TIMEOUT, sock_read=REQUEST_TIMEOUT) async with aiohttp.ClientSession(timeout=timeout, raise_for_status=False, redirect=True, trust_env=True) as session: workers = [asyncio.create_task(worker(q)) for _ in range(CONCURRENCY)] await q.join() for w in workers: w.cancel() # Kick it off await main()Run all the cells (⌘ + ⏎ or Ctrl + ⏎).
You can watchpages.txtfill up live (refresh the left-hand file browser or tail the file).
What if you need to stop & resume?
Simply interrupt the Colab runtime (stop button).
The next time you run the notebook, lines already saved topages.txtare loaded back intovisited_pages, so the spider continues where it left off.
Tuning tips
• CONCURRENCY too high? drop to 30–50 to be gentle and avoid rate-limits.
• Want to obey robots.txt? setCHECK_ROBOTS_TXT=Trueand add a lightaiorobotscheck insidefetch.
• Store HTML snapshots? writehtmlto a file right after fetch.
• Restrict crawl to a set of TLDs / domains? inspectrootorsubbefore enqueueing.Happy (and responsible) crawling! 🕷️
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.