Create a Python application that uses asyncio to handle thousands of concurrent web…
- User
Create a Python application that uses asyncio to handle thousands of concurrent web requests.
Assistant
1. Understand the Goal: The user wants a Python application that uses
asyncioto handle "thousands" of concurrent web requests. This implies a need for high concurrency, non-blocking I/O, and likely a client or server architecture. Since the phrasing "handle... web requests" is slightly ambiguous (could mean making requests as a scraper/client or serving requests as a server), but usually "handle requests" suggests a server, while "make requests" suggests a client. However, often people ask this wanting to build a scraper (client side) or an API server (server side). I should probably provide a comprehensive answer that covers both or clarifies the distinction. Given the context of "thousands of concurrent... requests," it often leans towards an HTTP client (like scraping or load testing) because writing a raw asyncio server for thousands of connections is standard for frameworks like FastAPI/Sanic, but writing the client logic to manage that concurrency is the tricky part whereasyncioshines.*Decision:* I will provide two examples: 1. **The Client (Requester):** A script that fetches data from many URLs concurrently (e.g., a scraper or load tester). This is the most common context for "handling thousands of requests" via `asyncio`. 2. **The Server (Handler):** A simple async web server (using `aiohttp` or just standard library `asyncio` streams) that can handle incoming connections. *Refinement:* The most valuable and distinct demonstration of `asyncio` power is usually the *client-side* semaphore/batching logic to prevent crashing the system or getting banned by the target server. I will focus primarily on the client side (fetching URLs) as the primary example, but briefly mention how to serve them.2. Key Concepts to Demonstrate: *
async/awaitsyntax. *aiohttp(the de-facto standard for async HTTP). *asyncio.gatherorasyncio.waitvs.TaskGroup(Python 3.11+). * Rate Limiting/Semaphore: Crucial for "thousands" of requests so you don't run out of sockets or RAM. * Session Management: Reusing aClientSessionis vital for performance (keep-alive).-
Drafting the Client Solution (The Main Attraction):
- Library:
aiohttp. - Structure:
- A worker function
fetch(session, url). - A main coordinator
bulk_fetch(urls).
- A worker function
- Concurrency Control: Using
asyncio.Semaphore. If we launch 10,000 requests at once without a semaphore, we might hit OS limits (file descriptors) or get banned. - Error Handling:
try/exceptinside the fetch function.
- Library:
-
Drafting the Code (Mental or Scratchpad):
import asyncio import aiohttp # Limit concurrency CONCURRENCY_LIMIT = 100 async def fetch(session, url): try: async with session.get(url) as response: return await response.text() except Exception as e: return None async def main(urls): semaphore = asyncio.Semaphore(CONCURRENCY_LIMIT) async with aiohttp.ClientSession() as session: tasks = [] for url in urls: # Wrap fetch with semaphore logic task = asyncio.create_task(bound_fetch(semaphore, session, url)) tasks.append(task) results = await asyncio.gather(*tasks)-
Refining the Code for "Modern" Python:
- Use
aiohttp. - Include a context manager for the semaphore to ensure strict limiting.
- Add timeouts (essential for network reliability).
- Use
asyncio.gatherto collect results. - Provide a dummy list of URLs or a way to generate them for the example to run.
- Use
-
Adding a Server Example (Briefly):
- Just to be thorough, I'll include a tiny
aiohttpserver snippet to show how to receive requests asynchronously. This covers the ambiguity of "handle".
- Just to be thorough, I'll include a tiny
-
Structuring the Explanation:
- Prerequisites: Mention
pip install aiohttp. - The "Client" Approach (Fetching Data): This is usually what people mean.
- Explain why standard
requestsfails (blocking). - Explain the Semaphore (limiting open connections).
- Provide the code.
- Explain why standard
- The "Server" Approach (Serving API):
- Briefly mention FastAPI or show a raw
aiohttpserver.
- Briefly mention FastAPI or show a raw
- Best Practices:
- Session reuse.
- Timeouts.
- Exception handling.
- Prerequisites: Mention
-
Review against "Thousands" constraint:
- Standard
requestswould time out or block. - Threads would hit GIL issues or context switching overhead at high counts (though threads can do it, async is cleaner).
asynciois perfect here.- Self-Correction: I need to ensure I explicitly handle the "thousands" aspect. Launching 10,000 coroutines is fine in Python, but 10,000 simultaneous HTTP connections might hit OS ulimits. I must emphasize the
Semaphore.
- Standard
-
Final Polish of the Code:
- Add comments.
- Make it runnable (include a
if __name__ == "__main__":block). - Use dummy URLs (httpbin is good for testing) so the user can actually run it.
-
Constructing the Output: (This aligns with the provided good response).
- Title: Asynchronous Web Request Client.
- Dependencies.
- Code block with
aiohttpandSemaphore. - Explanation of key components.
- Server example (using
aiohttpserver orFastAPIreference).
(Self-Correction during drafting the response): I'll stick to the Client (scraper/load tester) as the primary answer because "handling web requests" usually implies acting as the client consuming an API or scraping. If they meant a server, they usually say "web server". However, I will clarify this distinction. I will focus the code on fetching thousands of URLs efficiently.
Code specific checks:
aiohttp.ClientSession()must be used in an async context.asyncio.gathervsTaskGroup.TaskGroupis Python 3.11+. I should probably stick togatherfor broader compatibility or mention 3.11 features. Let's stick to standardgatherwith a semaphore wrapper for clarity on the limit logic.
Let's write the final response.
- Step 1: Install.
pip install aiohttp - Step 2: The Code.
- Define a list of URLs.
- Semaphore set to, say, 100 or 500 (don't do 5000 instantly).
fetchfunction with error handling.mainfunction managing the session.
- Step 3: Explanation. Why use
aiohttp? WhySemaphore? WhySession?
Let's also briefly touch upon the Server side just in case. "If you meant creating a server to receive requests..."
Okay, the plan is solid. I will implement a robust client script using
aiohttp. Here is a complete, robust Python application designed to handle thousands of concurrent web requests.
This solution uses
aiohttp(the standard asynchronous HTTP client) andasyncio. It implements a Semaphore to limit concurrency (preventing your operating system or the target server from crashing due to overload) and connection pooling for efficiency.Prerequisites
You will need to install
aiohttp:pip install aiohttpThe Application
This script fetches a list of URLs asynchronously. I have included a "Safe Session" wrapper to handle common issues like timeouts and connection errors automatically.
import asyncio import aiohttp import time from typing import List, Dict, Any # Configuration CONCURRENCY_LIMIT = 100 # Max simultaneous requests TIMEOUT_SECONDS = 10 # Timeout for each request async def fetch_url( session: aiohttp.ClientSession, url: str, semaphore: asyncio.Semaphore ) -> Dict[str, Any]: """ Fetches a single URL with error handling and concurrency control. """ async with semaphore: try: async with session.get(url, timeout=TIMEOUT_SECONDS) as response: # Read the response text. # Note: For large files, consider streaming with 'chunked' instead of reading all at once. data = await response.text() return { "url": url, "status": response.status, "data": data[:100], # Truncated for display "error": None } except asyncio.TimeoutError: return {"url": url, "status": 0, "data": None, "error": "Timeout"} except aiohttp.ClientError as e: return {"url": url, "status": 0, "data": None, "error": str(e)} except Exception as e: return {"url": url, "status": 0, "data": None, "error": f"Unexpected: {str(e)}"} async def bulk_fetch(urls: List[str]) -> List[Dict[str, Any]]: """ Orchestrates the concurrent fetching of thousands of URLs. """ # 1. Create a semaphore to limit concurrency. # Without this, trying to open 10,000 sockets at once might crash the OS or ban your IP. semaphore = asyncio.Semaphore(CONCURRENCY_LIMIT) # 2. Configure the session with connection pooling. # Connector limit determines total open connections in the pool. connector = aiohttp.TCPConnector(limit=CONCURRENCY_LIMIT, force_close=True) results = [] print(f"Starting fetch of {len(urls)} URLs with concurrency limit of {CONCURRENCY_LIMIT}...") start_time = time.time() async with aiohttp.ClientSession(connector=connector) as session: # 3. Create tasks for all URLs tasks = [fetch_url(session, url, semaphore) for url in urls] # 4. Run tasks concurrently # asyncio.gather runs them in parallel, but our semaphore inside fetch_url # ensures we don't exceed CONCURRENCY_LIMIT active requests. results = await asyncio.gather(*tasks) end_time = time.time() print(f"Completed in {end_time - start_time:.2f} seconds.") return results def generate_test_urls(count: int) -> List[str]: """Generates dummy URLs for testing purposes.""" # Using httpbin.org which is a standard service for testing HTTP requests return [f"https://httpbin.org/get?id={i}" for i in range(count)] # --- Main Entry Point --- async def main(): # Generate 1000 test URLs urls_to_fetch = generate_test_urls(1000) # Run the bulk fetch operation results = await bulk_fetch(urls_to_fetch) # Analyze results success_count = sum(1 for r in results if r["status"] == 200) fail_count = len(results) - success_count print(f"\nResults Summary:") print(f"Success: {success_count}") print(f"Failures: {fail_count}") # Print a sample result if results: print("\nSample Result:") print(results[0]) if __name__ == "__main__": # Run the async main loop asyncio.run(main())Key Concepts Explained
-
asyncio.Semaphore(CONCURRENCY_LIMIT):- Why? If you try to launch 5,000 requests simultaneously without a limiter, you will likely hit "Too many open files" errors on your OS or trigger Denial-of-Service protections on the target server.
- How it works: It acts like a counter. Only
CONCURRENCY_LIMITtasks can pass theasync with semaphore:line at once. Others wait until a spot opens up.
-
aiohttp.ClientSession:- Why? Creating a new HTTP connection for every request is slow (DNS lookup + TCP handshake + SSL handshake).
- How it works: The session keeps a "pool" of open connections (managed by
TCPConnector). When one request finishes, the connection is reused for the next URL if they share the same domain.
-
asyncio.gather:- This function takes a list of coroutine objects (the tasks) and schedules them to run on the event loop.
-
Timeouts:
- In a synchronous script, a hung request blocks the entire program. In
asyncio, a hung request just hangs that specific task. We explicitly settimeoutso that unresponsive URLs don't leave connections open indefinitely.
- In a synchronous script, a hung request blocks the entire program. In
Server Side (If you meant receiving requests)
If your goal was to create a Server that can handle thousands of incoming connections (an API), you should use a framework like FastAPI or Aiohttp (server mode).
Here is a minimal example of an async server using
aiohttp:from aiohttp import web async def handle(request): # Simulate async work (e.g., DB query) await asyncio.sleep(0.1) return web.json_response({"status": "ok", "message": "Hello world"}) app = web.Application() app.add_routes([web.get('/', handle)]) if __name__ == '__main__': web.run_app(app, port=8080)-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.