I have the following FastAPI server. Currently there is stand in code simulating sending…
- User
I have the following FastAPI server. Currently there is stand in code simulating sending SSE from an LLM generation then after the generation is complete, the full response is post-processed then sent over SSE. This two phase process shows progress to the user then replaces it with the full processed version when it's available. There's a fall back in the client to send a get request to fetch the response from a temporary cache if an error occurs. Double check the logic in what I've stated to see if it's the best way to handle things and then evaluate the code:
from fastapi import FastAPI from fastapi.responses import StreamingResponse, JSONResponse import asyncio import uuid import json import time
app = FastAPI() responses = {} # {response_id: {"data": processed_response, "expires_at": timestamp}} TTL_SECONDS = 300 # 5 minutes TTL
Background task to clean up expired responses
async def cleanup_expired_responses(): while True: current_time = time.time() expired_keys = [ key for key, value in responses.items() if current_time > value["expires_at"] ] for key in expired_keys: responses.pop(key, None) await asyncio.sleep(60) # Check every minute
@app.on_event("startup") async def startup_event(): asyncio.create_task(cleanup_expired_responses())
async def llm_stream(): response_id = str(uuid.uuid4()) tokens = ["Hello", ", ", "how ", "are ", "you", "?"] full_response = ""
for i, token in enumerate(tokens): full_response += token yield f"id: {response_id}-{i}\ndata: {token}\n\n" await asyncio.sleep(0.5) processed_response = { "text": full_response.strip(), "formatted": full_response.upper(), "metadata": {"length": len(full_response), "response_id": response_id} } responses[response_id] = { "data": processed_response, "expires_at": time.time() + TTL_SECONDS } yield f"event: processed\nid: {response_id}-final\ndata: {json.dumps(processed_response)}\n\n"@app.get("/llm-stream") async def stream_llm(): return StreamingResponse(llm_stream(), media_type="text/event-stream")
@app.get("/processed/{response_id}") async def get_processed(response_id: str): response = responses.pop(response_id, None) # Remove on retrieval if not response or time.time() > response["expires_at"]: return JSONResponse({"error": "Response not found or expired"}, status_code=404) return JSONResponse(response["data"])
Assistant
Your approach to handling SSE (Server-Sent Events) from an LLM (Large Language Model) generation, followed by post-processing and caching, is well-thought-out. However, there are a few areas where the logic and implementation can be refined for better robustness, clarity, and efficiency. Below is an evaluation of your approach and code, along with suggestions for improvement:
Evaluation of the Approach
-
Two-Phase SSE Process:
- Sending incremental tokens as SSE events provides real-time feedback to the user, which is good for UX.
- Sending the final processed response as a separate SSE event (
event: processed) is a clean way to signal completion and provide the final data. - The fallback mechanism (fetching the response from a cache via a GET request) is a good strategy for handling potential SSE connection issues.
-
Caching Mechanism:
- Using an in-memory cache (
responses) with a TTL (Time-To-Live) is a practical solution for temporary storage. - The background task (
cleanup_expired_responses) to remove expired entries is a good practice for memory management.
- Using an in-memory cache (
-
Error Handling:
- The fallback GET endpoint (
/processed/{response_id}) is a robust way to handle cases where the SSE connection might fail. - Removing the response from the cache upon retrieval ensures it's not accessed again, which is appropriate for this use case.
- The fallback GET endpoint (
Code Evaluation and Suggestions
-
SSE Event Naming:
- The
event: processedis a good choice, but consider using a more generic name likeevent: completeorevent: finalto indicate the end of the stream.
- The
-
Response ID Handling:
- The response ID is currently generated in
llm_stream(). Consider passing it as a parameter to the function for better flexibility and testability.
- The response ID is currently generated in
-
Caching Logic:
- The cache cleanup task runs every minute, which is reasonable. However, consider using a more precise mechanism like
asyncio.Timeoutfor TTL management instead of relying on periodic checks.
- The cache cleanup task runs every minute, which is reasonable. However, consider using a more precise mechanism like
-
Error Handling in SSE:
- Add error handling in the
llm_stream()function to gracefully handle exceptions and send appropriate SSE error events.
- Add error handling in the
-
Code Readability:
- The code is clean, but adding type hints and docstrings would improve readability and maintainability.
-
Security:
- Ensure the
response_idis securely generated and not guessable to prevent unauthorized access to cached responses.
- Ensure the
-
Performance:
- For high-traffic scenarios, consider using a more scalable caching solution like Redis instead of an in-memory dictionary.
Refined Code
Here’s an improved version of your code incorporating the above suggestions:
from fastapi import FastAPI from fastapi.responses import StreamingResponse, JSONResponse import asyncio import uuid import json import time from typing import Dict, Any, Optional app = FastAPI() responses: Dict[str, Dict[str, Any]] = {} # {response_id: {"data": processed_response, "expires_at": timestamp}} TTL_SECONDS = 300 # 5 minutes TTL # Background task to clean up expired responses async def cleanup_expired_responses(): while True: current_time = time.time() expired_keys = [ key for key, value in responses.items() if current_time > value["expires_at"] ] for key in expired_keys: responses.pop(key, None) await asyncio.sleep(60) # Check every minute @app.on_event("startup") async def startup_event(): asyncio.create_task(cleanup_expired_responses()) async def llm_stream(response_id: str): """Simulate LLM token generation and yield SSE events.""" tokens = ["Hello", ", ", "how ", "are ", "you", "?"] full_response = "" try: for i, token in enumerate(tokens): full_response += token yield f"id: {response_id}-{i}\ndata: {token}\n\n" await asyncio.sleep(0.5) processed_response = { "text": full_response.strip(), "formatted": full_response.upper(), "metadata": {"length": len(full_response), "response_id": response_id} } responses[response_id] = { "data": processed_response, "expires_at": time.time() + TTL_SECONDS } yield f"event: final\nid: {response_id}-final\ndata: {json.dumps(processed_response)}\n\n" except Exception as e: yield f"event: error\nid: {response_id}-error\ndata: {json.dumps({'error': str(e)})}\n\n" @app.get("/llm-stream") async def stream_llm(): """Endpoint to stream LLM responses as SSE.""" response_id = str(uuid.uuid4()) return StreamingResponse(llm_stream(response_id), media_type="text/event-stream") @app.get("/processed/{response_id}") async def get_processed(response_id: str): """Fallback endpoint to retrieve processed responses.""" response = responses.pop(response_id, None) # Remove on retrieval if not response or time.time() > response["expires_at"]: return JSONResponse({"error": "Response not found or expired"}, status_code=404) return JSONResponse(response["data"])
Key Changes
- Added type hints and docstrings for better readability.
- Renamed
event: processedtoevent: finalfor clarity. - Added error handling in
llm_stream()to send SSE error events. - Passed
response_idas a parameter tollm_stream()for better flexibility.
Additional Considerations
- Scalability: If this application is expected to handle many concurrent users, consider using a distributed cache like Redis instead of an in-memory dictionary.
- Security: Ensure the
response_idis securely generated and not exposed to unauthorized users. - Logging: Add logging to track cache hits, misses, and errors for debugging and monitoring.
Overall, your approach is solid, and with these refinements, it will be even more robust and maintainable.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.