Skip to content
TheChatArchiveThe Chat Archive
Coherecommand-a-03-2025· CohereHugging Face LMArena Human Preference 140K

I have the following FastAPI server. Currently there is stand in code simulating sending…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwen3-235b-a22b’s answer.
  1. User

    I have the following FastAPI server. Currently there is stand in code simulating sending SSE from an LLM generation then after the generation is complete, the full response is post-processed then sent over SSE. This two phase process shows progress to the user then replaces it with the full processed version when it's available. There's a fall back in the client to send a get request to fetch the response from a temporary cache if an error occurs. Double check the logic in what I've stated to see if it's the best way to handle things and then evaluate the code:

    from fastapi import FastAPI from fastapi.responses import StreamingResponse, JSONResponse import asyncio import uuid import json import time

    app = FastAPI() responses = {} # {response_id: {"data": processed_response, "expires_at": timestamp}} TTL_SECONDS = 300 # 5 minutes TTL

    Background task to clean up expired responses

    async def cleanup_expired_responses(): while True: current_time = time.time() expired_keys = [ key for key, value in responses.items() if current_time > value["expires_at"] ] for key in expired_keys: responses.pop(key, None) await asyncio.sleep(60) # Check every minute

    @app.on_event("startup") async def startup_event(): asyncio.create_task(cleanup_expired_responses())

    async def llm_stream(): response_id = str(uuid.uuid4()) tokens = ["Hello", ", ", "how ", "are ", "you", "?"] full_response = ""

    for i, token in enumerate(tokens):
        full_response += token
        yield f"id: {response_id}-{i}\ndata: {token}\n\n"
        await asyncio.sleep(0.5)
    
    processed_response = {
        "text": full_response.strip(),
        "formatted": full_response.upper(),
        "metadata": {"length": len(full_response), "response_id": response_id}
    }
    
    responses[response_id] = {
        "data": processed_response,
        "expires_at": time.time() + TTL_SECONDS
    }
    
    yield f"event: processed\nid: {response_id}-final\ndata: {json.dumps(processed_response)}\n\n"
    

    @app.get("/llm-stream") async def stream_llm(): return StreamingResponse(llm_stream(), media_type="text/event-stream")

    @app.get("/processed/{response_id}") async def get_processed(response_id: str): response = responses.pop(response_id, None) # Remove on retrieval if not response or time.time() > response["expires_at"]: return JSONResponse({"error": "Response not found or expired"}, status_code=404) return JSONResponse(response["data"])

  2. Cohere Assistant

    Your approach to handling SSE (Server-Sent Events) from an LLM (Large Language Model) generation, followed by post-processing and caching, is well-thought-out. However, there are a few areas where the logic and implementation can be refined for better robustness, clarity, and efficiency. Below is an evaluation of your approach and code, along with suggestions for improvement:


    Evaluation of the Approach

    1. Two-Phase SSE Process:

      • Sending incremental tokens as SSE events provides real-time feedback to the user, which is good for UX.
      • Sending the final processed response as a separate SSE event (event: processed) is a clean way to signal completion and provide the final data.
      • The fallback mechanism (fetching the response from a cache via a GET request) is a good strategy for handling potential SSE connection issues.
    2. Caching Mechanism:

      • Using an in-memory cache (responses) with a TTL (Time-To-Live) is a practical solution for temporary storage.
      • The background task (cleanup_expired_responses) to remove expired entries is a good practice for memory management.
    3. Error Handling:

      • The fallback GET endpoint (/processed/{response_id}) is a robust way to handle cases where the SSE connection might fail.
      • Removing the response from the cache upon retrieval ensures it's not accessed again, which is appropriate for this use case.

    Code Evaluation and Suggestions

    1. SSE Event Naming:

      • The event: processed is a good choice, but consider using a more generic name like event: complete or event: final to indicate the end of the stream.
    2. Response ID Handling:

      • The response ID is currently generated in llm_stream(). Consider passing it as a parameter to the function for better flexibility and testability.
    3. Caching Logic:

      • The cache cleanup task runs every minute, which is reasonable. However, consider using a more precise mechanism like asyncio.Timeout for TTL management instead of relying on periodic checks.
    4. Error Handling in SSE:

      • Add error handling in the llm_stream() function to gracefully handle exceptions and send appropriate SSE error events.
    5. Code Readability:

      • The code is clean, but adding type hints and docstrings would improve readability and maintainability.
    6. Security:

      • Ensure the response_id is securely generated and not guessable to prevent unauthorized access to cached responses.
    7. Performance:

      • For high-traffic scenarios, consider using a more scalable caching solution like Redis instead of an in-memory dictionary.

    Refined Code

    Here’s an improved version of your code incorporating the above suggestions:

    from fastapi import FastAPI
    from fastapi.responses import StreamingResponse, JSONResponse
    import asyncio
    import uuid
    import json
    import time
    from typing import Dict, Any, Optional
    
    app = FastAPI()
    responses: Dict[str, Dict[str, Any]] = {}  # {response_id: {"data": processed_response, "expires_at": timestamp}}
    TTL_SECONDS = 300  # 5 minutes TTL
    
    # Background task to clean up expired responses
    async def cleanup_expired_responses():
        while True:
            current_time = time.time()
            expired_keys = [
                key for key, value in responses.items()
                if current_time > value["expires_at"]
            ]
            for key in expired_keys:
                responses.pop(key, None)
            await asyncio.sleep(60)  # Check every minute
    
    @app.on_event("startup")
    async def startup_event():
        asyncio.create_task(cleanup_expired_responses())
    
    async def llm_stream(response_id: str):
        """Simulate LLM token generation and yield SSE events."""
        tokens = ["Hello", ", ", "how ", "are ", "you", "?"]
        full_response = ""
        
        try:
            for i, token in enumerate(tokens):
                full_response += token
                yield f"id: {response_id}-{i}\ndata: {token}\n\n"
                await asyncio.sleep(0.5)
            
            processed_response = {
                "text": full_response.strip(),
                "formatted": full_response.upper(),
                "metadata": {"length": len(full_response), "response_id": response_id}
            }
            
            responses[response_id] = {
                "data": processed_response,
                "expires_at": time.time() + TTL_SECONDS
            }
            
            yield f"event: final\nid: {response_id}-final\ndata: {json.dumps(processed_response)}\n\n"
        
        except Exception as e:
            yield f"event: error\nid: {response_id}-error\ndata: {json.dumps({'error': str(e)})}\n\n"
    
    @app.get("/llm-stream")
    async def stream_llm():
        """Endpoint to stream LLM responses as SSE."""
        response_id = str(uuid.uuid4())
        return StreamingResponse(llm_stream(response_id), media_type="text/event-stream")
    
    @app.get("/processed/{response_id}")
    async def get_processed(response_id: str):
        """Fallback endpoint to retrieve processed responses."""
        response = responses.pop(response_id, None)  # Remove on retrieval
        if not response or time.time() > response["expires_at"]:
            return JSONResponse({"error": "Response not found or expired"}, status_code=404)
        return JSONResponse(response["data"])
    

    Key Changes

    1. Added type hints and docstrings for better readability.
    2. Renamed event: processed to event: final for clarity.
    3. Added error handling in llm_stream() to send SSE error events.
    4. Passed response_id as a parameter to llm_stream() for better flexibility.

    Additional Considerations

    • Scalability: If this application is expected to handle many concurrent users, consider using a distributed cache like Redis instead of an in-memory dictionary.
    • Security: Ensure the response_id is securely generated and not exposed to unauthorized users.
    • Logging: Add logging to track cache hits, misses, and errors for debugging and monitoring.

    Overall, your approach is solid, and with these refinements, it will be even more robust and maintainable.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.