Skip to content
TheChatArchiveThe Chat Archive
Qwenqwen3-235b-a22b-no-thinking· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

How does OpenAI implements streaming http API, what stack was used? How to implement a…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-5-haiku-20241022’s answer.
  1. User

    How does OpenAI implements streaming http API, what stack was used? How to implement a scalable one on my own?

  2. Qwen Assistant

    Implementing a streaming HTTP API like the one used by OpenAI (e.g., for ChatGPT) involves several components working together to deliver real-time, incremental responses. This is commonly achieved using Server-Sent Events (SSE) or HTTP chunked transfer encoding, and it's particularly useful for applications like chatbots, real-time analytics, or live data feeds.

    Let’s break this down:


    🧠 How OpenAI Implements Streaming HTTP API

    1. Streaming Mechanism

    OpenAI uses Server-Sent Events (SSE) over standard HTTP for streaming model outputs (like GPT completions). Each token generated by the model is sent incrementally to the client as it becomes available.

    • Content-Type: text/event-stream
    • Transfer-Encoding: chunked

    This allows clients to process partial results without waiting for the full response.

    2. Underlying Stack (Inferred)

    While OpenAI does not publicly disclose its internal stack, based on their public-facing APIs and performance characteristics, we can infer they likely use:

    • Language/Runtime: Probably a mix of Python (for model serving wrappers), with some low-level components in C++/CUDA for inference acceleration.
    • Web Frameworks: Likely custom high-performance HTTP services (possibly Go or Rust-based microservices).
    • Model Serving Infrastructure:
      • TensorRT, ONNX, or vLLM for optimized inference.
      • Kubernetes for orchestration.
      • gRPC internally for service-to-service communication.
    • Load Balancing & Edge:
      • NGINX / Envoy Proxy / Custom Load Balancers
      • CDN + Reverse Proxies for global distribution
    • Caching & Rate Limiting:
      • Redis, rate limiting via middleware

    🔨 How to Implement Your Own Scalable Streaming API

    You can build a scalable streaming HTTP API yourself using various technologies depending on your language preference and scale requirements.

    Step-by-step: Basic Implementation

    ✅ Use Case: Stream tokens from an LLM (e.g., LLaMA, Mistral, etc.)


    1. Choose a Language & Framework

    LanguageFrameworksNotes
    PythonFastAPI, Flask, Quart (async)Easy to prototype; async support in FastAPI
    GoGin, Echo, standard net/httpHigh performance, native concurrency
    RustActix, Warp, HyperVery fast, memory-safe
    Node.jsExpress, FastifyGood for JS ecosystem

    For high throughput and scalability, Go or Rust are recommended.


    2. Set Up Chunked Streaming

    Use HTTP chunked transfer encoding to stream partial responses.

    Example in FastAPI (Python):

    from fastapi import FastAPI
    from fastapi.responses import StreamingResponse
    import asyncio
    
    app = FastAPI()
    
    async def token_stream():
        for i in range(10):
            yield f"data: Token {i}\n\n"
            await asyncio.sleep(0.5)
    
    @app.get("/stream")
    async def stream_tokens():
        return StreamingResponse(token_stream(), media_type="text/event-stream")
    

    Now you can curl http://localhost:8000/stream and see tokens arriving in real time.


    3. Integrate with LLM Inference

    To stream actual LLM output, integrate with a model that supports streaming generation:

    • HuggingFace Transformers (supports streamer interface)
    • vLLM (highly efficient for streaming)
    • llama.cpp (if running locally)
    • Custom model with token-by-token generation loop

    Example using HuggingFace Transformers:

    from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
    from threading import Thread
    import torch
    
    model_name = "mistralai/Mistral-7B-Instruct-v0.1"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(model_name)
    
    def generate_stream(input_text):
        inputs = tokenizer(input_text, return_tensors="pt").to("cpu")
        streamer = TextIteratorStreamer(tokenizer)
        
        def _generate():
            model.generate(**inputs, max_new_tokens=50, streamer=streamer)
        
        thread = Thread(target=_generate)
        thread.start()
        
        for token in streamer:
            yield f"data: {token}\n\n"
    
    @app.get("/llm-stream")
    async def llm_stream(prompt: str):
        return StreamingResponse(generate_stream(prompt), media_type="text/event-stream")
    

    4. Scale It Up

    Once the basic streaming works, here’s how to make it scalable:

    🧱 Architecture Overview

    Client → [Load Balancer] → [API Gateway] → [Model Server Pool]
                                          ↘ [Model Inference Workers]
    

    Components to Add:

    1. Reverse Proxy / Load Balancer

      • NGINX / Traefik / Envoy
      • Handles long-lived connections, retries, timeouts
    2. Distributed Task Queue

      • Celery, RabbitMQ, Redis Streams, or Kafka
      • Decouple request handling from model execution
    3. Containerization & Orchestration

      • Docker + Kubernetes
      • Auto-scale pods based on load
    4. GPU-Aware Scheduling

      • Use Triton Inference Server, vLLM, or Ray Serve for better GPU utilization
    5. Rate Limiting & Authentication

      • Middleware or API gateways like Kong, Tyk, or AWS API Gateway
    6. Monitoring & Logging

      • Prometheus + Grafana for metrics
      • ELK stack or Loki for logs

    5. Optional: WebSockets vs SSE

    FeatureSSEWebSockets
    DirectionServer → ClientBi-directional
    SimplicityEasier to implementMore complex
    HTTP CompatibilityWorks over regular HTTPNeeds WS handshake
    Use CaseStreaming model outputsReal-time chat/multiplayer

    For most LLM streaming, SSE is sufficient.


    🚀 Summary

    To build a scalable streaming HTTP API like OpenAI:

    1. Use HTTP chunked encoding or SSE.
    2. Choose a performant backend (FastAPI, Go, Rust).
    3. Integrate with a streaming-capable LLM engine.
    4. Deploy with containers and orchestrate with Kubernetes.
    5. Use reverse proxies and message queues for scale.
    6. Monitor everything.

    📦 Tools You Can Use

    PurposeTool(s)
    Streaming frameworkFastAPI, Flask (with async), Go net/http
    Model servingvLLM, HuggingFace Transformers, llama.cpp
    Load balancingNGINX, Envoy
    Container orchestrationDocker + Kubernetes
    Message queueRedis, RabbitMQ, Kafka
    MonitoringPrometheus, Grafana, Loki

    Would you like a complete example repo template or infrastructure-as-code setup (like Terraform + Docker compose)?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.