Skip to content
TheChatArchiveThe Chat Archive
o4-mini-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

Tell me more about Envoy requestbufferpolicy

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Coherecommand-a-03-2025’s answer.
  1. User

    Tell me more about Envoy request_buffer_policy

  2. Assistant

    Below is a deep-dive into Envoy’s request-buffering features—you’ll see how to configure Envoy to hold (“buffer”) incoming requests in memory before they’re sent upstream. This isn’t quite the same thing as a full “queueing” load-balancer (that would need back-pressure, timeouts, maybe disk spill-to-disk, etc.), but it’s the building block Envoy gives you for buffering HTTP bodies (including unary gRPC).


    1) route.request_buffering (a.k.a. request_buffer_policy)

    In Envoy v3, every HTTP route can opt-in to a BufferPolicy that says “don’t stream the request to the upstream immediately; hold it until the full request is in memory (or until it exceeds a size limit).”

    static_resources:
      listeners:
      - name: listener_http
        address:
          socket_address: { address: 0.0.0.0, port_value: 8080 }
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_http
              route_config:
                name: local_route
                virtual_hosts:
                - name: backend
                  domains: ["*"]
                  routes:
                  - match: { prefix: "/" }
                    route:
                      cluster: grpc_service
                      request_buffering:
                        # Maximum bytes to buffer for each request.
                        max_request_bytes:
                          value: 1048576   # 1 MiB
                      response_buffering:
                        max_request_bytes:
                          value: 1048576
              http_filters:
              - name: envoy.filters.http.router
    

    What it does

    • Envoy reads the entire request body into memory up to max_request_bytes.
    • Only when the body is fully received will Envoy open/forward to the upstream.
    • If the client sends more than max_request_bytes, Envoy rejects the request with 413 Payload Too Large.

    When it applies

    • Unary RPCs (single request message → single response).
    • Non-streaming HTTP POSTs, PUTs, etc.
    • Does not help you “queue up” gRPC stream frames for streaming RPCs.

    2) envoy.filters.http.buffer (the HTTP buffer filter)

    If you need more control—e.g. apply timeouts or support any HTTP/2 streaming scenario—you can insert Envoy’s Buffer filter before the router. It gives you two knobs:

    • max_request_bytes – limit on in-memory bytes
    • max_request_time – how long to wait for the full body

    static_resources:
      listeners:
      - name: listener_http
        address:
          socket_address: { address: 0.0.0.0, port_value: 8080 }
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: grpc_buffering
              route_config:
                name: local_route
                virtual_hosts:
                - name: backend
                  domains: ["*"]
                  routes:
                  - match: { prefix: "/" }
                    route:
                      cluster: grpc_service
              http_filters:
              # 1) First buffer everything
              - name: envoy.filters.http.buffer
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.buffer.v3.Buffer
                  max_request_bytes: 1048576      # 1 MiB
                  max_request_time: 15s           # give upstream up to 15s to become available
              # 2) Then route to your gRPC backend
              - name: envoy.filters.http.router
    

    Key differences vs. route.request_buffering

    • Works for any HTTP/2 stream, including gRPC streaming.
    • You can set a timeout (max_request_time) so that if the buffer never fills (e.g. upstream never comes up), you return a 504.
    • When the buffer is full or timeouts fire, Envoy returns an error to the client instead of hanging forever.

    3) What about “backends have no spare capacity”?

    Neither of the above by itself watches upstream concurrency and then queues new requests. They only buffer the body of a request in transit.

    To actually queue when your cluster is saturated you can combine:

    1. Circuit Breakers
      In your cluster config set max_pending_requests or max_requests. Once Envoy has that many in-flight RPCs, new calls will be immediately rejected (429).

    2. Local Rate/Limiter or Custom Filter
      You’d insert or develop an HTTP filter that:

      • Counts current in-flight RPCs to your cluster (via stats or an internal counter).
      • If under threshold → pass request through
      • If at threshold → hold it in a small in-memory queue (or fail with back-pressure)

      Envoy doesn’t ship a built-in “pending‐request queueing” filter today, so you’d either: • Write a tiny custom envoy.filters.http extension
      • Use an external queue (Redis Streams, RabbitMQ) as a gRPC-to-queue bridge


    4) Putting it all together (example sketch)

    static_resources:
      listeners:
      - name: grpc_listener
        address:
          socket_address: { address: 0.0.0.0, port_value: 50051 }
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: grpc_buffer
              route_config:
                name: grpc_route
                virtual_hosts:
                - name: grpc_vhost
                  domains: ["*"]
                  routes:
                  - match: { prefix: "/" }
                    route:
                      cluster: grpc_cluster
                      # 1) Route‐level size-only buffer:
                      request_buffering:
                        max_request_bytes: { value: 2_097_152 }   # 2 MiB
              http_filters:
              # 2) Global buffer with timeout (unary & streaming):
              - name: envoy.filters.http.buffer
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.buffer.v3.Buffer
                  max_request_bytes:    2_097_152
                  max_request_time:     10s
              # 3) (Optionally) your custom concurrency-limit filter here
              # - name: envoy.filters.http.your_concurrency_limiter
              #   typed_config: { … }
              # 4) Finally dispatch to backend
              - name: envoy.filters.http.router
    
      clusters:
      - name: grpc_cluster
        connect_timeout: 1s
        type: STRICT_DNS
        lb_policy: ROUND_ROBIN
        # Prevent unbounded in-flight calls:
        circuit_breakers:
          thresholds:
          - max_requests:          { value: 100 }
            max_pending_requests:  { value: 100 }
        load_assignment:
          cluster_name: grpc_cluster
          endpoints:
          - lb_endpoints:
            - endpoint: { address: { socket_address: { address: backend1, port_value: 50051 } } }
            - endpoint: { address: { socket_address: { address: backend2, port_value: 50051 } } }
    
    1. Buffer filter holds the gRPC payload (or stream frames) in memory up to 2 MiB or for at most 10 s.
    2. Circuit breakers keep at most 100 concurrent RPCs; extra RPC attempts return 429.
    3. If you really need to queue beyond that you must insert/customize a filter or front‐end queue.

    Takeaways

    • request_buffering (route level) = simple “hold the full body until it’s all here” (size only).
    • buffer filter = full HTTP/2 buffer + timeout. Works for unary & streaming.
    • true queuing under load = combo of circuit-breakers + custom filter (or external MQ).

    References:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.