Tell me more about Envoy requestbufferpolicy
- User
Tell me more about Envoy request_buffer_policy
- Assistant
Below is a deep-dive into Envoy’s request-buffering features—you’ll see how to configure Envoy to hold (“buffer”) incoming requests in memory before they’re sent upstream. This isn’t quite the same thing as a full “queueing” load-balancer (that would need back-pressure, timeouts, maybe disk spill-to-disk, etc.), but it’s the building block Envoy gives you for buffering HTTP bodies (including unary gRPC).
1) route.request_buffering (a.k.a. request_buffer_policy)
In Envoy v3, every HTTP route can opt-in to a BufferPolicy that says “don’t stream the request to the upstream immediately; hold it until the full request is in memory (or until it exceeds a size limit).”
static_resources: listeners: - name: listener_http address: socket_address: { address: 0.0.0.0, port_value: 8080 } filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager stat_prefix: ingress_http route_config: name: local_route virtual_hosts: - name: backend domains: ["*"] routes: - match: { prefix: "/" } route: cluster: grpc_service request_buffering: # Maximum bytes to buffer for each request. max_request_bytes: value: 1048576 # 1 MiB response_buffering: max_request_bytes: value: 1048576 http_filters: - name: envoy.filters.http.routerWhat it does
- Envoy reads the entire request body into memory up to
max_request_bytes. - Only when the body is fully received will Envoy open/forward to the upstream.
- If the client sends more than
max_request_bytes, Envoy rejects the request with 413 Payload Too Large.
When it applies
- Unary RPCs (single request message → single response).
- Non-streaming HTTP POSTs, PUTs, etc.
- Does not help you “queue up” gRPC stream frames for streaming RPCs.
2) envoy.filters.http.buffer (the HTTP buffer filter)
If you need more control—e.g. apply timeouts or support any HTTP/2 streaming scenario—you can insert Envoy’s Buffer filter before the router. It gives you two knobs:
•
max_request_bytes– limit on in-memory bytes
•max_request_time– how long to wait for the full bodystatic_resources: listeners: - name: listener_http address: socket_address: { address: 0.0.0.0, port_value: 8080 } filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager stat_prefix: grpc_buffering route_config: name: local_route virtual_hosts: - name: backend domains: ["*"] routes: - match: { prefix: "/" } route: cluster: grpc_service http_filters: # 1) First buffer everything - name: envoy.filters.http.buffer typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.buffer.v3.Buffer max_request_bytes: 1048576 # 1 MiB max_request_time: 15s # give upstream up to 15s to become available # 2) Then route to your gRPC backend - name: envoy.filters.http.routerKey differences vs. route.request_buffering
- Works for any HTTP/2 stream, including gRPC streaming.
- You can set a timeout (
max_request_time) so that if the buffer never fills (e.g. upstream never comes up), you return a 504. - When the buffer is full or timeouts fire, Envoy returns an error to the client instead of hanging forever.
3) What about “backends have no spare capacity”?
Neither of the above by itself watches upstream concurrency and then queues new requests. They only buffer the body of a request in transit.
To actually queue when your cluster is saturated you can combine:
-
Circuit Breakers
In your cluster config setmax_pending_requestsormax_requests. Once Envoy has that many in-flight RPCs, new calls will be immediately rejected (429). -
Local Rate/Limiter or Custom Filter
You’d insert or develop an HTTP filter that:- Counts current in-flight RPCs to your cluster (via stats or an internal counter).
- If under threshold → pass request through
- If at threshold → hold it in a small in-memory queue (or fail with back-pressure)
Envoy doesn’t ship a built-in “pending‐request queueing” filter today, so you’d either: • Write a tiny custom
envoy.filters.httpextension
• Use an external queue (Redis Streams, RabbitMQ) as a gRPC-to-queue bridge
4) Putting it all together (example sketch)
static_resources: listeners: - name: grpc_listener address: socket_address: { address: 0.0.0.0, port_value: 50051 } filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager stat_prefix: grpc_buffer route_config: name: grpc_route virtual_hosts: - name: grpc_vhost domains: ["*"] routes: - match: { prefix: "/" } route: cluster: grpc_cluster # 1) Route‐level size-only buffer: request_buffering: max_request_bytes: { value: 2_097_152 } # 2 MiB http_filters: # 2) Global buffer with timeout (unary & streaming): - name: envoy.filters.http.buffer typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.buffer.v3.Buffer max_request_bytes: 2_097_152 max_request_time: 10s # 3) (Optionally) your custom concurrency-limit filter here # - name: envoy.filters.http.your_concurrency_limiter # typed_config: { … } # 4) Finally dispatch to backend - name: envoy.filters.http.router clusters: - name: grpc_cluster connect_timeout: 1s type: STRICT_DNS lb_policy: ROUND_ROBIN # Prevent unbounded in-flight calls: circuit_breakers: thresholds: - max_requests: { value: 100 } max_pending_requests: { value: 100 } load_assignment: cluster_name: grpc_cluster endpoints: - lb_endpoints: - endpoint: { address: { socket_address: { address: backend1, port_value: 50051 } } } - endpoint: { address: { socket_address: { address: backend2, port_value: 50051 } } }- Buffer filter holds the gRPC payload (or stream frames) in memory up to 2 MiB or for at most 10 s.
- Circuit breakers keep at most 100 concurrent RPCs; extra RPC attempts return 429.
- If you really need to queue beyond that you must insert/customize a filter or front‐end queue.
Takeaways
- request_buffering (route level) = simple “hold the full body until it’s all here” (size only).
- buffer filter = full HTTP/2 buffer + timeout. Works for unary & streaming.
- true queuing under load = combo of circuit-breakers + custom filter (or external MQ).
References:
- https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#config-route-v3-routeaction-bufferpolicy
- https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/buffer_filter
- https://www.envoyproxy.io/docs/envoy/latest/configuration/cluster_manager/circuit_breakers
- Envoy reads the entire request body into memory up to
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.