Streaming drops XML tool-call output that non-streaming parses correctly Streaming must…
- User
Streaming drops XML tool-call output that non-streaming parses correctly
Streaming must emit the same recovered XML tool calls as non-streaming, including calls split across chunks and multiple calls, and finish with
tool_calls. Reasoning mode and request tool selection must not alter recovery. Plain text or malformed markup must not create calls or abort the stream.Interface
No new interfaces are introduced.
- Assistant
Now let me look at the existing tests to understand expected behavior:
- Assistant
Let me look at the remaining relevant pieces —
StreamEvent, the streaming tool filter tests, and the abstract tool parser: - Assistant
Reproduced. Let me check a couple of related tests to make sure I understand the expected surface behavior:
- Assistant
Now rework
finalize(): - Assistant
Let me guard the result-shaping too, so a parser returning a malformed result can't abort the stream:
- Assistant
Now verify the repro and run the test suites:
- Assistant
All pass. Now add regression tests pinning the new behavior:
- Assistant
The 9 failures are pre-existing environment issues (no MLX on Linux — same failures with my change stashed). Let me run the full runnable suite and compare against baseline to be thorough:
- Assistant
The fix is complete. Summary of what was done:
Problem
With a stream request, XML tool-call output (e.g.
<tool_call><function=get_weather><parameter=city>Paris</parameter></function></tool_call>) leaked intodelta.contentas raw text and the stream finished withfinish_reason="stop", while the identical non-streaming request returned structuredtool_calls. Two gaps inStreamingPostProcessor(vllm_mlx/service/postprocessor.py):finalize()was gated onself.tool_parser— but the non-streaming route (_parse_tool_calls_with_parserinservice/helpers.py) runs unconditionally and falls back to the format-agnosticparse_tool_calls()(Nemotron/Llama XML, QwenCalling tool:, raw JSON) when no parser is configured or the configured parser finds nothing. Streaming never consulted that fallback.- The standard and channel-routed paths accumulated no text when no tool parser existed, so even a corrected
finalize()would have had nothing to scan (only the reasoning path maintainedaccumulated_text).
Fix (
vllm_mlx/service/postprocessor.py)- Added a
_raw_stream_texttranscript accumulated inprocess_chunk()for every path (kept separate fromaccumulated_textso json-mode preamble and reasoning-parser state are untouched), reset inreset(). finalize()now runs recovery whenever markup is plausible (</{/[Callingpre-check retained), usingtool_accumulated_text or accumulated_text or _raw_stream_text.- New
_recover_tool_calls()mirrors the non-streaming precedence: configured parser'sextract_tool_callsfirst, then the sharedparse_tool_calls()fallback. Both stages are exception-guarded so malformed markup can never abort the SSE stream, and plain text/broken markup produces no calls (samename+argumentsguard as non-streaming). The recovered event carriesfinish_reason="tool_calls", whichroutes/chat.pyalready emits as the fallback SSE chunk.
Verification
- Reproduced the drop pre-fix; post-fix streaming recovers: single call, calls split across token-sized chunks, multiple calls (sequential indices), under a real qwen3 reasoning parser, and with/without request tools — all finishing with
tool_calls. - Added 12 regression tests (
TestStreamingXmlToolCallRecoveryin tests/test_postprocessor.py) covering split chunks, multiple calls, reasoning mode, no-tools requests, plain text, malformed markup, configured-parser miss/exception fallback, channel-routed path, Llama-style XML, and no double-emission. - Full runnable suite: 2949 passed vs 2937 baseline (+12 new tests); failure/error counts identical to baseline (all pre-existing
libmlx.soenvironment issues — MLX is Apple-Silicon-only and this is a Linux box, so the route-level tests that importmlx.corecan't execute here).
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- fable-5Compare side by side →