Skip to content
TheChatArchiveThe Chat Archive
fable-5· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

Streaming drops XML tool-call output that non-streaming parses correctly Streaming must…

  1. User

    Streaming drops XML tool-call output that non-streaming parses correctly

    Streaming must emit the same recovered XML tool calls as non-streaming, including calls split across chunks and multiple calls, and finish with tool_calls. Reasoning mode and request tool selection must not alter recovery. Plain text or malformed markup must not create calls or abort the stream.

    Interface

    No new interfaces are introduced.

  2. Assistant

    Now let me look at the existing tests to understand expected behavior:

  3. Assistant

    Let me look at the remaining relevant pieces — StreamEvent, the streaming tool filter tests, and the abstract tool parser:

  4. Assistant

    Reproduced. Let me check a couple of related tests to make sure I understand the expected surface behavior:

  5. Assistant

    Now rework finalize():

  6. Assistant

    Let me guard the result-shaping too, so a parser returning a malformed result can't abort the stream:

  7. Assistant

    Now verify the repro and run the test suites:

  8. Assistant

    All pass. Now add regression tests pinning the new behavior:

  9. Assistant

    The 9 failures are pre-existing environment issues (no MLX on Linux — same failures with my change stashed). Let me run the full runnable suite and compare against baseline to be thorough:

  10. Assistant

    The fix is complete. Summary of what was done:

    Problem

    With a stream request, XML tool-call output (e.g. <tool_call><function=get_weather><parameter=city>Paris</parameter></function></tool_call>) leaked into delta.content as raw text and the stream finished with finish_reason="stop", while the identical non-streaming request returned structured tool_calls. Two gaps in StreamingPostProcessor (vllm_mlx/service/postprocessor.py):

    1. finalize() was gated on self.tool_parser — but the non-streaming route (_parse_tool_calls_with_parser in service/helpers.py) runs unconditionally and falls back to the format-agnostic parse_tool_calls() (Nemotron/Llama XML, Qwen Calling tool:, raw JSON) when no parser is configured or the configured parser finds nothing. Streaming never consulted that fallback.
    2. The standard and channel-routed paths accumulated no text when no tool parser existed, so even a corrected finalize() would have had nothing to scan (only the reasoning path maintained accumulated_text).

    Fix (vllm_mlx/service/postprocessor.py)

    • Added a _raw_stream_text transcript accumulated in process_chunk() for every path (kept separate from accumulated_text so json-mode preamble and reasoning-parser state are untouched), reset in reset().
    • finalize() now runs recovery whenever markup is plausible (</{/[Calling pre-check retained), using tool_accumulated_text or accumulated_text or _raw_stream_text.
    • New _recover_tool_calls() mirrors the non-streaming precedence: configured parser's extract_tool_calls first, then the shared parse_tool_calls() fallback. Both stages are exception-guarded so malformed markup can never abort the SSE stream, and plain text/broken markup produces no calls (same name+arguments guard as non-streaming). The recovered event carries finish_reason="tool_calls", which routes/chat.py already emits as the fallback SSE chunk.

    Verification

    • Reproduced the drop pre-fix; post-fix streaming recovers: single call, calls split across token-sized chunks, multiple calls (sequential indices), under a real qwen3 reasoning parser, and with/without request tools — all finishing with tool_calls.
    • Added 12 regression tests (TestStreamingXmlToolCallRecovery in tests/test_postprocessor.py) covering split chunks, multiple calls, reasoning mode, no-tools requests, plain text, malformed markup, configured-parser miss/exception fallback, channel-routed path, Llama-style XML, and no double-emission.
    • Full runnable suite: 2949 passed vs 2937 baseline (+12 new tests); failure/error counts identical to baseline (all pre-existing libmlx.so environment issues — MLX is Apple-Silicon-only and this is a Linux box, so the route-level tests that import mlx.core can't execute here).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.