ISSUE DESCRIPTION Streaming drops XML tool-call output that non-streaming parses…
- User
ISSUE DESCRIPTION
Streaming drops XML tool-call output that non-streaming parses correctly
Streaming must emit the same recovered XML tool calls as non-streaming, including calls split across chunks and multiple calls, and finish with
tool_calls. Reasoning mode and request tool selection must not alter recovery. Plain text or malformed markup must not create calls or abort the stream.Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /Rapid-MLX, current file: none) bash-$
- Assistant
I’ll first map the repository and locate the non-streaming versus streaming XML tool-call parsing paths. Then I’ll reproduce the mismatch with existing tests or a temporary script, implement a shared/general recovery behavior, and run the relevant test suite.
- Assistant
The repository already contains targeted streaming-spec tests whose comments mention a
finalize/extract_tool_callsmismatch, so I’ll inspect those tests and the streaming post-processor implementation before constructing the smallest direct reproduction. - Assistant
The visible regression test covers a generic fallback, but the issue is specifically recovered XML, chunk splits, multiple calls, finish reason, reasoning, and request tool selection. I’ll inspect
StreamingPostProcessorstate/finalization and the XML auto-recovery parser, plus the remainder of the targeted tests, to identify where these dimensions diverge. - Assistant
The previous output was truncated and interleaved, so I’ll narrow to numbered sections around
_detect_tool_calls, finish handling, andfinalize, then compare the route’s post-loop event emission. - Assistant
A key structural problem is apparent: a normal final
stopevent is emitted duringprocess_chunk, and only afterward canfinalize()recover calls and emittool_calls, potentially producing two finish reasons. I also need the exact accumulation/finalize logic to see why bare XML is skipped (the fast pre-check currently only activates on<or[per chunk, while finalize may have another marker gate). - Assistant
I’ll now run the relevant tests to establish the baseline failure. I’ll include postprocessor, tool parser, streaming filter/spec, and tool-calling tests; this should reveal whether the checked-in regression test already fails and whether XML-specific cases exist elsewhere.
- Assistant
That command had a missing separator between the pytest and grep portions, so pytest interpreted
shortgrepas a traceback mode. I’ll rerun the same baseline correctly. - Assistant
Tests cannot currently import because the environment’s MLX native library is unavailable; I’ll still use static/unit-level reproduction by stubbing MLX imports if feasible. First I’ll inspect existing postprocessor expectations and the non-streaming extraction path to preserve conventions and avoid changing parser behavior itself.
- Assistant
The XML parser confirms the concrete defect: streaming completion checks
"</tool_call>" in delta_text, so an end tag split across chunks is never recognized; final recovery should compensate, but finish ordering and reasoning accumulation can still prevent parity. I’ll inspect the helper selecting parsers and Hermes’s exact format support, then create a direct postprocessor reproduction using lightweight module stubs to avoid loading MLX. - Assistant
The complete failure mode is broader than a split end tag:
- Recovery currently parses
tool_accumulated_text or accumulated_text, which can be only a fragment when reasoning routing is active. finalize()skips recovery once any call was incrementally detected, so later calls can be dropped.- Incremental parsers may re-emit earlier calls when parsing the accumulated prefix, causing duplicates.
- Parser exceptions on malformed markup can terminate the stream.
- The OpenAI route sends
stopbefore end-of-stream recovery can replace it withtool_calls.
I’ll inspect all postprocessor adapters before changing semantics, then implement raw-output accumulation, call reconciliation/deduplication, exception-safe fallback, and delayed final finish emission in each affected adapter as appropriate.
- Recovery currently parses
- Assistant
Only the OpenAI chat stream uses this postprocessor, so finish buffering is localized. Before adding deduplication, I’ll verify whether parser outputs are complete call snapshots or true argument deltas; deduplicating by full
(name, arguments)is safe only for snapshots. - Assistant
Some parsers (notably Qwen3 Coder) emit genuine header/argument deltas, so global signature deduplication would break valid streaming. Instead, final reconciliation should use call indices already emitted: track the highest emitted index, always run full non-stream extraction at EOF, and emit only recovered calls at indices not previously observed. This supports split/multiple XML while preserving true deltas for existing calls. I’ll confirm Qwen index behavior and then patch state, fallback, exception handling, and finish buffering.
- Assistant
I’ll implement the processor changes now. The recovery call indices will be reconciled against indices already emitted, preserving Qwen’s incremental argument-delta behavior while allowing EOF recovery to append missing calls. The raw accumulator will be independent of reasoning/content routing, and parser errors will degrade to plain content rather than aborting.
- Assistant
Next I’ll delay only the terminal finish event in the OpenAI SSE adapter until EOF recovery completes. If recovery emits calls, its
tool_callsfinish replaces the pendingstop; otherwise the original finish (including final content/reasoning and usage behavior) is emitted unchanged. I’ll also synthesize a terminal event for parser-suppressed final chunks, ensuring every recovered tool stream ends correctly. - Assistant
The route compiles because annotations are postponed, but
StreamEventis instantiated at runtime in the synthesized branch and is not currently imported. I’ll inspect existing imports and add the domain event import (while confirmingGenerationOutputis already present), then build a no-MLX direct reproduction. - Assistant
I’ll add the missing runtime import.
- Assistant
For verification without native MLX, I’ll inspect
tests/conftest.pyfor the project’s intended stubbing strategy. If none exists, I’ll inject minimalmlxmodules in a temporary Python process (without adding repository files) and reproduce split/multiple Nemotron XML through the real parser and postprocessor. - Assistant
StreamingPostProcessoritself avoids engine imports at runtime, so a direct import may work even though the full config/route tests do not. I’ll run a temporary reproduction with a minimal config object and real Nemotron parser: two XML calls, with the second closing tag split, plus reasoning-routing and malformed-parser cases. - Assistant
The split/multiple reproduction passed. My reasoning-mode assertion incorrectly assumed recovery must produce a new finalize event, but the chosen split let the incremental redirect recognize the call. I’ll adjust the reproduction to split inside the opening marker so all XML remains in the reasoning channel and only the raw EOF accumulator can recover it; I’ll also inspect malformed behavior without assuming a specific process event shape.
- Assistant
The direct reproductions now pass all issue dimensions at processor level. I’ll run the postprocessor test module independently (it does not import config/engine and thus should avoid MLX), then address any expectation regressions.
- Assistant
All 74 processor tests pass. To exercise the SSE adapter and targeted streaming-spec tests, I’ll locate the installed
libmlx.so; the Python package exists but its shared-library search path is missing, so settingLD_LIBRARY_PATHmay enable tests without repository changes. - Assistant
No shared library is installed, so route-level pytest remains unavailable on this Linux environment. I’ll run formatting/lint checks on the changed files; this should catch import ordering, undefined names, line length, and typing issues that byte-compilation does not.
- Assistant
ruffis not installed. I’ll install the lightweight lint tool in the environment (not the repository), run it, and apply only source formatting/import fixes if needed. - Assistant
Lint passes; only the route’s new long conditional needs formatter normalization. I’ll format that file and recheck the diff.
- Assistant
The route diff now cleanly preserves existing finish payload construction while deferring it. I’ll run parser-focused tests independently, including Hermes/Nemotron/MiniMax and native parser suites, since they do not require starting the MLX engine.
- Assistant
The parser suites pass (272 tests), in addition to 74 postprocessor tests and the direct issue reproduction. I’ll now run the repository’s configured static type checker if available, focusing on the modified modules, and inspect the final diff for accidental broad changes.
- Assistant
Mypy is configured but not installed, so I’ll install it in the environment and type-check just the changed files using repository settings.
- Assistant
Mypy cannot parse an installed NumPy stub under the repository’s configured target, an environment/toolchain issue unrelated to these changes. To run the route tests despite missing MLX, I’ll use a temporary
sitecustomize.pyoutside the repository that replaces only the eagervllm_mlx.enginepackage initializer with its lightweightengine.baseexports. This avoids loadingengine_core/MLX while leaving production code and tests untouched. - Assistant
With the temporary engine stub, all 17 streaming-tool-filter tests and both fallback-recovery tests pass; the remaining three route tests fail only because admission checking lazily imports the MLX-backed scheduler. I’ll extend the external stub with the scheduler’s
BackpressureErrortype and rerun the five targeted route tests, which will exercise the delayed finish path. - Assistant
The route streaming-spec suite now passes under the external non-MLX stub. I’ll perform one final end-to-end SSE reproduction by calling the real
stream_chat_completiongenerator with a mock engine and real Nemotron parser configuration, asserting two calls are assembled across chunks and that the only terminal reason istool_calls. - Assistant
The end-to-end SSE reproduction passes with both ordered calls and exactly one
tool_callsterminal reason. I’ll do the final repository review: inspect both complete diffs, ensure only intended source files changed, rerun byte compilation/lint/diff checks, and summarize test evidence before submission.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →