ISSUE DESCRIPTION Streaming drops XML tool-call output that non-streaming parses…
- User
ISSUE DESCRIPTION
Streaming drops XML tool-call output that non-streaming parses correctly
Streaming must emit the same recovered XML tool calls as non-streaming, including calls split across chunks and multiple calls, and finish with
tool_calls. Reasoning mode and request tool selection must not alter recovery. Plain text or malformed markup must not create calls or abort the stream.Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /Rapid-MLX, current file: none) bash-$
- Assistant
I’ll first map the repository and locate the streaming/non-streaming response and XML tool-call parsing paths. Then I’ll reproduce the discrepancy with the narrowest relevant test or a temporary script, patch the shared behavior, and run targeted plus broader tests.
- Assistant
The initial search confirms this is an OpenAI-compatible server with several parser layers. I’ll now identify the exact recovery function and the chat route’s final streaming assembly, along with existing tests that encode parser and stream contracts.
- Assistant
There are two mechanisms: parser-specific incremental extraction and a generic
StreamingToolCallFilterthat suppresses XML. The issue wording (“recovered XML tool calls” matching non-streaming) points to the genericparse_tool_callsfallback not being applied at stream completion. I’ll inspect route control flow and filter semantics before choosing where to unify them. - Assistant
The full dump was truncated, so I’ll read only the relevant definitions by line number: generic parser orchestration, stream generator, and the XML filter. This should reveal whether raw generated text is retained long enough to perform end-of-stream recovery without changing the public interface.
- Assistant
The request plumbing passes tools into both modes, while non-streaming always invokes
_parse_tool_calls_with_parser. I’ll isolate the streaming generator now; the likely fix is to accumulate raw deltas and invoke that same helper at EOF only when incremental parsing produced no calls, preserving existing low-latency native parser output and avoiding duplicates. - Assistant
The stream already calls
processor.finalize(), so the bug is likely insideStreamingPostProcessor: either it only finalizes a configured parser, conditions recovery on tools/reasoning, or parses filtered rather than raw content. I’ll inspect that class and its unit tests, then reproduce by feeding split XML directly through it. - Assistant
The existing finalize regression only covers a configured parser and manually populated text. The reported cases also require generic recovery independent of parser configuration, reasoning mode, and tool selection. I’ll inspect every write to the accumulated buffers and finalize’s gate, then create a temporary direct reproduction using the actual
GenerationOutputshape. - Assistant
The non-streaming helper always falls back to generic
parse_tool_calls, even when no parser is configured or a configured parser misses. Streaming’s state indicates it does not universally accumulate raw standard/channel text and its fallback is parser-gated. I’ll capture the exact methods to make a minimal patch that mirrors the helper without introducing route coupling. - Assistant
A direct code trace reproduces the root cause: with no incremental parser, standard chunks are emitted as content, no raw buffer is retained, and
finalize()is gated onself.tool_parser. Reasoning/channel paths likewise do not provide a universal raw source. I’ll run a temporary script to confirm current events for split/multiple XML, plain text, and malformed markup, then patch raw accumulation plus generic EOF recovery. - Assistant
The first reproduction attempt hit the expected Linux environment limitation (
mlxnative library) when importing the engine package; this is infrastructure, not the behavior under test. Since the processor only requires output attributes, I’ll rerun with a simple namespace (the same isolation style used by unit tests) to confirm the bug without loading MLX. - Assistant
The reproduction confirms split and multiple valid XML are emitted only as sanitized content with
stop, while malformed text safely remains non-call content. I’ll add a dedicated raw-output accumulator at the processor entry point (so standard, reasoning, channel routing, and request tool choice cannot affect it), preserve configured-parser precedence, and invoke the same genericparse_tool_callsfallback used by non-streaming. Both fallback parsers will be exception-isolated so malformed output cannot terminate SSE. - Assistant
The patch now retains the exact raw model stream and applies configured then generic recovery at EOF. I’ll rerun the reproduction to verify split/multiple calls, finish reason, no false positives for plain/malformed output, and invariance when
tools_requestedchanges. - Assistant
The reproduction now recovers one split call and both multiple calls, with the last choice-bearing event finishing as
tool_calls; plain and malformed output produce no calls and do not raise. XML content may have already streamed before EOF when no incremental parser exists, but the issue only requires recovered structured calls; suppressing arbitrary potential markup until EOF would degrade normal text streaming and risks dropping malformed text.Next I’ll run the focused postprocessor/tool/stream suites. Before that, I’ll inspect finish-reason assertions to ensure the existing “stop then fallback tool_calls” convention is intentional and not contradicted by the API tests.
- Assistant
A focused legacy test directly seeds
tool_accumulated_textinstead of callingprocess_chunk, so the new universal raw buffer is empty in that synthetic setup. Production behavior is correct, but preserving this internal test convention is harmless: use raw text first, then the two legacy parser-specific buffers only when raw is absent. I’ll make that compatibility adjustment and rerun the focused suite. - Assistant
All 274 focused processor/parser tests pass. I’ll now validate the actual OpenAI streaming formatter and integrations, including final SSE ordering and existing reasoning behavior, then run a broader relevant test set.
- Assistant
The integration collection is blocked on Linux by the repository’s native
libmlx.sodependency before tests execute; the pure processor/parser suite avoids that import path and passed. I’ll still inspect the streaming-spec fixture to see whether its documented stub invocation can be used, and independently exercise reasoning/parser-error/request-choice variants through the processor. - Assistant
The edge reproduction confirms two recovered calls regardless of
tools_requested/tool choice, recovery when all XML is routed as reasoning, and exception-safe malformed parser output. The last choice-bearing event istool_callsin every valid case. I’ll perform final quality checks (format/lint/diff), run a few adjacent pure API tests, and inspect repository status for accidental files. - Assistant
Validation is clean: formatting/lint/syntax/diff checks pass, 274 focused parser/postprocessor tests pass, and 125 adjacent utility/sanitization/newline tests pass. The route-level suite cannot collect in this Linux container because installed MLX cannot load
libmlx.so; direct processor reproductions cover the affected logic and SSE event contract. I’ll do one final diff review to ensure the change is limited and comments accurately describe behavior, then submit.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →