ISSUE DESCRIPTION [FEATURE] support non Result return types for AiService when tool…
- User
ISSUE DESCRIPTION
[FEATURE] support non Result return types for AiService when tool execution is IMMEDIATE Is your feature request related to a problem? Please describe. If using ReturnBehavior.IMMEDIATE, your AiService method has to return Result even if all tools called return
voidor all returnnullor there's one tool method that returns same type as AiService return type.Describe the solution you'd like I'd like to support non-Result return types for AiService methods when tool execution has IMMEDIATE results based on these rules:
- If ai service method return type is void, just return null
- If any tool executions are TO_LLM throw exception (IMMEDIATE_IF_LAST case)
- If all tool executions have a null result and return type is nullable (not a primitive), then return null
- If there is one non-null tool execution result, and the result can be resolved to the return type, then return that object
I have a PR for this open in Quarkus too:
https://github.com/quarkiverse/quarkus-langchain4j/pull/2529
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /langchain4j, current file: none) bash-$
Assistant
Let me start by exploring the repository structure to understand the
Assistant
This is the langchain4j repository. Let me explore the structure to find the relevant files for the AiService and IMMEDIATE behavior
Assistant
Let me look at the broader context including the test files for understanding the existing behavior:
Assistant
Now let me look at the
ToolServiceContext:Assistant
Now let me look at the ToolExecution to see how the result is stored:
Assistant
Now let me look at the ToolService for the main executeInferenceAndToolsLoop method:
Assistant
Now let me look at the relevant test file:
Assistant
Now let me look at the PR mentioned in the issue to understand the proposed solution:
Assistant
Now let me look at the existing immediate return tests:
Assistant
Let me also look at where the existing IMMEDIATE tests are located:
Assistant
Now let me look at the PR mentioned in the issue. The PR is from quarkiverse/quarkus-langchain4j. Let me also check if there's a specific test for the issue:
Assistant
Let me now understand the flow better. First, let me look at the full DefaultAiServices method that handles the AI service method invocation:
Assistant
Now I have a good understanding of the flow. Let me check the Quarkus PR to see the specific changes needed:
Assistant
Now let me look at the code that I need to modify:
Assistant
Now I understand the code. Let me look at the PR mentioned in the issue. I need to understand the Quarkus PR. Since I don't have internet access, I'll need to design the solution based on the issue description.
Let me also look at the relevant tool execution result to see how the result object is stored:
Assistant
Now let me look at what we have to make my plan. Let me first write a test to confirm the current behavior:
Assistant
Now let me first build a small test to reproduce the issue. Let me understand the use case first:
Assistant
Now let me reproduce the issue. Let me create a test file:
Assistant
Now I've confirmed the issue. Let me look more closely at the code to understand the architecture and design my fix:
Assistant
Now let me understand the design. I think the approach is to modify the
ToolServiceandDefaultAiServicesto allow non-Result return types when there's an immediate return. Let me design the solution:Assistant
Let me look at the Quarkus PR through git to understand the approach:
Assistant
Now let me look at the test for the streaming immediate return path:
Assistant
Now let me think about the architecture. Let me look at the structure of
ToolServiceResultagain:Assistant
Now let me look at the AbstractAiServiceWithToolsIT for the test to see what tools return types are being tested:
Assistant
Let me check the subclasses that extend this abstract test:
Assistant
The test is only defined in the abstract class, not in the subclasses. It's a
protectedmethod, so subclasses can override or call it. Let me check the subclasses:Assistant
The
should_throw_using_immediate_tool_on_service_not_returning_Resulttest isn't called by subclasses. So it's a test that won't actually run unless called by a subclass or directly. Since the issue says we cannot modify existing tests, this test would just sit in the abstract class but not be exercised unless someone explicitly enables it.Now, the test itself wouldn't break with the new behavior because:
- The tool returns
int(a primitive that gets boxed toInteger) - The return type is
String - 124 (Integer) cannot be assigned to String
So with the new behavior:
- "If there is one non-null tool execution result, and the result can be resolved to the return type, then return that object" - Integer is NOT assignable to String
- So we still need to throw an exception
The test would still pass.
Now let me also think about: if the test calls
assistant.chat(text)where the assistant isStringAssistant, and the tool returns anint, with our new rules, we should throw an exception. The current behavior is also to throw an exception. So the test passes either way.But wait, the test name is
should_throw_using_immediate_tool_on_service_not_returning_Result. With the new behavior, we throw because the types don't match, not because the return type isn't Result. The test still asserts:isInstanceOf(IllegalConfigurationException.class)- still truehasMessageContaining(ReturnBehavior.IMMEDIATE.name())- might not be true anymorehasMessageContaining(Result.class.getName())- might not be true anymore
The error message might change, but the test expects those specific substrings. If our new error message doesn't include "IMMEDIATE" or "Result", the test would fail. Let me make sure the error messages are aligned.
Actually, looking at the test more carefully:
assertThatThrownBy(() -> assistant.chat(text)) .isInstanceOf(IllegalConfigurationException.class) .hasMessageContaining(ReturnBehavior.IMMEDIATE.name()) .hasMessageContaining(Result.class.getName());The test expects the error message to contain "IMMEDIATE" and "dev.langchain4j.service.Result". With my new implementation, the error would be different - it would be something like "Cannot resolve tool result to return type".
But the test is
protectedand isn't called by any subclass. Let me check again:- The tool returns
Assistant
Let me check if other tests in the abstract class are also protected:
Assistant
Let me check if these tests actually run in a normal test run:
Assistant
Let me look at the streaming case to understand it:
Assistant
Good. The streaming case is handled by
AiServiceTokenStream. The non-streaming case is the one I need to fix. Let me also look at theAiServiceTokenStreamto see if it has a similar pattern:Assistant
Let me check if there's configuration validation for return types with immediate:
Assistant
No validation at validation time. Now let me start implementing the fix.
First, let me design the architecture. I'll add a new
resultfield toToolServiceResultto hold the resolved value:Assistant
Now let me make the changes. First, let me update
ToolServiceResultto add aresultfield:Assistant
Now let me plan and implement the solution. The cleanest approach is:
- Add a
resultfield toToolServiceResultto hold the resolved value for non-Result return types - Modify
executeInferenceAndToolsLoopto accept return type info and compute the result - Update
DefaultAiServicesto pass return type and handle the result
Let me start implementing:
- Add a
Assistant
Now let me update the
ToolServiceto compute the result for non-Result return types:Assistant
Now let me make the changes to
ToolService.java. First, update the method signature and add the resolution logic:Assistant
Now let me modify the
executeInferenceAndToolsLoopmethod to:- Accept return type info
- Resolve the result based on the new rules when needed
Assistant
Now let me update the
ToolServiceto apply the changes. I'll make the changes by editing the file:Assistant
Let me check if there are any other callers of
executeInferenceAndToolsLoop:Assistant
Now let me implement the changes to
ToolService.java:Assistant
Now let me make the changes to the
ToolService.java:Assistant
Now I need to add the
Typeimport. Let me check if it's already there:Assistant
The perl substitution added a duplicate. Let me fix this:
Assistant
Good. Now let me add the
resolveImmediateReturnValuemethod:Assistant
Now let me add the
resolveImmediateReturnValuemethod right aftershouldReturnImmediately. Let me find a good location:Assistant
I need to import
TypeUtilsand a few other things. Let me first add the necessary imports:Assistant
Now let me add the
resolveImmediateReturnValuemethod. I'll add it right aftershouldReturnImmediately:Assistant
Now let me update
DefaultAiServices.javato pass the return type:Assistant
Let me try a simpler approach - I'll edit the file directly using the
editcommand:Assistant
Now let me edit the file using the
editcommand. First, let me find a unique text to replace:Assistant
Let me verify the change:
Assistant
Now let me also check the returnType variable to ensure it's accessible:
Assistant
Good.
returnTypeis defined and accessible. Now let me try to build:Assistant
The compilation succeeded. Now let me run my reproduce test:
Assistant
The first test passed. Now let me create more comprehensive tests to verify all the cases:
Assistant
The test for the case where the tool returns null has an issue. Let me check
ToolExecutionResult:Assistant
Let me trace through what happens when the tool returns null:
Assistant
The error is from
ToolExecutionResultMessagenot fromToolExecutionResult. So when a tool returns null, the message building fails because both text and contents are null.Let me look at
ToolExecutionResultMessage:Assistant
This is a pre-existing issue. Let me check what the existing test
should_allow_empty_tool_resultdoes:Assistant
Let me check
Json.toJson:Assistant
Let me fix the pre-existing issue in
toTextto handle null results:Assistant
The perl substitution didn't work. Let me try a different approach:
Assistant
Now let me re-run the test:
Assistant
All tests pass. Now let me also write more comprehensive tests for the IMMEDIATE_IF_LAST case and other scenarios:
Assistant
Let me look at the failing test more carefully:
Assistant
The test was wrong - I need to fix it. Let me update the test:
Assistant
All 11 tests pass. Now let me run the existing tests to make sure I haven't broken anything:
Assistant
All 66 existing tests pass. Now let me run more tests to make sure I haven't broken anything else:
Assistant
All tests in the langchain4j module pass. Let me now also run the tests in other modules that might be affected:
Assistant
Only the
findToolsmethod is used. Let me also check the streaming case:Assistant
Good. The streaming case uses
shouldReturnImmediatelydirectly. Now let me also run more tests in the entire codebase to make sure nothing is broken:Assistant
All tests pass! Now let me also check the agentic module since it might use the ToolService:
Assistant
Good, only my files. Now let me also look at the existing tests to make sure they pass:
Assistant
All tests pass. Now let me also run the agentic tests since they may use the AI services:
Assistant
All tests pass. Now let me check the existing test in
AbstractAiServiceWithToolsIT.javamore carefully:Assistant
This tool returns
int. The testshould_throw_using_immediate_tool_on_service_not_returning_ResultusesStringAssistantand this tool. With the new rules:- Return type is String (not Result, not void, not primitive)
- Tool returns Integer(124)
- 124 is NOT assignable to String
- So we should throw
But the test expects:
isInstanceOf(IllegalConfigurationException.class)✓hasMessageContaining(ReturnBehavior.IMMEDIATE.name())- my error message contains "IMMEDIATE" ✓hasMessageContaining(Result.class.getName())- my error message contains "Result" ✓
Wait, my error message is:
"Cannot resolve tool result to AI Service return type java.lang.String: the single non-null tool result of type java.lang.Integer is not assignable to java.lang.String"This doesn't contain "IMMEDIATE" or "Result"!
Let me check the test that I'm worried about:
Assistant
I need to update the error messages to include "IMMEDIATE" and "Result" substrings to maintain backward compatibility with the existing test. Let me update the error messages:
Assistant
Let me update the error messages to include "IMMEDIATE" and "Result" to maintain backward compatibility:
Assistant
The last error message still has the old format. Let me fix it manually:
Assistant
Now let me also check the
TO_LLMrule:Assistant
Now let me also handle the case where the return type is generic (e.g.
List<String>). Let me also check whether I should consider theisReturnTypeResultflag earlier. Let me also make sure the streaming case is properly handled.Let me first run all my tests to make sure nothing is broken:
Assistant
Now let me also test the abstract test (which doesn't run by default) to make sure my error messages pass:
Assistant
The test fails because the mock model doesn't make a tool call. Let me update the test:
Assistant
The tool argument name is "arg0" not "a". Let me fix the test:
Assistant
The test passes! The error message includes both "IMMEDIATE" and "Result" as expected by the existing test in
AbstractAiServiceWithToolsIT.java.Now let me delete my test files and run the entire test suite to make sure everything passes:
Assistant
Good, my test files are removed. Now let me run the full test suite for the
langchain4jmodule:Assistant
All tests pass. Let me also run the langchain4j-skills tests since I changed
DefaultToolExecutor:Assistant
All tests pass. Now let me think about edge cases:
-
What if the return type is a generic type like
List<String>and the tool returnsList<String>? This is aParameterizedType. ThegetRawClass()should returnList.class, andList.class.isInstance(result)should return true. Let me test this. -
What if the return type is
Object?Object.class.isInstance(anything)returns true, so any tool result would be assignable. -
What if the return type is
Stringand the tool returnsObject? The result would be assignable becauseString.class.isInstance(object)is false. Wait, but my code callsgetRawClass(result.getClass())for the error message. Let me re-read my code:
Actually, my code does:
if (isAssignableTo(result, rawReturnType)) { return result; }Where
isAssignableTocheckstargetType.isInstance(value). SoString.class.isInstance("hello")is true.String.class.isInstance(42)is false. Good.-
What if the tool has no return value (void)? Then
resultObject()returns null. We should treat this as a null result. -
What if multiple tools are called and some return null and some return values? The current logic counts all non-null results. If multiple non-null, throw. If one non-null, return that.
Let me also think about the case where the tool has a primitive return type. The result of a primitive int would be
Integer(autoboxed). Soint.class.isInstance(42)is false, butint.classwould needInteger.class.isInstance(42). Let me re-check my code:private static boolean isAssignableTo(Object value, Class<?> targetType) { if (targetType.isInstance(value)) { return true; } if (!targetType.isPrimitive()) { return false; } return primitiveWrapperFor(targetType).isInstance(value); }This handles the case where the target is primitive and the value is the corresponding wrapper. Good.
- What about the
Resultfield whenisReturnTypeResultis true? I leave it as null. The caller inDefaultAiServiceschecksisReturnTypeResultand builds aResultwrapping null.
Let me also think about whether I should handle the case where the immediate return is triggered but
aiServiceReturnTypeis null. Actually, my code does:if (aiServiceReturnType == null) { throw illegalConfiguration(...); }So it throws an error if the return type is null. But this should not happen in practice because
DefaultAiServicesalways provides a return type.Now let me also think about whether the existing test
should_throw_using_immediate_tool_on_service_not_returning_Resultshould still pass. Looking at it:StringAssistant(return type String)ImmediateToolWithPrimitiveParameters(returns int 124)- The test expects an exception with "IMMEDIATE" and "Result" in the message
With my new behavior:
- Return type is String, tool returns Integer(124)
- Integer is not assignable to String
- I throw an error with "IMMEDIATE" and "Result" in the message
- The test should pass
Now let me also look at the existing behavior. The test is checking the OLD behavior. But with my new behavior, the error message format is different. Let me check if the test would still pass:
The test is
protected void should_throw_using_immediate_tool_on_service_not_returning_Result(ChatModel model)and the model is an actual chat model (not a mock). The mock in my test was different from a real chat model.Actually, looking at the test again, the test is testing the OLD behavior where ANY non-Result return type with IMMEDIATE tools throws an error. The test is in the abstract class but it's
protectedand not annotated with@Testdirectly - it's a@ParameterizedTestwith@MethodSource("models").Wait, let me check:
-
Assistant
The test is
@ParameterizedTestand usesmodels()which is overridden in subclasses. So when subclasses extend this class, the test will be inherited and run.Now let me verify the test will pass with my new error message. The test expects:
isInstanceOf(IllegalConfigurationException.class)✓hasMessageContaining(ReturnBehavior.IMMEDIATE.name())- "IMMEDIATE" ✓ (my message contains "IMMEDIATE")hasMessageContaining(Result.class.getName())- "dev.langchain4j.service.Result" ✓ (my message contains "Result")
So the test should pass.
Let me also look at the test for the streaming case to make sure I haven't broken it:
Assistant
No streaming tests for this case. Now let me also verify my changes don't break anything by running all the tests in the langchain4j module:
Assistant
All 1385 tests pass with only 2 skipped (which is expected). Let me also check the agentic module tests:
Assistant
All tests pass. Now let me also look at my code to see if there are any other improvements. Let me look at the
shouldReturnImmediatelystatic method:Assistant
Let me create a final test to verify the issue is fully fixed:
Assistant
Objectis not a valid return type for AI services. Let me remove that test:Assistant
The sed command didn't fully remove the test - it left duplicate @Test annotations. Let me rewrite the test:
Assistant
All 11 tests pass! Now let me delete this temporary test and run the full test suite:
Assistant
All 1385 tests pass. Now let me also verify the issue is fully addressed by looking at the Quarkus PR. Let me also check my final code:
Assistant
Let me also look at the case where the return type is a
ParameterizedType(likeList<String>):Assistant
Good.
getRawClass()handlesClass<?>andParameterizedType. For aList<String>, it returnsList.class. ThenList.class.isInstance(result)works correctly.Let me also think about the streaming case. Looking at
AiServiceStreamingResponseHandler:Assistant
Let me also check the
isReturnTypeResultflag in the rest of the code:Assistant
Good, the existing logic for
Resulthandling is preserved.Now let me look at the full picture of the changes I've made:
Assistant
Let me look at the diff of each file:
Assistant
Let me also run the tests one more time to make sure everything passes:
Assistant
All 1385 tests pass. Let me also run the agentic and skills tests:
Assistant
All tests pass. Let me also run the agentic-patterns and langchain4j-mcp tests:
Assistant
All tests pass. Now let me also write a quick test to verify the changes work correctly:
Assistant
All 4 tests pass. Now let me delete the verification test:
Assistant
All my temporary tests are cleaned up. Let me also verify the diff one more time to make sure everything is correct:
Assistant
Let me also do a final check by running the full test suite:
Assistant
All 1385 tests pass with 0 failures and 0 errors.
Now let me also look at whether my changes affect the streaming case. Looking at
AiServiceStreamingResponseHandler:Assistant
Let me also check if my changes properly handle the case when the
aiServiceReturnTypeis a generic type:Assistant
Generic types work too! Let me clean up:
Assistant
All test files are cleaned up. Let me also look at the final diff of the changes I made:
Assistant
Now let me also go over the checklist:
Checklist:
-
If an error was reported in the issue description, I have successfully reproduced it.
- I created a test that showed the error:
AI Service method must return a dev.langchain4j.service.Result type to use tools with ReturnBehavior.IMMEDIATE/IMMEDIATE_IF_LAST
- I created a test that showed the error:
-
If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I verified the fix works by writing multiple test cases that exercise the new behavior:
voidreturn type with various tool return types- Non-
Resultreturn type matching tool return type - All null results with nullable return type
- Error cases for type mismatches, multiple non-null results, etc.
- Mix of
TO_LLMandIMMEDIATEtools - Primitive vs wrapper types (
intvsInteger)
- I verified the fix works by writing multiple test cases that exercise the new behavior:
-
I have thought about edge cases and made sure my fix handles them as well.
- Primitive return types vs wrapper types
- Generic types like
List<String> - Mix of
TO_LLMandIMMEDIATE/IMMEDIATE_IF_LASTin the same response - Multiple non-null tool results
- All null tool results with primitive return type (throws error)
- Failed tool executions (throws error)
- Void return type with various tool return types
- Pre-existing test for
should_throw_using_immediate_tool_on_service_not_returning_Resultstill passes
-
I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass.
- All 1385 tests in the
langchain4jmodule pass - All tests in
langchain4j-agentic,langchain4j-skills,langchain4j-agentic-patterns,langchain4j-agentic-a2a,langchain4j-agentic-mcppass - All
ReturnBehaviorCombinationsTesttests pass (66 tests) - All
ProgrammaticCreatedImmediateReturnToolTesttests pass - All
StreamingProgrammaticCreatedImmediateReturnToolTesttests pass
- All 1385 tests in the
Let me also do a final test run to make sure everything is in order:
-
Assistant
All 1385 tests pass. My implementation is complete. Let me also list the changes I made:
Summary of Changes
1.
langchain4j/src/main/java/dev/langchain4j/service/tool/ToolServiceResult.java- Added a new field
immediateReturnValueto hold the resolved value when the AI service method does not returnResultand the tool execution returned immediately. - Added getter method
immediateReturnValue()and builder methodimmediateReturnValue(Object). - Updated
equals(),hashCode(), andtoString()to include the new field.
2.
langchain4j/src/main/java/dev/langchain4j/service/tool/ToolService.java- Modified
executeInferenceAndToolsLoopto accept the AI service method's return type (Type aiServiceReturnType). - When
shouldReturnImmediatelyreturns true and the return type is notResult, call the newresolveImmediateReturnValuemethod to compute the value to return. - Added the new
resolveImmediateReturnValuemethod that implements the rules:- If the AI service return type is
void, returnnull - If any tool in the response is
TO_LLM, throw anIllegalConfigurationException - If every tool result is
nulland the return type is nullable, returnnull - If there is exactly one non-null tool result, and it's assignable to the return type, return that
- Otherwise, throw an
IllegalConfigurationExceptionwith a descriptive message
- If the AI service return type is
- Added helper methods
isAssignableToandprimitiveWrapperForto handle primitive type compatibility.
3.
langchain4j/src/main/java/dev/langchain4j/service/DefaultAiServices.java- Updated the call to
executeInferenceAndToolsLoopto pass the AI service method's return type. - Updated the
immediateToolReturncheck to handle bothResultand non-Resultreturn types:- If
Result, build aResultwrappingnullcontent (existing behavior) - If non-
Result, return the resolved value fromtoolServiceResult.immediateReturnValue()
- If
4.
langchain4j/src/main/java/dev/langchain4j/service/tool/DefaultToolExecutor.java- Fixed a pre-existing bug in
toTextwhere aStringtool returningnullwould cause anIllegalArgumentException("Either text or contents must be provided") when building theToolExecutionResultMessage. - Now uses
Json.toJson(result)forStringreturn types withnullresults, which returns the string"null".
- Added a new field
Assistant
Now let me submit the changes:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →