ISSUE DESCRIPTION Add reasoning.effort support for OpenAI gpt-realtime-2 Feature Type…
- User
ISSUE DESCRIPTION
Add reasoning.effort support for OpenAI gpt-realtime-2
Feature Type
Would make my life easier
Feature Description
OpenAI
gpt-realtime-2supports configurable reasoning viareasoning.effort, but the Python OpenAI Realtime plugin does not seem to expose this option onopenai.realtime.RealtimeModel.It would be useful to support something like:
from livekit.plugins import openai llm = openai.realtime.RealtimeModel( model="gpt-realtime-2", reasoning={"effort": "low"}, )Workarounds / Alternatives
I don't think there is a clean workaround through the current Python
RealtimeModelAPI.The only workaround I can see is patching the plugin locally to add
reasoningto the session update payload.Additional Context
OpenAI docs list these effort values:
minimal,low,medium,high,xhigh.References:
- https://developers.openai.com/api/docs/models/gpt-realtime-2
- https://developers.openai.com/api/docs/guides/realtime-models-prompting#set-reasoning-effort
- Related JS issue: https://github.com/livekit/agents-js/issues/1433
- Related JS PR: https://github.com/livekit/agents-js/pull/1575
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /agents, current file: none) bash-$
- Assistant
I'll start by exploring the repository structure to understand the codebase and locate the OpenAI realtime plugin. Let's first see what we're working with.
- Assistant
Let me look at the actual repository structure, not the virtual environment.
- Assistant
Let me explore the livekit-plugins directory to find the OpenAI plugin.
- Assistant
Let me explore the OpenAI plugin structure.
- Assistant
Let me check the exact path.
- Assistant
Now let me examine the realtime model file which is the main focus of this issue.
- Assistant
Let me continue reading the file to see the RealtimeModel class definition.
- Assistant
Let me continue reading to find the RealtimeModel class definition.
- Assistant
Let me continue reading the RealtimeModel class to see all its overloads and implementation.
- Assistant
Let me continue reading the init method to see how options are stored and then find where the session update is sent.
- Assistant
Now I need to find where the session update is sent to understand how to add reasoning support. Let me search for where session updates are created.
- Assistant
Let me look at the
_create_session_update_eventmethod to understand how the session configuration is built. - Assistant
Let me continue reading the update_options method to see how other options are handled.
- Assistant
Let me check what the RealtimeSessionCreateRequest type looks like to see if it already supports a reasoning field.
- Assistant
Let me look at the venv
- Assistant
Now let me check what
RealtimeReasoninglooks like. - Assistant
Let me check the effort type.
- Assistant
Now I understand the structure. The OpenAI SDK already supports
RealtimeReasoninginRealtimeSessionCreateRequest, but the plugin'sRealtimeModeldoesn't expose it. I need to:- Add
reasoningto_RealtimeOptions - Add
reasoningparameter toRealtimeModel.__init__(all overloads) - Pass
reasoningthrough to_create_session_update_event - Add
reasoningtoupdate_options - Map
reasoningin_oai_session_to_azure
Let me first check if there are any Azure-related reasoning fields in the Azure types.
- Add
- Assistant
Let me check if AzureSession has a reasoning field.
- Assistant
Let me check the
models.pyfile for any type definitions related to reasoning. - Assistant
Let me look at the
__init__.pyto see what's exported. - Assistant
Now I have a clear picture of what needs to be done. Let me implement the changes:
- Import
RealtimeReasoningfromopenai.types.realtime - Add
reasoningto_RealtimeOptionsdataclass - Add
reasoningparameter to all three__init__overloads - Set
reasoningin the constructor - Set
reasoningonRealtimeSessionCreateRequestin_create_session_update_event - Add
reasoningtoupdate_options
Let me start implementing these changes:
- Import
- Assistant
Let me now look at the exact imports section and add the
RealtimeReasoningimport: - Assistant
Now let me add the
RealtimeReasoningimport: - Assistant
Let me verify the import was added correctly:
- Assistant
Now let me add
reasoningto the_RealtimeOptionsdataclass: - Assistant
Let me add
reasoningto the_RealtimeOptionsdataclass: - Assistant
I need to add
reasoning: RealtimeReasoning | Noneaftertruncationin the_RealtimeOptionsdataclass. - Assistant
Let me verify the dataclass change:
- Assistant
Now let me add the
reasoningparameter to the first__init__overload (line ~269 originally, now offset by 2): - Assistant
Now let me add the
reasoningparameter to the first__init__overload aftertruncation(currently on line 288): - Assistant
Let me verify the first overload change:
- Assistant
Now let me update the second
__init__overload (for Azure): - Assistant
I need to add
reasoningto the second overload as well. Let me find the line number fortruncationin the second overload: - Assistant
Now let me add
reasoningto the second overload (Azure-specific) aftertruncationon line 320: - Assistant
I need to add reasoning after truncation in the second overload too.
- Assistant
Now let me add
reasoningto the third__init__overload (the actual implementation): - Assistant
Let me look at the third overload (the actual implementation) around line 347:
- Assistant
I need to add reasoning after truncation in the third overload too.
- Assistant
Now let me add
reasoningto the docstring. Let me find the docstring: - Assistant
Let me add the
reasoningparameter to the docstring: - Assistant
Let me add the reasoning docstring after truncation:
- Assistant
Now let me update the constructor body to handle the
reasoningparameter. Let me find where the options are created: - Assistant
Let me look at the constructor body around line 461:
- Assistant
Now I need to add the
reasoningfield to the_RealtimeOptionsconstructor call. Let me add it aftertruncation: - Assistant
Now I need to create the
_to_reasoninghelper function. Let me first check how other helper functions liketo_audio_transcriptionandto_noise_reductionwork: - Assistant
Let me look at those helper functions:
- Assistant
Let me check the imports from utils:
- Assistant
Let me check the structure of utils.py to understand the pattern:
- Assistant
I'll add a
to_reasoninghelper function in utils.py. Let me find a good place to add it: - Assistant
Let me look at the end of the
to_turn_detectionfunction to find where to add the new helper: - Assistant
Let me add the
to_reasoninghelper function afterto_turn_detection: - Assistant
Let me add the
to_reasoninghelper function after line 128: - Assistant
Let me verify the addition:
- Assistant
Now I need to import
to_reasoningin the realtime_model.py file. Let me add it to the imports: - Assistant
Let me add
to_reasoningto the imports: - Assistant
Let me verify the import was added correctly:
- Assistant
Now I need to update the
_create_session_update_eventmethod to includereasoning. Let me find that method: - Assistant
Let me look at the
_create_session_update_eventmethod: - Assistant
Now I need to add
reasoningto theRealtimeSessionCreateRequestconstructor call. I'll add it aftertruncation: - Assistant
Let me verify the change:
- Assistant
The indentation got messed up. Let me fix it:
- Assistant
The lines got inserted inside the return statement. I need to fix this. Let me use replace to fix it:
- Assistant
Let me verify the fix:
- Assistant
Now I need to add
reasoningto theupdate_optionsmethod. Let me find it: - Assistant
Let me look at both
update_optionsmethods. Let me first check the one at line 1216: - Assistant
Now I need to add
reasoningto theupdate_optionsmethod signature and handling. Let me first add it to the signature: - Assistant
Let me verify the change:
- Assistant
Now I need to add the handling code for
reasoningin the body ofupdate_options. Let me find wheretruncationis handled: - Assistant
Let me look at the truncation handling in
update_options(the second one at line 1255): - Assistant
Now let me add the reasoning handling after the truncation handling:
- Assistant
Let me verify the change:
- Assistant
Now I need to check the first
update_optionsmethod at line 658 (which is likely on theRealtimeModelclass, notRealtimeSession): - Assistant
I see there's also an
update_optionsmethod in theRealtimeModelclass. This one also needs to be updated with thereasoningparameter. Let me look at the full method: - Assistant
I need to add
reasoningto theRealtimeModel.update_optionsmethod as well. Let me add it to the signature first: - Assistant
Let me verify the change:
- Assistant
The line got inserted in the wrong place again. Let me fix this properly using a Python script:
- Assistant
Let me verify the fix:
- Assistant
There's still a misplaced line. Let me fix this properly with a Python script that fixes all the issues:
- Assistant
Let me verify the fix:
- Assistant
Good, the signature is fixed. Now I need to add the handling for
reasoningin the body ofRealtimeModel.update_optionsand pass it to sessions. Let me see the full method: - Assistant
Now I need to add reasoning handling after truncation and pass it to sessions. Let me first add the handling after truncation:
- Assistant
Let me add the reasoning handling after line 704 (truncation handling in RealtimeModel.update_options):
- Assistant
Let me add the reasoning handling after line 704:
- Assistant
Let me verify the change:
- Assistant
Good, now I need to add
reasoningto thesess.update_optionscall. Let me find it: - Assistant
I need to add
reasoning=reasoning,aftertruncation=truncation,on line 718: - Assistant
Let me verify the change:
- Assistant
Good. Now I need to verify that the
RealtimeReasoningtype is properly imported. Let me check the import I added earlier: - Assistant
Now let me run the tests to make sure everything works. Let me first check if there are any existing tests for the realtime model:
- Assistant
Let me check the tests to see if there are any tests for the OpenAI realtime model:
- Assistant
Let me check the test files to see what they test:
- Assistant
Let me run a basic import test to make sure our changes don't break anything:
- Assistant
Good, the import works. Now let me test that we can create a RealtimeModel with the reasoning parameter:
- Assistant
I used
_to_reasoninginstead ofto_reasoning. Let me fix that: - Assistant
I need to fix the function name:
- Assistant
Let me verify the fix:
- Assistant
Now let me run the test again:
- Assistant
All tests pass. Now let me check that the session update event includes reasoning when it's set. Let me also check if there are any other places where reasoning might need to be handled, such as in the
_oai_session_to_azurefunction: - Assistant
Let me check if the Azure session type has a reasoning field:
- Assistant
Good, Azure doesn't have a reasoning field, so I don't need to add it to
_oai_session_to_azure. Now let me check if there are any other places where reasoning might need to be handled. Let me also run the existing tests to make sure nothing is broken: - Assistant
Good. Let me also check the
RealtimeSession.connectmethod to ensure reasoning is properly passed through: - Assistant
Let me look for how sessions are started and the session update event is sent:
- Assistant
Let me look at how the session uses the model options:
- Assistant
Good, the session copies options from the model via
self._opts = replace(realtime_model._opts), which will include thereasoningfield. Let me also check if there are any other tests I should run. Let me run a more comprehensive test: - Assistant
Let me run the existing tests to make sure nothing is broken:
- Assistant
Let me also check if there are any type checking issues. Let me run a basic type check:
- Assistant
All typing tests pass. Let me now also verify that the
reasoningis properly included in the session update event by checking the_create_session_update_eventmethod one more time: - Assistant
The code looks correct. Let me also verify the
_oai_session_to_azurefunction to see if we need to handle reasoning there (though Azure doesn't support it): - Assistant
Good, the
_oai_session_to_azurefunction doesn't need to handlereasoningsince Azure doesn't support it. Let me now run the existing tests to make sure nothing is broken: - Assistant
The errors are unrelated to our changes (missing bs4 module and invalid onnx model file). Let me run a more targeted test:
- Assistant
Let me now also check if there are any other session update paths that might need reasoning. Let me look at the
reconnectcode: - Assistant
Good, the reconnect method uses
self._create_session_update_event()which we've already updated to include reasoning. Let me now create a comprehensive test script to verify everything works correctly: - Assistant
All tests pass. Let me now also check if there are any other files that might need updates. Let me check the main
__init__.pyof the openai plugin: - Assistant
The main
__init__.pydoesn't need to exportRealtimeReasoning- it's already accessible through the OpenAI SDK. Let me do a final review of all changes by checking the git diff: - Assistant
Let me check the full diff to make sure everything is correct:
- Assistant
The diff looks good. Let me now run one more comprehensive test to make sure everything works correctly:
- Assistant
All tests pass. Let me now also check if there are any existing tests that might be affected by our changes:
- Assistant
All existing tests pass. Let me now summarize the checklist items:
-
If an error was reported in the issue description, I have successfully reproduced it. - No error was reported; this is a feature request. I implemented the feature.
-
If an error was reported in the issue description, I have confirmed that my fix resolves the error. - Not applicable, as no error was reported.
-
I have thought about edge cases and made sure my fix handles them as well. - Yes, I handled:
- RealtimeReasoning object input
- dict input (converted to RealtimeReasoning)
- None input (sets reasoning to None)
- NOT_GIVEN (default, leaves reasoning as None)
- All valid effort values: 'minimal', 'low', 'medium', 'high', 'xhigh'
- update_options support (both RealtimeModel and RealtimeSession)
- Azure compatibility (reasoning is not sent to Azure since it doesn't support it)
-
I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. - Yes, I ran:
- test_config.py: PASSED
- test_personaplex_realtime_model.py: PASSED (all 36 tests)
- All tests passed.
Let me also verify the import of
RealtimeReasoningis correct: -
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →