ISSUE DESCRIPTION Stale 'clientversion' fallback causes 503 'noplansupportformodel' for…
- User
ISSUE DESCRIPTION
Stale 'client_version' fallback causes 503 'no_plan_support_for_model' for newer models (e.g. gpt-5.5) when GitHub releases API rate-limits
Summary
When
app/core/clients/codex_version.pycannot fetch the latest codex release fromapi.github.com/repos/openai/codex/releases/latest(e.g. anonymous-rate-limited with HTTP 403), it logs:GitHub releases API returned HTTP 403 GitHub fetch failed and no cached version; falling back to settings default 0.101.0and falls back to
Settings.model_registry_client_version(default0.101.0). The model registry refresh (app/core/openai/model_refresh_scheduler.py→clients/model_fetcher.fetch_models_for_plan) then asks upstream/codex/models?client_version=0.101.0. Upstream gates newer model slugs behind newer client versions, so the response excludes e.g.gpt-5.5.As a result,
ModelRegistry.model_plansnever gets agpt-5.5entry,_filter_accounts_for_modelinmodules/proxy/load_balancer.py:1168returns an empty set, and every Codex request forgpt-5.5is rejected with:proxy_error_response status=503 code=no_plan_support_for_model message=No accounts with a plan supporting model 'gpt-5.5'…even though the actual Codex CLI is on
0.130.0and the account's ChatGPT subscription does support the model (calling/backend-api/codex/models?client_version=0.130.0through the LB returns 200 with the model present).Reproduction
- Run
ghcr.io/soju06/codex-lb:1.17.0on a host where anonymous GitHub API requests are 403'ed. - Connect a current Codex CLI (e.g.
@openai/codex@0.130.0) configured withmodel = \"gpt-5.5\". - Observe startup log line:
GitHub fetch failed and no cached version; falling back to settings default 0.101.0
- Every
POST /backend-api/codex/responsesreturns 503 with codeno_plan_support_for_model.
Setting
CODEX_LB_MODEL_REGISTRY_CLIENT_VERSION=0.130.0and recreating the container fully resolves the symptom.Suggested fixes
A few low-effort options, any one would help:
- Bump the baked-in default
model_registry_client_versionon each release so it tracks the recent@openai/codexversion (or read it from a generated file at build time). - Add a secondary version source.
npm view @openai/codex versionagainsthttps://registry.npmjs.orgis not rate-limited the way the GitHub anonymous API is, and a static fetch ofhttps://registry.npmjs.org/@openai/codex/latestreturns JSON containing\"version\". - Cache the fetched version on disk (e.g. inside the data volume) so a single successful lookup survives container restarts and survives transient 403s for the cache TTL.
- Honor the client's own
client_versionquery string as a hint for which model list to refresh (only when greater than the configured default), so a newer client effectively bootstraps the registry.
Workaround
For anyone hitting this, pass
-e CODEX_LB_MODEL_REGISTRY_CLIENT_VERSION=<your-codex-cli-version>todocker run:docker run -d \\ -e CODEX_LB_DASHBOARD_BOOTSTRAP_TOKEN=... \\ -e CODEX_LB_MODEL_REGISTRY_CLIENT_VERSION=0.130.0 \\ -p 2455:2455 -p 1455:1455 \\ -v codex-lb-data:/var/lib/codex-lb \\ ghcr.io/soju06/codex-lb:1.17.0Environment
- codex-lb image:
ghcr.io/soju06/codex-lb:1.17.0 - Codex CLI:
@openai/codex@0.130.0 - Host: macOS, Docker Desktop / Colima
- Account plan:
pro(3 accounts, allplan_type=proinstore.db)
Related: #480 ("Model metadata for
gpt-5.5not found. Defaulting to fallback metadata") looks like the same underlying cause.ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /codex-lb, current file: none) bash-$
- Run
Assistant
I'll start by exploring the repository structure to understand the codebase and then locate the relevant files mentioned in the issue.
Assistant
Let me explore the directory structure more carefully.
Assistant
Now let me look at the key files mentioned in the issue:
Assistant
I made an error with the arguments format. Let me use the correct format for the open command.
Assistant
Now let me look at the settings file to understand the default:
Assistant
Now let me look at the other files mentioned in the issue:
Assistant
Now let me check the load balancer file to understand the 503 error:
Assistant
Now let me check the existing tests related to
codex_version.py:Assistant
Now let me understand the full structure of the repository better and check the
.env.examplefile:Assistant
Now let me review the issue more carefully and develop a plan. The issue suggests 4 possible fixes:
- Bump the baked-in default
model_registry_client_versionon each release - Add a secondary version source (NPM registry)
- Cache the fetched version on disk
- Honor the client's own
client_versionquery string as a hint
Let me also check if there are any more details about how clients pass their version:
- Bump the baked-in default
Assistant
Let me look at the model registry to understand how it works:
Assistant
Now I have a clear understanding of the issue. Let me plan my approach:
Problem Analysis:
- When GitHub API is rate-limited (403),
CodexVersionCachefalls back to the hardcoded default0.101.0 - This stale version is used to fetch models from upstream, which doesn't include newer models like
gpt-5.5 - Result: 503 errors for newer models even though they should be supported
My Fix Plan: Following the suggested fixes from the issue, I'll implement:
- Option 3: Persistent disk cache - Cache the fetched version on disk so it survives restarts and transient 403s
- Option 2: NPM registry as fallback - Add a secondary version source when GitHub fails
Let me first check how the data directory is used elsewhere:
- When GitHub API is rate-limited (403),
Assistant
Let me check the conftest for test fixtures:
Assistant
Now let me implement my solution. I'll modify
codex_version.pyto:- Add a persistent disk cache for the fetched version
- Add NPM registry as a secondary fallback source
- Keep the existing stale cache fallback behavior
Let me first check what the npm registry API looks like:
Assistant
Now I understand the issue clearly. Let me implement the fix by modifying
codex_version.pyto:- Add a persistent disk cache that survives restarts
- Add NPM registry as a secondary version source when GitHub fails
Assistant
Now I need to update the conftest.py to also reset the persistent cache state:
Assistant
I need to update the conftest to also reset the persistent cache fields:
Assistant
I need to update the
_reset_codex_version_cachefixture to also reset the persistent cache fields:Assistant
I see the sed only replaced the first occurrence. Let me fix the yield section too:
Assistant
Good, the conftest is updated correctly. Now let me run the existing tests to make sure they pass:
Assistant
The tests are failing because my persistent cache is picking up state from previous tests. The issue is that the conftest fixture resets the persistent cache fields, but the tests that run earlier (like
test_fallback_to_stale_cache_on_github_failure) populate the persistent cache and it's being reused by subsequent tests.The issue is that my tests share a common persistent cache file (in memory, since we're not using actual disk). I need to make the persistent cache isolated per test. Looking at the failures more carefully:
- Tests that populate the cache in earlier runs (like
test_fallback_to_stale_cache_on_github_failurewith1.5.0) - Later tests fail because the persistent cache is still populated
The conftest resets
_persistent_versionand_persistent_aton each test, but there's an issue: the conftest fixture only resets beforeyield, and tests that run aftertest_fallback_to_stale_cache_on_github_failurewill see the persistent cache from that test.Wait, let me re-check the conftest. The fixture should reset both before and after the test. Let me verify:
- Tests that populate the cache in earlier runs (like
Assistant
The fixture looks correct. But looking at the test order, the tests run in sequence and there seems to be state leaking. Let me check if the tests themselves create a new cache instance or use the global one:
Assistant
I see the issue! The tests create new
CodexVersionCacheinstances, and my_load_persistent_cache()method is called in__init__. This means each new test creates a fresh cache instance that loads from the persistent cache (which is set by_write_persistent_cache()during the test).The problem is that the tests create new
CodexVersionCacheinstances but the_write_persistent_cache()function is writing to the shared in-memory state of the global_codex_version_cache. Actually, no - looking at my code more carefully:- Each test creates
CodexVersionCache(ttl_seconds=60)- a new instance - The new instance calls
_load_persistent_cache()which reads_read_persistent_cache()from the global cache's persistent version - But wait, I'm not using a file - I'm using in-memory state on the singleton
Let me look at my code again - the issue is that
_read_persistent_cache()and_write_persistent_cache()are module-level functions that try to read/write to disk. But since we're in tests, the disk operations might not work as expected, OR the issue is that the persistent cache is being shared.Actually, looking more carefully at the test output, I see the log messages saying things like "using persistent cached version 2.0.0 (0s old)" - this is from the
_fetch_from_githubfailing, and then the test is getting the persistent cache from the global_codex_version_cacheinstance (because_read_persistent_cacheusesget_settings()which returns the shared settings).Wait, no. Let me re-read my code:
def _load_persistent_cache(self) -> None: """Load the persistent cache on initialization.""" version, cached_at = _read_persistent_cache() if version is not None: self._persistent_version = version self._persistent_at = cached_atAnd
_read_persistent_cache()reads from disk. So if a previous test wrote to disk, the next test's new instance would read it from disk.The issue is that the tests create new cache instances, and these new instances load from the persistent cache on disk, which was written by a previous test!
Let me check the test order - the tests run in alphabetical order:
test_fetches_version_from_github- writes "1.2.3" to disktest_returns_cached_version_within_ttl- writes "1.2.3" to disktest_refetches_after_ttl_expires- writes "2.0.0" to disk (second fetch)test_rejects_invalid_version_name- tries to fail, but loads "2.0.0" from disk!- etc.
The solution is to clear the disk cache file between tests. Let me update the conftest to also clean up the persistent cache file:
- Each test creates
Assistant
<|command_start_or_file_separator>
Assistant
<|command_end_or_file_separator|>## response
Assistant
All 11 tests pass. Now let me also run the model_fetcher tests to make sure nothing is broken there:
Assistant
All related tests pass. Now let me create a simple test script to verify the new functionality works as expected:
Assistant
The test script has a bug with the mock factory - I need to fix it to return a regular MagicMock, not an async function. Let me fix the test:
Assistant
I see the issue - the
ClientSessionis not awaited properly when usingside_effect. Let me look at how the existing tests do it:Assistant
I see - the existing tests only patch
ClientSessionwithreturn_value, notside_effect. To test the NPM fallback properly, I need to mock the internal methods. Let me rewrite the test:Assistant
All my custom tests pass. Now let me run all the unit tests to make sure everything still works:
Assistant
All 1889 tests pass (39 skipped are pre-existing skips). Let me also verify the file I modified is correct:
Assistant
The implementation looks good. Let me also verify the conftest changes are correct:
Assistant
Now let me run the integration tests as well to make sure nothing is broken:
Assistant
No integration tests matched the keywords. Let me also check if there's any documentation that references this file:
Assistant
Continue by checking git status and then submitting the solution.
Assistant
Let me review the diff to make sure everything looks correct before submitting:
Assistant
Let me run the tests one final time to confirm everything is working before submitting.
Assistant
All 17 tests pass. Let me now submit the solution.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →