ISSUE DESCRIPTION auth: spurious REFRESHGENERATIONS bump on early-cancel in…
- User
ISSUE DESCRIPTION
auth: spurious _REFRESH_GENERATIONS bump on early-cancel in _coalesced_run_refresh_cmd
Summary
_coalesced_run_refresh_cmdinsrc/notebooklm/_auth/refresh.py(moved fromsrc/notebooklm/auth.pyin tier-10 PR-B-high #814, but the code is byte-identical to origin/main since tier-9 E #803) can bump_REFRESH_GENERATIONS[refresh_key]even when no real refresh happened, in a narrow cancel-before-await race.Reproducer (race condition)
- Caller invokes
await _coalesced_run_refresh_cmd(refresh_key, path, profile)from the cancel-recovery loop in_fetch_tokens_with_refresh. - Task is cancelled in the window between
try:and the firstwith _REFRESH_STATE_LOCK:block inside_coalesced_run_refresh_cmd— so NO future is registered in_REFRESH_INFLIGHT_BY_LOOP[loop][refresh_key]. - Control jumps to
except asyncio.CancelledError:in the caller. registry.get(refresh_key)returnsNone.- The inner
if inflight is None or inflight.done():is taken; theif inflight is not None:block is skipped;subprocess_excstaysNone;break. - Outer
if subprocess_exc is not None:is False, so no raise. - Generation is bumped at the
with _REFRESH_STATE_LOCK: ... max(existing, refresh_generation + 1)line, even though no subprocess ran. - Then
if caller_cancelled: raise asyncio.CancelledError().
Impact
A concurrent waiter on a different event loop sharing the same
refresh_keywould observe the phantom generation bump on its lock-re-acquire path, treat the storage as freshly-refreshed, and skip its own_run_refresh_cmd— proceeding tobuild_httpx_cookies_from_storageagainst STALE on-disk state.Bounded blast radius: the stale-storage caller will fail again on the next token fetch and trigger its own real refresh. No data corruption. But it's a wasted round-trip.
Suggested fix (from CodeRabbit on PR #814)
Track whether
inflightwas ever non-None during the cancel-recovery loop. Only bump generation ifsubprocess_exc is None AND observed_inflight is True. Or per the alternative suggestion: raise immediately wheninflight is Noneat the top of the cancel handler (the subprocess never registered, so we can't be the leader-cancel-before-await case for an actually-running refresh).Why not fixed in PR #814
That PR's scope was "extract verbatim from auth.py into _auth/refresh.py + facade-forward". Changing async semantics was explicitly out of scope (
must_not_do). Filing here for a dedicated follow-up.References
- CodeRabbit comment: https://github.com/teng-lin/notebooklm-py/pull/814#discussion_r3256142201
- Original code:
src/notebooklm/_auth/refresh.py:487-537(tier-10),src/notebooklm/auth.py:961-1026(tier-9 E origin)
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /notebooklm-py, current file: none) bash-$
- Caller invokes
Assistant
Let me start by exploring the repository structure to understand the codebase
Assistant
Let me see the actual
_coalesced_run_refresh_cmdfunction and the surrounding codeAssistant
Now let me look at the
_fetch_tokens_with_refreshfunction more carefully to understand the cancel-recovery loopAssistant
me explore the tests for refresh to understand how to verify the fix
Assistant
Let me check how auth.py imports things from _auth.refresh
Assistant
Let me check the imports and module-level state of refresh.py
Assistant
Now let me first create a reproducer script to confirm the bug
Assistant
let me write a reproducer script to confirm the bug exists
Assistant
Let me check the actual function being called in auth.py
Assistant
The auth module re-exports the symbols from
_auth_refresh. The issue is that when I doauth_mod._coalesced_run_refresh_cmd = fake_coalesced, I'm setting it on the auth module, but the actual function used inside_fetch_tokens_with_refresh(which lives in_auth_refresh) is the local reference.Let me check if there's a facade write-through mechanism:
Assistant
Let me update my reproducer to use the facade properly
Assistant
The bug is confirmed. Now let me apply the fix using the
observed_inflightflag approach as the primary suggestion from the issue.Assistant
Let me try using the replace command instead
Assistant
me verify the change was made correctly
Assistant
The fix is in place. Now let me run the reproducer to confirm it fixes the bug:
Assistant
The fix works. Now let me verify the existing tests pass:
Assistant
existing tests pass. Let me also run a broader test suite to ensure nothing is broken:
Assistant
The test failures are unrelated to our fix (they're about playwright/CLI session tests). Let me verify these failures are pre-existing by checking against the original code:
Assistant
found a bug in my fix. The success path (normal await) doesn't set
observed_inflight = True, so a normal successful refresh would also skip the generation bump. Let me fix this:Assistant
Let me verify the fix and then re-run the tests:
Assistant
let me write a comprehensive reproducer to test all the edge cases:
Assistant
Let me simplify the test by using the real
_coalesced_run_refresh_cmdwhere possible:Assistant
5 scenarios pass. Let me verify the test would have failed with the original code by temporarily reverting the fix:
Assistant
The original code fails scenario 5 with a phantom bump. The fix is correctly applied. Let me re-run the test suite to make sure nothing else is broken:
Assistant
338 tests pass. Let me do a final review of the change:
Assistant
The diff is clean and minimal. Let me run all the affected tests one more time to confirm everything works:
Assistant
All 64 relevant tests pass. Let me also verify the wider test suite (excluding pre-existing playwright failures):
Assistant
The 10 failures and 21 errors are all in tests/unit/cli/test_session.py (which I should not be ignoring — but it's a CLI test directory). Wait, I did
--ignore=tests/clibut there are still test_session.py failures. Let me check what's going on.Actually wait, I did
--ignore=tests/cli. But the errors are fromtests/unit/cli/test_session.py. The--ignore=tests/cliignores thetests/cli/directory, buttests/unit/cli/is a different directory.Let me check the structure:
Assistant
4211 tests pass (excluding pre-existing CLI/playwright failures). Let me clean up my reproducer files and verify the final state:
Assistant
Summary
The fix is complete. Here's a summary of the changes:
Issue
_coalesced_run_refresh_cmdcould bump_REFRESH_GENERATIONS[refresh_key]even when no subprocess ran, in the narrow cancel-before-register race described in the issue.Fix
Following the primary suggestion from the issue (the
observed_inflightflag approach), I added a boolean flag that tracks whether the in-flight future was ever observed during the cancel-recovery loop:- Set to
Trueafter normal completion (the subprocess definitely ran ifawait _coalesced_run_refresh_cmd(...)returns normally — the shielded future was resolved) - Set to
Trueinside theexcept asyncio.CancelledError:handler wheninflight is not None(the in-flight future was registered, so a subprocess ran or is running) - The generation bump is now wrapped in
if observed_inflight:— only happens when a subprocess was actually driven
Edge cases handled
- Normal success: gen=1 (bumped) ✓
- Refresh fails: gen=0 (no bump) ✓
- Cancel before register (the bug): gen=0 (no phantom bump) ✓
- Cancel mid-flight, subprocess succeeds: gen=1 (bumped for other callers) ✓
- Cancel mid-flight, subprocess fails: gen=0 (no bump) ✓
Checklist
- ✅ Reproduced the bug with a script that simulates the cancel-before-register race (gen went from 0 to 1)
- ✅ Confirmed the fix prevents the phantom bump (gen stays at 0)
- ✅ Tested all edge cases for the cancel-recovery loop
- ✅ All 4211 existing tests in the relevant test suite pass (excluding pre-existing CLI/playwright failures unrelated to this change)
- ✅ Confirmed the original code FAILS scenario 5 (phantom bump) and the fix PASSES
- Set to
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →