ISSUE DESCRIPTION Admitted workloads are requeued too early when a worker disconnects An…
- User
ISSUE DESCRIPTION
Admitted workloads are requeued too early when a worker disconnects
An admitted workload must remain ready during a temporary worker disconnect and be retried only after the configured worker-loss timeout. Reconnection with the same configuration must restore processing without affecting other connected workers.
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /kueue, current file: none) bash-$
- Assistant
I'll start by exploring the repository structure and searching for code related to worker disconnects and workload requeuing.
- Assistant
Searching for worker-loss timeout and requeue logic related to MultiKueue.
- Assistant
rg is not available, I'll use grep instead.
- Assistant
Inspecting the workload reconciler and worker-disconnect handling next.
- Assistant
Reading the full reconcile path and related unit tests for the worker-lost behavior.
- Assistant
Let me look at how remote clients are obtained and what happens when a client is disconnected. Also look at git history for related issues, and the tests in more detail.
- Assistant
I'll inspect
syncReservingRemoteStateand the worker-lost unit tests to confirm how the timeout is measured. - Assistant
Examining connection handling and recent related changes.
- Assistant
There's a recent commit "MultiKueue: prevent a hung remote watch from stopping all-cluster admission (#11207)" which might be related. Also "Don't use reconnecting clients in WL reconnecting". Let me ...
- Assistant
Checking for uncommitted changes and reading the worker-lost paths and related tests.
- Assistant
Let me look at setup_test and jobs_test for the connection lost scenarios in integration tests.
- Assistant
Let me read those test cases carefully, especially "Should requeue the workload with a delay when the connection to the admitting worker is lost"
- Assistant
Checking how
ErrNoActiveClustersand failed SyncJob affect the worker-lost timeout path. - Assistant
I'll check
SetAdmissionCheckStatenext, and look for recent regressions related to the worker lost timeout. - Assistant
Found a likely bug with LastTransitionTime — verifying how FindAdmissionCheck works.
- Assistant
Checking how PatchAdmissionStatus applies mutations — acs may point at the wrong workload object.
- Assistant
Looking more carefully at PatchAdmissionStatus and how the callback works with the acs pointer:
- Assistant
Continuing to inspect the update callback path in
patchStatus. - Assistant
LastTransitionTime isn't refreshed once ACS is Ready — that matches early requeue. Checking when that behavior was introduced.
- Assistant
Checking how the original keep-ready heartbeat was implemented and how it evolved.
- Assistant
Tracing how the PatchAdmissionStatus API changed and whether that broke LTT refresh.
- Assistant
Checking unit tests for expected LTT heartbeat behavior and reviewing KeepAdmissionCheckPending.
- Assistant
Looking for reconnect issues that could affect other workers — review of setConfig and connecting.
- Assistant
Inspecting the reconnect backoff commit for issues around "same configuration" reconnection.
- Assistant
Checking locking around remote client reconnect and any related open issues in the KEPs/docs.
- Assistant
Checking when KeepAdmissionCheckPending was removed and whether it's still needed.
- Assistant
Checking BaseSSAWorkload and verifying feature gate defaults.
- Assistant
Identified the bug:
syncReservingRemoteStatenever persists LastTransitionTime refreshes while ACS stays Ready, so disconnects after the timeout yield immediate requeue. Fixing that path. - Assistant
Also notice that even if we set LTT correctly on the wl copy, SetAdmissionCheckState itself won't update LTT because state is same. So the fix must either:
- Assistant
Applying the fix so the ACS heartbeat updates
LastTransitionTimeon the patched workload object. - Assistant
The edit landed in the wrong file — reverting that and applying the fix to the MultiKueue workload controller.
- Assistant
Good, now the correct file is open. Let me apply thefil edit.
- Assistant
Cleaning up the comment and verifying the imports.
- Assistant
Writing a temporary test to verify the LastTransitionTime heartbeat is persisted.
- Assistant
I'll add a temporary test case that asserts LTT is refreshed, then run the unit tests.
- Assistant
I'll write a temporary Go test file that specifically checks the LTT heartbeat.
- Assistant
I'll create a temporary test file in the multikueue package to verify the LTT heartbeat.
- Assistant
I need to put the test in the package directory. Let me create tmp file there and remove it later. The instruction says I can create temporary scripts but should not modi...
- Assistant
Let me write the temporary test carefully, modeled after TestWlReconcile setup.
- Assistant
Checking which condition marks a remote as reserving.
- Assistant
There's a bug in remainingWaitTime calculation!
- Assistant
Also remainingWaitTime uses time.Since not clock - and after heartbeat acs is from freh GET. Good.
- Assistant
Need to set GOCACHE and maybe HOME.
- Assistant
Confirming the test fails without the fix:
- Assistant
Confirmed the regression; restoring the fix and running the full MultiKueue unit tests.
- Assistant
All multiKueue unit tests pass. Checking whether
remainingWaitTimeshould use the clock, and verifying the final diff. - Assistant
Fixing
remainingWaitTimeto use the injected clock for consistency. - Assistant
Re-running unit tests and checking for other similar ACS mutation bugs.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →