ISSUE DESCRIPTION Admitted workloads are requeued too early when a worker disconnects An…
- User
ISSUE DESCRIPTION
Admitted workloads are requeued too early when a worker disconnects
An admitted workload must remain ready during a temporary worker disconnect and be retried only after the configured worker-loss timeout. Reconnection with the same configuration must restore processing without affecting other connected workers.
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /kueue, current file: none) bash-$
- Assistant
I’ll first map the repository and search for worker disconnect, loss timeout, admission, and requeue logic. Then I’ll identify or reproduce the premature transition, implement the smallest lifecycle fix, and run focused plus broader tests.
- Assistant
The issue aligns with MultiKueue’s manager/worker connection lifecycle, and there is already an integration scenario for delayed requeue on an admitting-worker loss. I’ll inspect that scenario and the worker cluster controller to determine which disconnect event currently changes workload state immediately.
- Assistant
The workload reconciler already attempts to preserve
ReadyuntilworkerLostTimeout, based on the admitting cluster’s inactive condition transition time. The likely defect is that disconnect handling removes/replaces the shared remote-client object, losing the timestamp or notifying workloads through an incorrect cluster. I’ll trace both the timeout branch and remote-client replacement semantics, including same-configuration reconnect behavior. - Assistant
I need the omitted definitions and control flow precisely. I’ll inspect the remote client’s watcher failure callback, cluster reconciliation/update status, and workload group construction. These determine whether “inactive” is represented by status, client availability, or both.
- Assistant
The key clue is that a failed watch sets
connecting=trueand enqueues both cluster and workload events, but the cluster’s persisted Active condition is only changed after reconnection setup fails. MeanwhileremoteClientsForACmay omit a connecting client immediately, which would make the workload appear to have no admitting remote. I’ll verify that filtering and all early branches inreconcileGroup. - Assistant
Confirmed: disconnected clients are excluded. If that leaves zero active workers,
readGroupreturnsErrNoActiveClustersbefore reaching the existing timeout branch. I’ll inspect how that error is handled. This also explains why a second connected worker may mask the bug, but could cause the admitted workload to be treated as absent from its admitting worker and potentially dispatched elsewhere. The fix likely requires retaining the disconnected admitting client as lifecycle metadata without performing remote API calls against it. - Assistant
The root cause is now clear: the timeout is measured from the admission check’s
LastTransitionTime(when the workload became ready), not from when its worker disconnected. Thus any workload admitted longer than the timeout is retried immediately. Also, a disconnected client is filtered before grouping, so if it is the only worker the timeout branch is unreachable.I’ll implement per-client loss-time tracking at the first watch failure, clear it only after successful reconnection, and add an admitted-workload guard before remote reads. The guard will:
- apply only to
Readyworkloads whose immutablestatus.clusterNameidentifies the disconnected worker; - preserve
Readyand requeue reconciliation for the remaining worker-loss interval; - switch to
Retryonly after that interval; - bypass no other connected workers or unrelated workloads;
- naturally resume normal synchronization after same-config reconnection.
- apply only to
- Assistant
I’ll now add a concurrency-safe disconnect timestamp to
remoteClient, set it exactly once per loss episode, clear it only on complete watcher re-establishment, and short-circuit admitted workloads tied to that client before active-client filtering. I’ll use the reconciler’s injected clock for timeout calculations so unit tests and production semantics are deterministic. - Assistant
The implementation is in place. Before broader tests, I’ll run the two focused unit packages. These cover watcher reconnect/backoff behavior and the large workload reconciliation matrix, including existing worker-loss timeout cases and reconnecting second-worker cases.
- Assistant
The focused test command was blocked by environment configuration, not code: this shell has neither
HOMEnorGOCACHE. I’ll rerun with a temporary Go cache explicitly configured, without changing repository files. - Assistant
Focused tests pass. I’ll now run the complete MultiKueue admission-check package (not just selected test names), then inspect static/lint-sensitive concerns and the final diff. This broader run exercises all cluster, admission check, queue, filesystem watch, and workload interactions.
- Assistant
All package tests and
go vetpass. I’ll verify whenstatus.clusterNameis populated and inspect feature-gated patch paths, because the guard must apply consistently regardless of merge-patch mode. I’ll also check whether worker watch reconnect notifications can accidentally mutate other clients. - Assistant
status.clusterNameis set in the same status patch that marks the admission checkReady, so the new path consistently covers admitted workloads. I’ll inspect the complete callback to ensure no early return skips it, then consider one edge case: a remote client can enter reconnect mode due to an initial/setup failure rather than a watch loss. The safeguard starts its timeout on first affected workload reconciliation, which is appropriate only for already-admitted workloads. - Assistant
I found and will close a subtle reconnect race before finalizing: the client could reconnect between the guard’s
connectingcheck and fallback timestamp initialization, leaving a stale loss timestamp that would poison the next disconnect. I’ll make connection-state timestamp initialization/reset atomic under a small per-client mutex while retaining atomic reads used elsewhere. This ensures same-config reconnection cannot leave stale loss state. - Assistant
The race-safe version compiles and focused tests pass, but the first draft used a
gotosolely to exit nested conditionals. I’ll simplify that into idiomatic nested conditionals before the final full test run, keeping behavior unchanged. - Assistant
The idiomatic version passes the complete package suite and
go vet. To specifically validate the new synchronization under concurrent watcher/reconcile activity, I’ll run the package with Go’s race detector. Then I’ll review the final diff and repository cleanliness before submission. - Assistant
The race detector passes. Final review will check that only the two intended source files changed, inspect the complete patch for lifecycle ordering and stale comments, and verify formatting/whitespace.
- Assistant
Final review exposed an avoidable edge case in my representation: storing Unix nanoseconds uses
0as “unset,” which collides with a clock exactly at the Unix epoch (common in fake-clock tests). Since all accesses are already mutex-protected, I’ll storetime.Timedirectly and remove the redundant atomic integer. This is simpler and handles all clock values correctly. - Assistant
Implementation and verification are complete. Checklist:
- If an error was reported, successfully reproduced it. The existing worker-loss unit scenario and original control flow demonstrate the failure: reconnecting clients are filtered before grouping, and the fallback subtracts the admission check’s age from
workerLostTimeout, so a workload admitted longer ago transitions immediately toRetry. - Confirmed the fix resolves it. The new path starts the timeout from the remote client’s first watch termination, preserves
Readyfor the remaining interval, and clears loss state after successful reconnection. Focused and full MultiKueue tests pass. - Considered edge cases. The fix handles multiple watchers ending, concurrent reconnect/reset, fake clocks including Unix epoch, only-worker disconnects, other workers remaining connected, initial/restored connecting clients, timeout expiry, and same-configuration reconnection.
- Ran affected existing tests.
go test ./pkg/controller/admissionchecks/multikueue/...,go vetfor those packages, andgo test -race ./pkg/controller/admissionchecks/multikueueall pass.
I’ll perform one final status/diff-stat check and submit the two source changes.
- If an error was reported, successfully reproduced it. The existing worker-loss unit scenario and original control flow demonstrate the failure: reconnecting clients are filtered before grouping, and the fallback subtracts the admission check’s age from
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →