ISSUE DESCRIPTION Admitted workloads are requeued too early when a worker disconnects An…
- User
ISSUE DESCRIPTION
Admitted workloads are requeued too early when a worker disconnects
An admitted workload must remain ready during a temporary worker disconnect and be retried only after the configured worker-loss timeout. Reconnection with the same configuration must restore processing without affecting other connected workers.
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /kueue, current file: none) bash-$
Assistant
Let me start by exploring the repository structure to understand what we're working with.
Assistant
Let me explore the repository structure.
Assistant
This is a Go project - Kueue, a Kubernetes queueing system. Let me explore the structure more.
Assistant
I need to make only one tool call at a time.
Assistant
Let me explore the main package structure.
Assistant
Let me search for worker-related code and "worker-loss" timeout.
Assistant
Let me run one command at a time.
Assistant
This looks like it's about MultiKueue. Let me look at the MultiKueue cluster controller since that deals with worker connections.
Assistant
Let me look at the workload.go file which handles workload admission in multikueue.
Assistant
Let me see the middle part of the file that was truncated, especially the
reconcileGroupfunction and any worker-loss logic.Assistant
Let me look at the
reconcileGroupandsyncToSingleCluster/nominateAndSynchronizeWorkersfunctions in detail.Assistant
Now let me understand the issue. The issue says "Admitted workloads are requeued too early when a worker disconnects". Let me look at the logic around the
workerLostTimeoutand the reserving remote lost handling.Looking at the code in
reconcileGroup:if remoteCond, reservingRemote := group.bestMatchByCondition(conditionToCheck); remoteCond != nil { // ... admitted path return reconcile.Result{RequeueAfter: w.workerLostTimeout}, nil } else if acs.State == kueue.CheckStateReady { // If there is no reserving and the AC is ready, the connection with the reserving remote might // be lost, keep the workload admitted for keepReadyTimeout and put it back in the queue after that. remainingWaitTime := w.workerLostTimeout - time.Since(acs.LastTransitionTime.Time) if remainingWaitTime > 0 { log.V(3).Info("Reserving remote lost, retry", "retryAfter", remainingWaitTime) return reconcile.Result{RequeueAfter: remainingWaitTime}, nil } else { return reconcile.Result{}, w.updateACS(ctx, group.local, acs, kueue.CheckStateRetry, "Reserving remote lost") } }So in the admitted path (when the remote workload is still admitted), we requeue after
w.workerLostTimeout. But what happens when the worker disconnects? The remote workload might temporarily not be found (because the connection is lost). SobestMatchByConditionreturns nil for theWorkloadAdmittedcondition, and we fall into theelse if acs.State == kueue.CheckStateReadybranch.Wait, but let me understand the flow. When a worker disconnects, the informer for that cluster stops. So the remote workload object in the cache becomes stale - it might still show the workload as admitted, or it might be removed from the cache.
Let me look at how
readGroupworks and how remote workloads are fetched.Assistant
Let me look at the remoteClient structure and how the connection state is tracked.
Assistant
Let me find the remoteClient struct definition.
Assistant
Let me look at the watch ending and reconnection logic.
Assistant
Let me look at the git log and recent changes to understand the context.
Assistant
Now I understand the issue. Let me look at the test file to understand the expected behavior.
Assistant
Let me look at the clustersReconciler and how clients are accessed.
Assistant
Let me look at the test setup to understand how the tests work.
Assistant
Now I need to understand the test cases for the "WorkerLostTimeout not exceeded" and "exceeded" tests. Let me look at what worker1Workloads look like in those tests. The
LastTransitionTimeis set to 50% or 150% of the timeout. Let me check if there are remote workloads in those test cases.Assistant
Let me look at the specific test cases for WorkerLostTimeout to see if they have worker1Workloads (remote workloads).
Assistant
Let me look at how the clock is set on remoteClient and the setConfig function.
Assistant
Let me look at integration tests for multikueue worker disconnect scenarios.
Assistant
Let me look at the
BreakConnectionutility and the other integration test about connection loss.Assistant
Let me look at the full clustersReconciler Reconcile and retry logic.
Assistant
Let me find all callers of
stopAndRemoveCluster.Assistant
Let me look at the
newClustersReconcilerandnewWlReconcilerfunctions.Assistant
Let me search for how the remoteClient clock is used in tests.
Assistant
Let me look at
SetAdmissionCheckStateand howLastTransitionTimeis managed.Assistant
Let me look at the other integration test about connection loss.
Assistant
Let me look at
getKubeConfigto understand the BreakConnection flow.Assistant
Let me check the multikueuecluster_test.go for tests related to loadClientConfig failure and stopAndRemoveCluster.
Assistant
Let me look at the "update client with invalid path config" test case. It has:
- A cluster with a path location type and empty path (invalid).
- An existing remoteClient.
- Expected: cluster status is BadKubeConfig,
wantCancelCalled: 1(the watchCancel is called). - Expected error.
The test checks
wantRemoteClients? Let me look at whatwantRemoteClientsis for this case. Actually, looking at the test, it doesn't havewantRemoteClientsfor the "update client with invalid path config" case. Let me look at how the test verifies the remoteClients.Assistant
Let me look at how the test verifies
wantRemoteClientsand the test runner.Assistant
Let me look at the test verification logic for
wantRemoteClients.Assistant
Let me find the test runner code that checks
wantRemoteClients.Assistant
Let me look at the
newTestClienthelper andsetReconnectState.Assistant
Let me check how
clustersReconcilerandwlReconcilerare created in production (controllers.go).Assistant
Now I have a thorough understanding of the issue. Let me implement the fix. First, let me open the multikueuecluster.go file to make changes to the
remoteClientstruct.Assistant
Now let me make the changes. First, let me add the
lastLostAtfield to theremoteClientstruct.Assistant
Now let me add the
lastLostAtfield to the struct:Assistant
Let me use the
editcommand instead.Assistant
Now let me update
newRemoteClientto setlastLostAt:Assistant
Now let me add the helper methods for
lastLostAt. Let me add them afterresetFailedConnAttempt:Assistant
Now let me update
setConfigsuccess path to callmarkConnected():Assistant
The edit went wrong - it broke the code structure. Let me fix it:
Assistant
Good, the
setConfigis fixed. Now let me update the watch-end goroutine to callmarkDisconnected:Assistant
There's an extra
}at line 427. Let me fix it:Assistant
Good. Now let me add the
disconnectedAtmap to theclustersReconcilerstruct. Let me find it.Assistant
There's a duplicate
remoteClientsline at 612. Let me fix that:Assistant
Good. Now let me update
stopAndRemoveClusterto record the disconnect time:Assistant
Now let me update
setRemoteClientConfigto cleardisconnectedAtafter success. Let me add a helper method and update the function:Assistant
Now let me add the
clearDisconnectedAtandclusterLostSincemethods aftercontrollerFor:Assistant
Now let me initialize the
disconnectedAtmap innewClustersReconciler:Assistant
There's an issue -
configNamespaceis missing andwlUpdateChis duplicated. Let me fix that:Assistant
Good,
newClustersReconcileris now correct. Now let me update the workload reconciler'selse if acs.State == kueue.CheckStateReadybranch. Let me open the workload.go file.Assistant
Now let me update the
else if acs.State == kueue.CheckStateReadybranch to use the disconnect time:Assistant
There's an extra
}at line 524. Let me fix it:Assistant
Let me verify
workload.ClusterNameexists and check the imports are fine. Let me build the project:Assistant
The build succeeds. Now let me run the existing unit tests to see if they pass:
Assistant
All existing unit tests pass. Now let me create a temporary test to verify the fix works for the disconnect scenario. Let me first look at the existing test structure for the "WorkerLostTimeout" cases to understand how to set up a connecting (disconnected) client.
Assistant
Let me check if the test workload builder has a
Clustermethod and understand the test setup for ClusterName.Assistant
Let me create a temporary test to verify the fix. First, let me understand the test helper for the
Reconcileresult check:Assistant
Let me check if the test captures the RequeueAfter from Reconcile:
Assistant
Let me create a temporary test to verify the fix works for the disconnect scenario:
Assistant
Let me check the correct import path for the fake clock used in the existing tests:
Assistant
OK, the import path for the testing util is
sigs.k8s.io/kueue/pkg/util/testing(aliased asutiltesting). And the API testing helpers areutiltestingapi(probablysigs.k8s.io/kueue/pkg/util/testing/v1beta2or similar). Let me check.Assistant
Let me fix the temporary test file with the correct imports:
Assistant
The test passes. The fix works:
- "disconnect recent - stay Ready": The workload was admitted an hour ago (
muchEarlier), but the worker disconnected recently (50% of timeout ago). The workload stays Ready with a requeue after 2m30s (remaining timeout). - "disconnect old - Retry": The worker disconnected 150% of the timeout ago. The workload is requeued (Retry).
Without the fix, both cases would have been immediately requeued (Retry) because
acs.LastTransitionTimewas an hour ago.Let me verify this by reverting the fix temporarily and running the test:
- "disconnect recent - stay Ready": The workload was admitted an hour ago (
Assistant
Let me verify the test fails without the fix by temporarily reverting the workload.go change:
Assistant
This confirms the bug exists without the fix. The "disconnect recent - stay Ready" case fails (gets "Retry" instead of "Ready") because the old code uses
acs.LastTransitionTime(an hour ago) instead of the actual disconnect time. Now let me restore the fix:Assistant
The fix works. Now let me also test the scenario where the remoteClient is removed (like BreakConnection) — the
disconnectedAtmap should be used. Let me update the temporary test:Assistant
All test cases pass. Now let me clean up the temporary test file and run the full existing test suite:
Assistant
Now let me run the full test to make sure I didn't break anything broader:
Assistant
Let me also run the broader controller tests to make sure nothing is broken:
Assistant
All tests pass. Now let me review the full diff to make sure everything is clean:
Assistant
Let me restore the blank line that was removed in workload.go:
Assistant
Now let me rebuild and run the tests once more to confirm everything is clean:
Assistant
Let me run
go vetto check for any issues:Assistant
Let me do a final review of the complete diff:
Assistant
Let me verify there are no leftover temporary files:
Assistant
The temporary files in /tmp are outside the repo. Let me clean them up:
Assistant
Now let me run the full multikueue unit test suite one final time to confirm everything passes:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →