ISSUE DESCRIPTION Admitted workloads are requeued too early when a worker disconnects An…
- User
ISSUE DESCRIPTION
Admitted workloads are requeued too early when a worker disconnects
An admitted workload must remain ready during a temporary worker disconnect and be retried only after the configured worker-loss timeout. Reconnection with the same configuration must restore processing without affecting other connected workers.
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /kueue, current file: none) bash-$
Assistant
Let me start by understanding the repository structure and the issue. The issue is about "admitted workloads are requeued too early when a worker disconnects." This seems to be related to Kubernetes and Kueue (a Kubernetes-native job queueing system).
Let me explore the repository structure first.
Assistant
Let me explore the repository more to understand the relevant code. The issue mentions "admitted workloads", "requeued", "worker disconnect", and "worker-loss timeout". Let me search for these terms.
Assistant
Let me look at the core workload controller and scheduler code, which are likely most relevant.
Assistant
Let me look at the workload controller and the scheduler, which handle the requeue logic
Assistant
Let me see the full reconcile method and related logic.
Assistant
Let me look at the
reconcileOnEventfunction and the requeue/eviction logic:Assistant
Let me continue reading the reconcile function and the PodsReady timeout logic.
Assistant
Let me look at the
reconcileNotReadyTimeoutandadmittedNotReadyWorkloadfunctions more closelyAssistant
Let me look at the workload package and understand how PodsReady and the recovery timeout work.
Assistant
Let me search for
WorkloadWaitForRecoveryand related recovery logicAssistant
Let me look at the jobframework reconciler, which sets the PodsReady condition for recovery:
Assistant
Let me look at the workload controller's event handling, specifically
reconcileOnEvent:Assistant
Let me look at the Update and Create methods of the WorkloadReconciler which handle events:
Assistant
Let me continue reading the Update function:
Assistant
Let me look at the scheduler code to understand the requeue flow:
Assistant
Let me search for the requeue logic in the scheduler:
Assistant
Let me look at the
requeueAndUpdatefunction and theRequeueWorkloadin theAssistant
Let me look at the queue manager's RequeueWorkload:
Assistant
Let me see the RequeueWorkload function and PushOrUpdate:
Assistant
Let me look at
RequeueIfNotPresentin cluster_queue:Assistant
Let me look at the RequeueIfNotPresent function:
Assistant
Let me find the private
requeueIfNotPresentmethodAssistant
Let me see the full requeueIfNotPresent method:
Assistant
Let me look at the
IsAdmissiblefunction and related admission checksAssistant
Let me look at the IsAdmissible, IsActive, and IsEvicted functions
Assistant
Let me look at the scheduler's schedule function and how it processes workloads:
Assistant
Let me see the schedule function:
Assistant
Let me look at the
Headsfunction:Assistant
Let me see the Heads function and nearby related functions:
Assistant
Now let me think about the issue differently. Let me look at the
Assistant
Let me search for MultiKueue worker disconnect related code:
Assistant
Let me look at the MultiKueue workload controller
Assistant
Let me look at the reconcileGroup function and the worker lost timeout logic
Assistant
Let me look at the reconcileGroup function
Assistant
Let me look at the multikueuecluster.go to understand how worker disconnect is handled:
Assistant
Let me look at the watch-related code in multikueuecluster.go:
Assistant
Let me look at the queueWorkloadEvent and queueWatchEndedEvent functions
Assistant
Let me see those functions
Assistant
Let me look at the nominateAndSynchronizeWorkers function
Assistant
Let me look at the full nominateAndSynchronizeWorkers function
Assistant
Let me look at the
readGroupfunction to understand how remotes are populated:Assistant
Let me look at readGroup:
Assistant
Let me look at the SelectivelyCachingClient implementation to understand what happens when the client is disconnected
Assistant
Let me look at the remote_client.go file
Assistant
Let me look at recent git history for relevant changes
Assistant
Let me search for the issue more specifically - maybe there's a bug when the worker disconnects and the reconciler triggers the core workload controller to requeue
Assistant
Let me look at the NeedsRequeueAtUpdate function in workload/admissionchecks
Assistant
Let me see the rest of the reconcileCheckBasedEviction function
Assistant
Let me search for where connecting is set to false
Assistant
Let me see the context around line 294 where connecting.Store(false) is called:
Assistant
Let me read the full setConfig function:
Assistant
Let me search for tests related to worker disconnect or worker lost timeout
Assistant
Let me look at the integration tests for worker lost timeout
Assistant
Let me search for the worker lost test cases
Assistant
Let me look at the multikueue workload unit tests
Assistant
Let me look at the integration test for worker lost behavior in the multikueue tests:
Assistant
Let me look at the test for worker lost behavior
Assistant
Let me find and look at the BreakConnection function
Assistant
Let me look at the BreakConnection function in test/util/util.go
Assistant
Let me look at the issue description more carefully and search for the relevant code path.
The issue mentions "Admitted workloads are requeued too early when a worker disconnects." Let me look at the
admittedNotReadyWorkloadfunction and thereconcileNotReadyTimeoutfunction again:Assistant
Let me search for where CheckStatePending is set in the admission check flow
Assistant
Let me look at line 1002 of the workload controller for the syncAdmissionCheckConditions function
Assistant
Let me look at the
reconcileSyncAdmissionChecksfunction and how it interacts with the flowAssistant
Let me look at the clustersReconciler.Reconcile function
Assistant
Let me look at the clustersReconciler.Reconcile function
Assistant
Let me look at the setRemoteClientConfig function
Assistant
Let me look at setRemoteClientConfig function
Assistant
Let me look at the remoteClientsForAC function
Assistant
Let me check how ErrNoActiveClusters is handled in the wlReconciler's Reconcile
Assistant
Let me look at how readGroup error is handled
Assistant
Let me look at the ErrNoActiveClusters definition and how it's used
Assistant
Let me look at the admissioncheck.go file
Assistant
Let me look at the admissioncheck.go file
Assistant
Let me search for how
GetRemoteClustersis usedAssistant
Let me look at the incremental dispatcher
Assistant
Let me look at the git log for recent changes related to worker disconnect/requeue
Assistant
Let me look at the specific test case at line 2161 about keeping admission check reference when connection is lost
Assistant
Let me look at stopAndRemoveCluster function
Assistant
Let me look at stopAndRemoveCluster
Assistant
Let me look at the admissioncheck controller
Assistant
Let me look at the AdmissionCheckReconciler's Reconcile function
Assistant
Let me check the CreatePodsReadyCondition
Assistant
Let me look at CreatePodsReadyCondition
Assistant
Let me check how the PodsReady condition is applied in the reconciler
Assistant
Let me look at how the PodsReady condition is applied in the reconciler
Assistant
Let me check HasConditionWithTypeAndReason
Assistant
Let me see HasConditionWithTypeAndReason
Assistant
Let me try to run the specific test to see if there's a
Assistant
Let me try setting up the build cache
Assistant
Let me look at the test case for worker lost timeout
Assistant
Let me look at these test cases
Assistant
Let me search for any TODO or FIXME comments related to worker disconnect or worker loss
Assistant
Let me look at the pod termination controller
Assistant
Let me search for all references to ErrNoActiveClusters to see how it's handled
Assistant
Let me check if there are any uncommitted
Assistant
Let me take a step back and look at the big picture. Let me search for relevant code paths that handle the transition when all remote clusters are unavailable.
Assistant
Let me look at the WorkerLostTimeout documentation
Assistant
Let me check the git log for the reconcileGroup function
Assistant
Let me look at the commit that introduced the workerLostTimeout logic
Assistant
Let me look at commit a52766471 that introduced worker cluster unavailability management
Assistant
Let me look at the diff from that commit for the reconcileGroup changes
Assistant
Based on my analysis, I've identified the issue. The problem is in the
reconcileGroupfunction in the MultiKueue workload controller. When a worker is healthy, theifbranch updatesLastTransitionTimeto now and returnsRequeueAfter: workerLostTimeout. If the worker disconnects shortly before the next reconciliation,LastTransitionTimeisworkerLostTimeoutold, and theelse ifbranch computesremainingWaitTime ≈ 0, causing an immediate retry even though the worker just disconnected.The fix: when the
else ifbranch detects thatremainingWaitTime <= 0and this is the FIRST time the disconnect is detected (not a subsequent check), it should reset the timeout by updatingLastTransitionTimeto now and keeping the state asReady. Only on the SECOND detection should it retry.Let me look at the current code one more time and implement the fix
Assistant
Let me look at the
updateACSfunctionAssistant
Now I'll implement the fix. The issue is in the
reconcileGroupfunction where theelseclause immediately retries whenremainingWaitTime <= 0, without considering that the disconnect may have happened recently. I'll modify it to give the worker one fullworkerLostTimeoutgrace period from the first detection.Assistant
I need to use the open command properly.
Assistant
Let me verify that "Reserving remote lost" is only set in one
Assistant
Now let me implement the fix. I'll modify the
elseclause to give the worker a fullworkerLostTimeoutgrace period from the first detection.Assistant
The edit command failed. Let me try using replace instead, or use a different format for edit. Let me first see the exact content of lines 504-514 to match them precisely:
Assistant
Let me use the replace command to make the change:
Assistant
Let me use the edit command with proper syntax, referencing the file with --file option
Assistant
Let me use sed directly to make the change:
Assistant
Let me verify the change
Assistant
I see there's an extra
}from the originalelseblock. Let me fix thisAssistant
I need to remove the extra
}on line 521Assistant
Let me verify the
Assistant
Let me check more context around the
Assistant
Now let me run the relevant unit tests to verify the change:
Assistant
Let me look at the failing test cases to understand what needs to be
Assistant
Let me check the diff of my changes
Assistant
I need to update the failing test case. The test "the_local_workload's_admission_check_is_set_to_Retry_if_the_WorkerLostTimeout_is_exceeded" now needs to expect the new behavior where the first detection gives a fresh timeout (state stays Ready), and only the second detection retries.
Let me look at the test cases:
Assistant
Let me look at the other test case that uses "Reserving remote lost" message:Let me look at the other test case that uses "Reserving remote lost" message:
Assistant
Let me find the full test definition for "elastic_job_local_workload_out-of
Assistant
Let me search for the test name
Assistant
Let me search for just "elastic" in the test file
Assistant
Let me look at the test at line 1072:
Assistant
The message-based tracking approach is fragile and breaks existing tests. Let me revert my change and think about a better approach.
Assistant
Let me take a different approach. The root cause is that the reconciliation interval in the
ifbranch equalsworkerLostTimeout, which meansLastTransitionTimecan be up toworkerLostTimeoutold when a disconnect is detected. The fix should be to reduce this reconciliation interval so thatLastTransitionTimeis always recent.Actually, let me re-read the issue description one more time:
"Admitted workloads are requeued too early when a worker disconnects. An admitted workload must remain ready during a temporary worker disconnect and be retried only after the configured worker-loss timeout. Reconnection with the same configuration must restore processing without affecting other connected workers."
Let me look at the code from the perspective of what happens when only one specific worker disconnects (not all workers). The key phrase is "without affecting other connected workers."
In the current code, when a worker is reconnecting (
connecting = true),remoteClientsForACskips it. If there are other workers, those are included. Theelse ifbranch is entered because the workload was on the disconnected worker.The
remainingWaitTimeis computed fromLastTransitionTimewhich was set when the workload was last confirmed on the disconnected worker. If this was up toworkerLostTimeoutago, the timeout might have expired, causing an immediate retry.But the disconnect might be very recent! The workload should have the full
workerLostTimeoutfrom the disconnect time.Now, here's a cleaner fix: in the
else ifbranch, whenremainingWaitTime <= 0, check if theLastTransitionTimewas updated by a healthy check (theifbranch) or by a previous disconnect detection. If it's from a healthy check, this is the first detection - reset the timeout. If it's from a previous disconnect, retry.We can distinguish by checking if the
LastTransitionTimeis significantly older thanworkerLostTimeout. If it is, it means the healthy check was long ago, and the disconnect might be recent. But we can also check iftime.Since(LastTransitionTime) >= 2 * workerLostTimeout. If so, the timeout has genuinely expired and we should retry.Wait, but what if the healthy check was just done (LastTransitionTime is recent) and the disconnect happens right after? Then
remainingWaitTime > 0, and we wait. That's correct.What if the healthy check was
workerLostTimeoutago, and the disconnect happens right now? ThenremainingWaitTime = 0, and we should give a fresh timeout. That's the fix.What if the disconnect was detected once, we gave a fresh timeout, and now we're checking again? Then
remainingWaitTime = 0again, and this time we should retry.So the algorithm is:
- Compute
remainingWaitTime = workerLostTimeout - time.Since(LastTransitionTime) - If
remainingWaitTime > 0: wait - If
remainingWaitTime <= 0:- If
time.Since(LastTransitionTime) < 2 * workerLostTimeout: this is the first detection, reset timeout - Else: this is the second detection, retry
- If
But this has a problem: what if the
ifbranch was updatingLastTransitionTimeeveryworkerLostTimeoutfor a long time? Thentime.Since(LastTransitionTime)is always aboutworkerLostTimeout, which is< 2 * workerLostTimeout. So we'd keep resetting the timeout forever.The issue is that we can't distinguish between:
- First detection after a disconnect (should reset timeout)
- Healthy check from
workerLostTimeoutago + recent disconnect (should reset timeout) - Multiple cycles of reset (should eventually retry)
Using
time.Since(LastTransitionTime) < 2 * workerLostTimeoutdoesn't help because theifbranch updatesLastTransitionTimeeveryworkerLostTimeout.What if we track the number of times the
else ifbranch has been entered consecutively? We could use a field in the admission check state. But "No new interfaces are introduced."What if we use the
RetryCountfield? Currently,RetryCountis only incremented when the state changes toCheckStateRetry. But we could increment it on each reset and cap it at 1 for the reset behavior.Wait, that's getting complicated. Let me think about the simplest fix.
Actually, I think the simplest and most correct fix is: in the
ifbranch, changeRequeueAfter: w.workerLostTimeoutto a much shorter interval. This ensuresLastTransitionTimeis always recent. When the disconnect happens, theelse ifbranch has an accurateremainingWaitTime.But I need to choose a reasonable interval. 1 second is too frequent. What about not returning
RequeueAfterat all (i.e.,RequeueAfter: 0)?If
RequeueAfteris 0, the reconciler won't be requeued. It will only run when events trigger it. For healthy workloads, the next event would be:- A watch event from the remote (workload status change)
- A local workload change
- A MultiKueueConfig change
If no events happen, the reconciler won't run. This is fine - if the workload is healthy and nothing changes, there's nothing to reconcile.
When the worker disconnects, the watch ends. This triggers
queueWatchEndedEvent→clustersReconciler→ setsconnecting = true. But thewlReconcileris NOT triggered. ThewlReconcilerwould only run when the next workload update happens.But what if the next workload update doesn't happen for a long time? The
wlReconcilerwouldn't detect the disconnect. This is a problem.However, the
wlUpdateChmight still receive events during the watch restart process. But if the watch ended, no more events come.So removing the
RequeueAfterentirely would mean the disconnect might never be detected. That's not acceptable.What if we change
RequeueAfterto something like30 * time.Second? Then:LastTransitionTimeis at most 30 seconds oldremainingWaitTimein theelse ifbranch is at leastworkerLostTimeout - 30s ≈ 14.5 minutes(for a 15-minute timeout)- The error is at most 30 seconds
This is a good compromise: the reconciliation is frequent enough to keep
LastTransitionTimerecent, but not so frequent as to cause performance issues.Actually, wait. Let me re-read the code. The
ifbranch returnsRequeueAfter: w.workerLostTimeout. But before that, it callssyncReservingRemoteState, which updatesLastTransitionTimetonowand the message to "The workload got reservation on X."If I change
RequeueAfterto a shorter interval, theLastTransitionTimewould be updated more frequently. ButsyncReservingRemoteStatealso patches the workload status. Patching every 30 seconds for every admitted MultiKueue workload would be too frequent.Hmm, what if I don't change
RequeueAfterat all, but instead change theelse ifbranch logic?Let me reconsider the approach. The issue is:
LastTransitionTimeis set by theifbranch (healthy check)- The
else ifbranch uses it to computeremainingWaitTime - If
LastTransitionTimeis old (up toworkerLostTimeout),remainingWaitTimecan be 0, causing immediate retry
What if the
else ifbranch doesn't useLastTransitionTimebut instead uses a different reference point? For example, what if theelse ifbranch, when entered for the first time, updatesLastTransitionTimetonowand then waitsworkerLostTimeout? The second time it's entered (after the wait), it retries.This is what I attempted with the message-based tracking. But it broke the out-of-sync test.
The out-of-sync test enters the
else ifbranch NOT because the worker is lost, but because the remote workload was deleted (out of sync). In this case, we should retry immediately, not wait.So the fix should distinguish between:
- Worker lost (remote client not available/reconnecting) → wait
workerLostTimeoutfrom first detection - Remote workload deleted (out of sync, evicted, etc.) → retry immediately
How to distinguish? In case 1, the remote client was skipped by
remoteClientsForAC(becauseconnecting = true). In case 2, the remote client was available, and the remote workload was fetched and then deleted in step 5.But by the time we reach step 6, we don't know which case we're in.
We could track this in the
wlGroupstruct. For example, we could add a flagworkerLostthat is set when the remote client was skipped.But the issue says "No new interfaces are introduced." Adding a field to
wlGroupis an internal change, not a new interface. So it should be OK.Alternatively, we could check if the
bestMatchByConditionreturned nil because the remote workload was deleted vs. because the client was skipped. But in both cases, the remote workload is nil ingroup.remotes.Actually, wait. In the out-of-sync test case, the remote workload is fetched and then deleted in step 5 (
group.remotes[rem] = nil). In the worker-lost case, the remote client is skipped, so the remote workload is never fetched andgroup.remotes[rem]is nil from the beginning.But
readGroupinitializes all remotes by fetching them:for remote, rClient := range rClients { wl := &kueue.Workload{} err := rClient.client.Get(ctx, client.ObjectKeyFromObject(local), wl) if client.IgnoreNotFound(err) != nil { return nil, err } if err != nil { wl = nil } grp.remotes[remote] = wl }If the client is skipped (because
connecting = true), it's not inrClients, so the remote is not ingroup.remotesat all.If the client is available, the remote is fetched (or set to nil if not found).
In step 5, when a remote workload is out of sync:
for rem, remWl := range group.remotes { if remWl != nil && isRemoteSpecOutOfSync(group.local.Spec, remWl.Spec) { ... if err := client.IgnoreNotFound(group.RemoveRemoteObjects(ctx, rem)); err != nil { ... } group.remotes[rem] = nil } }So after step 5, the remote is set to nil.
In step 6:
if remoteCond, reservingRemote := group.bestMatchByCondition(conditionToCheck); remoteCond != nil { // found } else if acs.State == kueue.CheckStateReady { // not found }bestMatchByConditioniterates overgroup.remotes. If the remote was deleted in step 5, it's nil. If the client was skipped, the remote key doesn't exist in the map.We can distinguish by checking if the remote key exists in the map but is nil (deleted in step 5) vs. doesn't exist at all (client skipped).
But this distinction is not perfect. In the worker-lost case, if the client was removed entirely (
stopAndRemoveCluster), the key doesn't exist either.Hmm, this is getting complicated. Let me think about the simplest fix that addresses the issue without breaking tests.
I think the simplest fix is: in the
else ifbranch, whenremainingWaitTime <= 0, check if the first healthyifbranch'sLastTransitionTimeupdate was recent. Iftime.Since(LastTransitionTime) < workerLostTimeout + someMargin, it means the disconnect might be recent, so reset the timeout.But that's exactly what
remainingWaitTime <= 0checks! IfremainingWaitTime <= 0, thentime.Since(LastTransitionTime) >= workerLostTimeout. So it's NOT recent.So we can't distinguish based on
LastTransitionTimealone.OK, let me try yet another approach. What if I change the
ifbranch to returnRequeueAfter: 0(no requeue) and instead rely on the watch events to trigger reconciliation?When the worker is healthy:
- Watch events trigger
wlReconcilerviawlUpdateCh - If no events, no reconciliation needed
When the worker disconnects:
- Watch ends → nothing triggers
wlReconcilerdirectly - But the
ifbranch no longer returnsRequeueAfter, so there's no pending reconciliation
This means the disconnect is NEVER detected by the
wlReconciler! The workload staysReadyforever.That's worse.
Unless we add a mechanism to trigger the
wlReconcilerwhen the watch ends. But that's complex.What if I combine: change
RequeueAfterto a small value like5 * time.SecondAND ensure thatsyncReservingRemoteStateonly patches the status when something actually changes?Looking at
syncReservingRemoteState:func (w *wlReconciler) syncReservingRemoteState(ctx context.Context, group *wlGroup, reservingRemote string, acs *kueue.AdmissionCheckState) error { ... needsACUpdate := acs.State != kueue.CheckStateRetry && acs.State != kueue.CheckStateRejected if !needsTopologyUpdate && !needsACUpdate { return nil } if err := workload.PatchAdmissionStatus(ctx, w.client, group.local, w.clock, func(wl *kueue.Workload) (bool, error) { ... if needsACUpdate { acs.State = kueue.CheckStateReady acs.Message = fmt.Sprintf("The workload got reservation on %q", reservingRemote) acs.LastTransitionTime = metav1.NewTime(w.clock.Now()) ... } return true, nil }); err != nil {If
acs.Stateis alreadyReadyand the message hasn't changed,needsACUpdateis false. So it doesn't patch. ButLastTransitionTimeis ALSO not updated!Wait,
needsACUpdatechecks:needsACUpdate := acs.State != kueue.CheckStateRetry && acs.State != kueue.CheckStateRejectedIf
acs.StateisReady,needsACUpdateis true (becauseReady != Retry && Ready != Rejected). So it always patches when the state isReady.This means every reconciliation updates
LastTransitionTimeeven if nothing changed. If we changeRequeueAfterto 5 seconds, it would patch the status every 5 seconds. That's too frequent.What if we modify
syncReservingRemoteStateto only updateLastTransitionTimeif it's been a while? Or what if we add a condition toneedsACUpdateto check if the message is already correct?Actually, looking at the code, the
ifbranch always callssyncReservingRemoteState, which always patches ifacs.State == Ready. The patch updatesLastTransitionTimetonow. Then it returnsRequeueAfter: workerLostTimeout.I think the cleanest fix is to change
syncReservingRemoteStateto NOT updateLastTransitionTimeif nothing changed. Then we can reduceRequeueAfterto a small value without causing excessive patches.But that's a bigger change. Let me think about the minimal fix.
Actually, the minimal fix might be: in the
else ifbranch, whenremainingWaitTime <= 0, just returnRequeueAfter: 1 * time.Secondinstead of immediately retrying. This gives a small grace period. The next reconciliation will check again.But if nothing changed,
remainingWaitTimewill still be<= 0(becauseLastTransitionTimedidn't change), and we'd get into a tight loop.Unless the
else ifbranch updatesLastTransitionTimewithout patching... but we need to callupdateACSto update it, which patches the status.OK, I think the right approach is to update
LastTransitionTimein theelse ifbranch (first detection) and not in theifbranch (or only when something actually changes). This way:- Healthy
ifbranch: don't updateLastTransitionTime(or update only when state changes) - First
else ifdetection: updateLastTransitionTimeto now → remaining time = full timeout - Second
else ifdetection:remainingWaitTime = 0→ retry
But we still need to distinguish first vs second detection.
Hmm, what if we simply change the logic: in the
else ifbranch, ALWAYS updateLastTransitionTimetonowand returnRequeueAfter: workerLostTimeout. This means: every time we detect the worker is lost, we give it a freshworkerLostTimeoutfrom now.This would loop forever! Unless... the
ifbranch would break the loop when the worker reconnects.If the worker never reconnects:
- First
else if:LastTransitionTime = t0, waitWT - Second
else if:LastTransitionTime = t0 + WT, waitWT - Third
else if:LastTransitionTime = t0 + 2*WT, waitWT... infinite loop
That's not acceptable. We need to eventually retry.
OK, what if we add a check: if
time.Since(acs.LastTransitionTime) >= 2 * workerLostTimeout, then retry?Let me trace:
- Healthy
ifbranch att0:LastTransitionTime = t0(current behavior) - Worker disconnects shortly after
t0 - First
else ifatt0 + WT:time.Since(LastTransitionTime) = WT < 2*WT→ updateLastTransitionTime = t0 + WT, waitWT - Second
else ifatt0 + 2*WT:time.Since(LastTransitionTime) = WT < 2*WT→ update AGAIN!
Still loops!
I need to track that we've already reset once. Let me use a separate approach.
What if instead of using
LastTransitionTimefor the timeout, I add a new timestamp in the code that tracks when the disconnect was first detected? This doesn't require API changes - it could be a field in thewlGroupstruct or even just the message.Actually, let me go back to the message-based approach but handle the out-of-sync case properly.
In the out-of-sync test case:
- The remote workload exists (fetched successfully)
- Step 5 deletes it because it's out of sync
- Step 6:
bestMatchByConditionreturns nil acs.State == CheckStateReady→ enterselse ifremainingWaitTimeis computed fromLastTransitionTime(not set in test, so zero value)- With my fix: if
acs.Message != "Reserving remote lost", reset timeout
But in this case, the remote was accessible and the workload was just out of sync. We should retry immediately, not wait.
The distinction between "worker lost" and "remote deleted" can be made by checking if the remote was EVER in the group. If the remote key exists in
group.remotesbut is nil (meaning it was deleted), the worker is accessible. If the remote key doesn't exist (meaning the client was skipped), the worker is lost.So I can check: if
group.remoteshas entries (even if nil), the worker was accessible. But this is a heuristic - the client could have been removed entirely.Actually, in the
else ifbranch, we have access togroupwhich hasgroup.remotes. If the remote for the previously-admitted worker is present but nil, it means the worker was accessible but the workload was deleted. If the remote key doesn't exist at all, it might mean the worker is lost.But this is still fragile. Let me think about this more carefully.
Wait, I just realized something. The issue description says "No new interfaces are introduced." This means I can modify existing structs and functions, but I shouldn't add new API types or public methods.
What if I modify the
wlGroupstruct to add a field that tracks whether the reserving remote was lost (client not available) vs. the remote workload was deleted? This is an internal change.type wlGroup struct { local *kueue.Workload localClient client.Client remotes map[string]*kueue.Workload remoteClients map[string]*remoteClient acName kueue.AdmissionCheckReference jobAdapter jobframework.MultiKueueAdapter controllerKey types.NamespacedName reservingRemoteLost bool // NEW: true if reserving remote client was skipped/lost }Then in
remoteClientsForAC, when a client is skipped because it's reconnecting, we set this flag. InreconcileGroup, we use this flag to decide whether to reset the timeout or retry immediately.But
remoteClientsForAConly returns the clients, it doesn't know which one was the reserving remote.OK, I think the cleanest approach is to modify the
reconcileGroupfunction to check if any remote client was unavailable (due to reconnecting). We can do this by comparing the set of configured clusters with the set of available remote clients.Let me look at
remoteClientsForACagain:func (w *wlReconciler) remoteClientsForAC(ctx context.Context, acName kueue.AdmissionCheckReference) (map[string]*remoteClient, error) { cfg, err := w.helper.ConfigForAdmissionCheck(ctx, acName) if err != nil { return nil, err } clients := make(map[string]*remoteClient, len(cfg.Spec.Clusters)) for _, clusterName := range cfg.Spec.Clusters { if client, found := w.clusters.controllerFor(clusterName); found { if !client.connecting.Load() { clients[clusterName] = client } } } if len(clients) == 0 { return nil, admissioncheck.ErrNoActiveClusters } return clients, nil }This returns only non-reconnecting clients. The caller doesn't know which clusters were skipped.
I could modify this function to also return the set of reconnecting clusters. But that changes the interface.
Alternatively, I could add a check in
reconcileGroup: compare the set of configured clusters withgroup.remoteClients. If all configured clusters are ingroup.remoteClients, no cluster was lost. If some are missing, they were lost.But
group.remoteClientsis already the filtered set (only non-reconnecting clients). So if a cluster was lost, it wouldn't be ingroup.remoteClients.Wait,
group.remoteClientsis set to the return value ofremoteClientsForAC, which already filters out reconnecting clients. Sogroup.remoteClientsonly contains available clients.If the reserving remote was lost (reconnecting), it's not in
group.remoteClients, and hence not ingroup.remotes. SobestMatchByConditionreturns nil. We enter theelse ifbranch.But we also need to know WHICH remote was the reserving one to check if it was lost. We can check
group.local.Status.ClusterNameto know which cluster was the reserving one.Then in the
else ifbranch:} else if acs.State == kueue.CheckStateReady { remainingWaitTime := w.workerLostTimeout - time.Since(acs.LastTransitionTime.Time) if remainingWaitTime > 0 { ... } // Check if the reserving remote is among the available clients. reservingRemote := ptr.Deref(group.local.Status.ClusterName, "") if reservingRemote != "" { if _, found := group.remoteClients[reservingRemote]; found { // The reserving remote is available but workload not found. // This could be due to deletion or eviction. Retry immediately. return reconcile.Result{}, w.updateACS(ctx, group.local, acs, kueue.CheckStateRetry, "Reserving remote lost") } } // The reserving remote is not available (lost). // If this is the first detection, start a fresh timeout. if acs.Message != "Reserving remote lost" { log.V(3).Info("Reserving remote lost, starting worker lost timeout") return reconcile.Result{}, w.updateACS(ctx, group.local, acs, kueue.CheckStateReady, "Reserving remote lost") } return reconcile.Result{}, w.updateACS(ctx, group.local, acs, kueue.CheckStateRetry, "Reserving remote lost") }This distinguishes:
- Reserving remote is available but workload not found (out-of-sync, etc.) → retry immediately
- Reserving remote is NOT available (lost) → give fresh timeout (first detection) or retry (second detection)
This should fix the issue without breaking the out-of-sync test.
But wait, there's a subtlety: what if
ClusterNameis not set? In MultiKueue,ClusterNameis set insyncReservingRemoteState:if wl.Status.ClusterName == nil { wl.Status.ClusterName = &reservingRemote }So when the workload is first admitted on a remote,
ClusterNameis set. But what if the workload was admitted before this feature was added? ThenClusterNamemight be nil.If
ClusterNameis nil, we can't determine which remote was the reserving one. In that case, we should fall back to the current behavior (retry immediately) to be safe.Let me also check: in the out-of-sync test case, what is
ClusterNameset to?Looking at the test:
managersWorkloads: []kueue.Workload{ *baseWorkloadBuilder.Clone(). ... ClusterName("worker1"). Obj(), },It's set to "worker1". And
worker1Workloadsexists (the remote workload is on worker1). Sogroup.remoteClientsincludes worker1 (it's not reconnecting). SoreservingRemoteis "worker1", and it IS found ingroup.remoteClients. So the code retries immediately. Good!For the "WorkerLostTimeout is exceeded" test, the
ClusterNameis NOT set (the builder doesn't callClusterName). SoreservingRemoteis "". The code falls to the message-based tracking. But since there are no remote workloads at all, the message tracking resets the timeout once and then retries.Wait, but the test expects immediate retry. Let me trace through with my new logic:
Test: "WorkerLostTimeout is exceeded"
ClusterNameis not set (default nil)reservingRemote = ""- Falls to message tracking
acs.Message != "Reserving remote lost"→ reset timeout → state staysReady
But the test expects
Retry. So this still breaks the test.Hmm. But in this test, the
LastTransitionTimeis set to 150% of the timeout ago. This simulates a case where the last healthy check was very long ago. In my new logic, ifClusterNameis not set, we fall back to the message-based tracking, which resets the timeout once. The test expects immediate retry.But is the test's expectation correct? If
LastTransitionTimeis 150% of the timeout ago, the worker has been lost for at least that long. The timeout should have expired. But the disconnect might be more recent thanLastTransitionTime(e.g., the worker was healthy att0, then disconnected att0 + 1.5*WT, and the check is att0 + 1.5*WT). In that case, the timeout should be fresh.The issue is we don't know when the disconnect happened. The test assumes it happened at
LastTransitionTime(150% ago). My fix assumes it happened just now.I think the test needs to be updated to reflect the new behavior. But the "out-of-sync" test should NOT be affected because the reserving remote is available.
Let me implement this approach.Let me implement a more targeted fix that distinguishes between a lost worker and a deleted/out-of-sync remote workload.
- Compute
Assistant
Now let me implement the proper fix. The key insight is: we need to distinguish between:
- The reserving remote worker is lost (client not available) → should give fresh workerLostTimeout
- The remote workload was deleted/out-of-sync but the worker is available → should retry immediately
We can check
group.local.Status.ClusterNameto identify the reserving remote, and then check if that remote's client is ingroup.remoteClients.Let me implement this:
Assistant
Let me apply the changes manually using sed. This is a bit complex, so let me write a script to make the precise edit:
Assistant
Let me verify the change
Assistant
Let me check if
ptris already imported in this fileAssistant
Good,
ptris already imported. Now let me run the tests:Assistant
The "out-of-sync" test now passes. I need to update the "WorkerLostTimeout is exceeded" test to reflect the new behavior. In the new logic, when
ClusterNameis not set (reserving remote not identifiable), the first detection resets the timeout instead of retrying immediately.Assistant
Let me run the tests again
Assistant
All tests pass now. Let me run the other test suites to verify nothing else is broken:
Assistant
Let me run all tests in the multikueue package
Assistant
All tests pass. Let me verify the final state of the changes
Assistant
Let me also run the core workload controller tests to make sure nothing is broken:
Assistant
Let me run a broader set of tests to ensure nothing is broken:
Assistant
The fix looks good. Let me review the final diff one more time and then submit:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →