Skip to content
TheChatArchiveThe Chat Archive
DeepSeekdeepseek/deepseek-v4-pro· deepseekHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Admitted workloads are requeued too early when a worker disconnects An…

  1. User

    ISSUE DESCRIPTION

    Admitted workloads are requeued too early when a worker disconnects

    An admitted workload must remain ready during a temporary worker disconnect and be retried only after the configured worker-loss timeout. Reconnection with the same configuration must restore processing without affecting other connected workers.

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /kueue, current file: none) bash-$

  2. DeepSeek Assistant

    Let me start by understanding the repository structure and the issue. The issue is about "admitted workloads are requeued too early when a worker disconnects." This seems to be related to Kubernetes and Kueue (a Kubernetes-native job queueing system).

    Let me explore the repository structure first.

  3. DeepSeek Assistant

    Let me explore the repository more to understand the relevant code. The issue mentions "admitted workloads", "requeued", "worker disconnect", and "worker-loss timeout". Let me search for these terms.

  4. DeepSeek Assistant

    Let me look at the core workload controller and scheduler code, which are likely most relevant.

  5. DeepSeek Assistant

    Let me look at the workload controller and the scheduler, which handle the requeue logic

  6. DeepSeek Assistant

    Let me see the full reconcile method and related logic.

  7. DeepSeek Assistant

    Let me look at the reconcileOnEvent function and the requeue/eviction logic:

  8. DeepSeek Assistant

    Let me continue reading the reconcile function and the PodsReady timeout logic.

  9. DeepSeek Assistant

    Let me look at the reconcileNotReadyTimeout and admittedNotReadyWorkload functions more closely

  10. DeepSeek Assistant

    Let me look at the workload package and understand how PodsReady and the recovery timeout work.

  11. DeepSeek Assistant

    Let me search for WorkloadWaitForRecovery and related recovery logic

  12. DeepSeek Assistant

    Let me look at the jobframework reconciler, which sets the PodsReady condition for recovery:

  13. DeepSeek Assistant

    Let me look at the workload controller's event handling, specifically reconcileOnEvent:

  14. DeepSeek Assistant

    Let me look at the Update and Create methods of the WorkloadReconciler which handle events:

  15. DeepSeek Assistant

    Let me continue reading the Update function:

  16. DeepSeek Assistant

    Let me look at the scheduler code to understand the requeue flow:

  17. DeepSeek Assistant

    Let me search for the requeue logic in the scheduler:

  18. DeepSeek Assistant

    Let me look at the requeueAndUpdate function and the RequeueWorkload in the

  19. DeepSeek Assistant

    Let me look at the queue manager's RequeueWorkload:

  20. DeepSeek Assistant

    Let me see the RequeueWorkload function and PushOrUpdate:

  21. DeepSeek Assistant

    Let me look at RequeueIfNotPresent in cluster_queue:

  22. DeepSeek Assistant

    Let me look at the RequeueIfNotPresent function:

  23. DeepSeek Assistant

    Let me find the private requeueIfNotPresent method

  24. DeepSeek Assistant

    Let me see the full requeueIfNotPresent method:

  25. DeepSeek Assistant

    Let me look at the IsAdmissible function and related admission checks

  26. DeepSeek Assistant

    Let me look at the IsAdmissible, IsActive, and IsEvicted functions

  27. DeepSeek Assistant

    Let me look at the scheduler's schedule function and how it processes workloads:

  28. DeepSeek Assistant

    Let me see the schedule function:

  29. DeepSeek Assistant

    Let me look at the Heads function:

  30. DeepSeek Assistant

    Let me see the Heads function and nearby related functions:

  31. DeepSeek Assistant

    Now let me think about the issue differently. Let me look at the

  32. DeepSeek Assistant

    Let me search for MultiKueue worker disconnect related code:

  33. DeepSeek Assistant

    Let me look at the MultiKueue workload controller

  34. DeepSeek Assistant

    Let me look at the reconcileGroup function and the worker lost timeout logic

  35. DeepSeek Assistant

    Let me look at the reconcileGroup function

  36. DeepSeek Assistant

    Let me look at the multikueuecluster.go to understand how worker disconnect is handled:

  37. DeepSeek Assistant

    Let me look at the watch-related code in multikueuecluster.go:

  38. DeepSeek Assistant

    Let me look at the queueWorkloadEvent and queueWatchEndedEvent functions

  39. DeepSeek Assistant

    Let me see those functions

  40. DeepSeek Assistant

    Let me look at the nominateAndSynchronizeWorkers function

  41. DeepSeek Assistant

    Let me look at the full nominateAndSynchronizeWorkers function

  42. DeepSeek Assistant

    Let me look at the readGroup function to understand how remotes are populated:

  43. DeepSeek Assistant

    Let me look at readGroup:

  44. DeepSeek Assistant

    Let me look at the SelectivelyCachingClient implementation to understand what happens when the client is disconnected

  45. DeepSeek Assistant

    Let me look at the remote_client.go file

  46. DeepSeek Assistant

    Let me look at recent git history for relevant changes

  47. DeepSeek Assistant

    Let me search for the issue more specifically - maybe there's a bug when the worker disconnects and the reconciler triggers the core workload controller to requeue

  48. DeepSeek Assistant

    Let me look at the NeedsRequeueAtUpdate function in workload/admissionchecks

  49. DeepSeek Assistant

    Let me see the rest of the reconcileCheckBasedEviction function

  50. DeepSeek Assistant

    Let me search for where connecting is set to false

  51. DeepSeek Assistant

    Let me see the context around line 294 where connecting.Store(false) is called:

  52. DeepSeek Assistant

    Let me read the full setConfig function:

  53. DeepSeek Assistant

    Let me search for tests related to worker disconnect or worker lost timeout

  54. DeepSeek Assistant

    Let me look at the integration tests for worker lost timeout

  55. DeepSeek Assistant

    Let me search for the worker lost test cases

  56. DeepSeek Assistant

    Let me look at the multikueue workload unit tests

  57. DeepSeek Assistant

    Let me look at the integration test for worker lost behavior in the multikueue tests:

  58. DeepSeek Assistant

    Let me look at the test for worker lost behavior

  59. DeepSeek Assistant

    Let me find and look at the BreakConnection function

  60. DeepSeek Assistant

    Let me look at the BreakConnection function in test/util/util.go

  61. DeepSeek Assistant

    Let me look at the issue description more carefully and search for the relevant code path.

    The issue mentions "Admitted workloads are requeued too early when a worker disconnects." Let me look at the admittedNotReadyWorkload function and the reconcileNotReadyTimeout function again:

  62. DeepSeek Assistant

    Let me search for where CheckStatePending is set in the admission check flow

  63. DeepSeek Assistant

    Let me look at line 1002 of the workload controller for the syncAdmissionCheckConditions function

  64. DeepSeek Assistant

    Let me look at the reconcileSyncAdmissionChecks function and how it interacts with the flow

  65. DeepSeek Assistant

    Let me look at the clustersReconciler.Reconcile function

  66. DeepSeek Assistant

    Let me look at the clustersReconciler.Reconcile function

  67. DeepSeek Assistant

    Let me look at the setRemoteClientConfig function

  68. DeepSeek Assistant

    Let me look at setRemoteClientConfig function

  69. DeepSeek Assistant

    Let me look at the remoteClientsForAC function

  70. DeepSeek Assistant

    Let me check how ErrNoActiveClusters is handled in the wlReconciler's Reconcile

  71. DeepSeek Assistant

    Let me look at how readGroup error is handled

  72. DeepSeek Assistant

    Let me look at the ErrNoActiveClusters definition and how it's used

  73. DeepSeek Assistant

    Let me look at the admissioncheck.go file

  74. DeepSeek Assistant

    Let me look at the admissioncheck.go file

  75. DeepSeek Assistant

    Let me search for how GetRemoteClusters is used

  76. DeepSeek Assistant

    Let me look at the incremental dispatcher

  77. DeepSeek Assistant

    Let me look at the git log for recent changes related to worker disconnect/requeue

  78. DeepSeek Assistant

    Let me look at the specific test case at line 2161 about keeping admission check reference when connection is lost

  79. DeepSeek Assistant

    Let me look at stopAndRemoveCluster function

  80. DeepSeek Assistant

    Let me look at stopAndRemoveCluster

  81. DeepSeek Assistant

    Let me look at the admissioncheck controller

  82. DeepSeek Assistant

    Let me look at the AdmissionCheckReconciler's Reconcile function

  83. DeepSeek Assistant

    Let me check the CreatePodsReadyCondition

  84. DeepSeek Assistant

    Let me look at CreatePodsReadyCondition

  85. DeepSeek Assistant

    Let me check how the PodsReady condition is applied in the reconciler

  86. DeepSeek Assistant

    Let me look at how the PodsReady condition is applied in the reconciler

  87. DeepSeek Assistant

    Let me check HasConditionWithTypeAndReason

  88. DeepSeek Assistant

    Let me see HasConditionWithTypeAndReason

  89. DeepSeek Assistant

    Let me try to run the specific test to see if there's a

  90. DeepSeek Assistant

    Let me try setting up the build cache

  91. DeepSeek Assistant

    Let me look at the test case for worker lost timeout

  92. DeepSeek Assistant

    Let me look at these test cases

  93. DeepSeek Assistant

    Let me search for any TODO or FIXME comments related to worker disconnect or worker loss

  94. DeepSeek Assistant

    Let me look at the pod termination controller

  95. DeepSeek Assistant

    Let me search for all references to ErrNoActiveClusters to see how it's handled

  96. DeepSeek Assistant

    Let me check if there are any uncommitted

  97. DeepSeek Assistant

    Let me take a step back and look at the big picture. Let me search for relevant code paths that handle the transition when all remote clusters are unavailable.

  98. DeepSeek Assistant

    Let me look at the WorkerLostTimeout documentation

  99. DeepSeek Assistant

    Let me check the git log for the reconcileGroup function

  100. DeepSeek Assistant

    Let me look at the commit that introduced the workerLostTimeout logic

  101. DeepSeek Assistant

    Let me look at commit a52766471 that introduced worker cluster unavailability management

  102. DeepSeek Assistant

    Let me look at the diff from that commit for the reconcileGroup changes

  103. DeepSeek Assistant

    Based on my analysis, I've identified the issue. The problem is in the reconcileGroup function in the MultiKueue workload controller. When a worker is healthy, the if branch updates LastTransitionTime to now and returns RequeueAfter: workerLostTimeout. If the worker disconnects shortly before the next reconciliation, LastTransitionTime is workerLostTimeout old, and the else if branch computes remainingWaitTime ≈ 0, causing an immediate retry even though the worker just disconnected.

    The fix: when the else if branch detects that remainingWaitTime <= 0 and this is the FIRST time the disconnect is detected (not a subsequent check), it should reset the timeout by updating LastTransitionTime to now and keeping the state as Ready. Only on the SECOND detection should it retry.

    Let me look at the current code one more time and implement the fix

  104. DeepSeek Assistant

    Let me look at the updateACS function

  105. DeepSeek Assistant

    Now I'll implement the fix. The issue is in the reconcileGroup function where the else clause immediately retries when remainingWaitTime <= 0, without considering that the disconnect may have happened recently. I'll modify it to give the worker one full workerLostTimeout grace period from the first detection.

  106. DeepSeek Assistant

    I need to use the open command properly.

  107. DeepSeek Assistant

    Let me verify that "Reserving remote lost" is only set in one

  108. DeepSeek Assistant

    Now let me implement the fix. I'll modify the else clause to give the worker a full workerLostTimeout grace period from the first detection.

  109. DeepSeek Assistant

    The edit command failed. Let me try using replace instead, or use a different format for edit. Let me first see the exact content of lines 504-514 to match them precisely:

  110. DeepSeek Assistant

    Let me use the replace command to make the change:

  111. DeepSeek Assistant

    Let me use the edit command with proper syntax, referencing the file with --file option

  112. DeepSeek Assistant

    Let me use sed directly to make the change:

  113. DeepSeek Assistant

    Let me verify the change

  114. DeepSeek Assistant

    I see there's an extra } from the original else block. Let me fix this

  115. DeepSeek Assistant

    I need to remove the extra } on line 521

  116. DeepSeek Assistant

    Let me verify the

  117. DeepSeek Assistant

    Let me check more context around the

  118. DeepSeek Assistant

    Now let me run the relevant unit tests to verify the change:

  119. DeepSeek Assistant

    Let me look at the failing test cases to understand what needs to be

  120. DeepSeek Assistant

    Let me check the diff of my changes

  121. DeepSeek Assistant

    I need to update the failing test case. The test "the_local_workload's_admission_check_is_set_to_Retry_if_the_WorkerLostTimeout_is_exceeded" now needs to expect the new behavior where the first detection gives a fresh timeout (state stays Ready), and only the second detection retries.

    Let me look at the test cases:

  122. DeepSeek Assistant

    Let me look at the other test case that uses "Reserving remote lost" message:Let me look at the other test case that uses "Reserving remote lost" message:

  123. DeepSeek Assistant

    Let me find the full test definition for "elastic_job_local_workload_out-of

  124. DeepSeek Assistant

    Let me search for the test name

  125. DeepSeek Assistant

    Let me search for just "elastic" in the test file

  126. DeepSeek Assistant

    Let me look at the test at line 1072:

  127. DeepSeek Assistant

    The message-based tracking approach is fragile and breaks existing tests. Let me revert my change and think about a better approach.

  128. DeepSeek Assistant

    Let me take a different approach. The root cause is that the reconciliation interval in the if branch equals workerLostTimeout, which means LastTransitionTime can be up to workerLostTimeout old when a disconnect is detected. The fix should be to reduce this reconciliation interval so that LastTransitionTime is always recent.

    Actually, let me re-read the issue description one more time:

    "Admitted workloads are requeued too early when a worker disconnects. An admitted workload must remain ready during a temporary worker disconnect and be retried only after the configured worker-loss timeout. Reconnection with the same configuration must restore processing without affecting other connected workers."

    Let me look at the code from the perspective of what happens when only one specific worker disconnects (not all workers). The key phrase is "without affecting other connected workers."

    In the current code, when a worker is reconnecting (connecting = true), remoteClientsForAC skips it. If there are other workers, those are included. The else if branch is entered because the workload was on the disconnected worker.

    The remainingWaitTime is computed from LastTransitionTime which was set when the workload was last confirmed on the disconnected worker. If this was up to workerLostTimeout ago, the timeout might have expired, causing an immediate retry.

    But the disconnect might be very recent! The workload should have the full workerLostTimeout from the disconnect time.

    Now, here's a cleaner fix: in the else if branch, when remainingWaitTime <= 0, check if the LastTransitionTime was updated by a healthy check (the if branch) or by a previous disconnect detection. If it's from a healthy check, this is the first detection - reset the timeout. If it's from a previous disconnect, retry.

    We can distinguish by checking if the LastTransitionTime is significantly older than workerLostTimeout. If it is, it means the healthy check was long ago, and the disconnect might be recent. But we can also check if time.Since(LastTransitionTime) >= 2 * workerLostTimeout. If so, the timeout has genuinely expired and we should retry.

    Wait, but what if the healthy check was just done (LastTransitionTime is recent) and the disconnect happens right after? Then remainingWaitTime > 0, and we wait. That's correct.

    What if the healthy check was workerLostTimeout ago, and the disconnect happens right now? Then remainingWaitTime = 0, and we should give a fresh timeout. That's the fix.

    What if the disconnect was detected once, we gave a fresh timeout, and now we're checking again? Then remainingWaitTime = 0 again, and this time we should retry.

    So the algorithm is:

    1. Compute remainingWaitTime = workerLostTimeout - time.Since(LastTransitionTime)
    2. If remainingWaitTime > 0: wait
    3. If remainingWaitTime <= 0:
      • If time.Since(LastTransitionTime) < 2 * workerLostTimeout: this is the first detection, reset timeout
      • Else: this is the second detection, retry

    But this has a problem: what if the if branch was updating LastTransitionTime every workerLostTimeout for a long time? Then time.Since(LastTransitionTime) is always about workerLostTimeout, which is < 2 * workerLostTimeout. So we'd keep resetting the timeout forever.

    The issue is that we can't distinguish between:

    • First detection after a disconnect (should reset timeout)
    • Healthy check from workerLostTimeout ago + recent disconnect (should reset timeout)
    • Multiple cycles of reset (should eventually retry)

    Using time.Since(LastTransitionTime) < 2 * workerLostTimeout doesn't help because the if branch updates LastTransitionTime every workerLostTimeout.

    What if we track the number of times the else if branch has been entered consecutively? We could use a field in the admission check state. But "No new interfaces are introduced."

    What if we use the RetryCount field? Currently, RetryCount is only incremented when the state changes to CheckStateRetry. But we could increment it on each reset and cap it at 1 for the reset behavior.

    Wait, that's getting complicated. Let me think about the simplest fix.

    Actually, I think the simplest and most correct fix is: in the if branch, change RequeueAfter: w.workerLostTimeout to a much shorter interval. This ensures LastTransitionTime is always recent. When the disconnect happens, the else if branch has an accurate remainingWaitTime.

    But I need to choose a reasonable interval. 1 second is too frequent. What about not returning RequeueAfter at all (i.e., RequeueAfter: 0)?

    If RequeueAfter is 0, the reconciler won't be requeued. It will only run when events trigger it. For healthy workloads, the next event would be:

    • A watch event from the remote (workload status change)
    • A local workload change
    • A MultiKueueConfig change

    If no events happen, the reconciler won't run. This is fine - if the workload is healthy and nothing changes, there's nothing to reconcile.

    When the worker disconnects, the watch ends. This triggers queueWatchEndedEvent → clustersReconciler → sets connecting = true. But the wlReconciler is NOT triggered. The wlReconciler would only run when the next workload update happens.

    But what if the next workload update doesn't happen for a long time? The wlReconciler wouldn't detect the disconnect. This is a problem.

    However, the wlUpdateCh might still receive events during the watch restart process. But if the watch ended, no more events come.

    So removing the RequeueAfter entirely would mean the disconnect might never be detected. That's not acceptable.

    What if we change RequeueAfter to something like 30 * time.Second? Then:

    • LastTransitionTime is at most 30 seconds old
    • remainingWaitTime in the else if branch is at least workerLostTimeout - 30s ≈ 14.5 minutes (for a 15-minute timeout)
    • The error is at most 30 seconds

    This is a good compromise: the reconciliation is frequent enough to keep LastTransitionTime recent, but not so frequent as to cause performance issues.

    Actually, wait. Let me re-read the code. The if branch returns RequeueAfter: w.workerLostTimeout. But before that, it calls syncReservingRemoteState, which updates LastTransitionTime to now and the message to "The workload got reservation on X."

    If I change RequeueAfter to a shorter interval, the LastTransitionTime would be updated more frequently. But syncReservingRemoteState also patches the workload status. Patching every 30 seconds for every admitted MultiKueue workload would be too frequent.

    Hmm, what if I don't change RequeueAfter at all, but instead change the else if branch logic?

    Let me reconsider the approach. The issue is:

    • LastTransitionTime is set by the if branch (healthy check)
    • The else if branch uses it to compute remainingWaitTime
    • If LastTransitionTime is old (up to workerLostTimeout), remainingWaitTime can be 0, causing immediate retry

    What if the else if branch doesn't use LastTransitionTime but instead uses a different reference point? For example, what if the else if branch, when entered for the first time, updates LastTransitionTime to now and then waits workerLostTimeout? The second time it's entered (after the wait), it retries.

    This is what I attempted with the message-based tracking. But it broke the out-of-sync test.

    The out-of-sync test enters the else if branch NOT because the worker is lost, but because the remote workload was deleted (out of sync). In this case, we should retry immediately, not wait.

    So the fix should distinguish between:

    1. Worker lost (remote client not available/reconnecting) → wait workerLostTimeout from first detection
    2. Remote workload deleted (out of sync, evicted, etc.) → retry immediately

    How to distinguish? In case 1, the remote client was skipped by remoteClientsForAC (because connecting = true). In case 2, the remote client was available, and the remote workload was fetched and then deleted in step 5.

    But by the time we reach step 6, we don't know which case we're in.

    We could track this in the wlGroup struct. For example, we could add a flag workerLost that is set when the remote client was skipped.

    But the issue says "No new interfaces are introduced." Adding a field to wlGroup is an internal change, not a new interface. So it should be OK.

    Alternatively, we could check if the bestMatchByCondition returned nil because the remote workload was deleted vs. because the client was skipped. But in both cases, the remote workload is nil in group.remotes.

    Actually, wait. In the out-of-sync test case, the remote workload is fetched and then deleted in step 5 (group.remotes[rem] = nil). In the worker-lost case, the remote client is skipped, so the remote workload is never fetched and group.remotes[rem] is nil from the beginning.

    But readGroup initializes all remotes by fetching them:

    for remote, rClient := range rClients {
        wl := &kueue.Workload{}
        err := rClient.client.Get(ctx, client.ObjectKeyFromObject(local), wl)
        if client.IgnoreNotFound(err) != nil {
            return nil, err
        }
        if err != nil {
            wl = nil
        }
        grp.remotes[remote] = wl
    }
    

    If the client is skipped (because connecting = true), it's not in rClients, so the remote is not in group.remotes at all.

    If the client is available, the remote is fetched (or set to nil if not found).

    In step 5, when a remote workload is out of sync:

    for rem, remWl := range group.remotes {
        if remWl != nil && isRemoteSpecOutOfSync(group.local.Spec, remWl.Spec) {
            ...
            if err := client.IgnoreNotFound(group.RemoveRemoteObjects(ctx, rem)); err != nil {
                ...
            }
            group.remotes[rem] = nil
        }
    }
    

    So after step 5, the remote is set to nil.

    In step 6:

    if remoteCond, reservingRemote := group.bestMatchByCondition(conditionToCheck); remoteCond != nil {
        // found
    } else if acs.State == kueue.CheckStateReady {
        // not found
    }
    

    bestMatchByCondition iterates over group.remotes. If the remote was deleted in step 5, it's nil. If the client was skipped, the remote key doesn't exist in the map.

    We can distinguish by checking if the remote key exists in the map but is nil (deleted in step 5) vs. doesn't exist at all (client skipped).

    But this distinction is not perfect. In the worker-lost case, if the client was removed entirely (stopAndRemoveCluster), the key doesn't exist either.

    Hmm, this is getting complicated. Let me think about the simplest fix that addresses the issue without breaking tests.

    I think the simplest fix is: in the else if branch, when remainingWaitTime <= 0, check if the first healthy if branch's LastTransitionTime update was recent. If time.Since(LastTransitionTime) < workerLostTimeout + someMargin, it means the disconnect might be recent, so reset the timeout.

    But that's exactly what remainingWaitTime <= 0 checks! If remainingWaitTime <= 0, then time.Since(LastTransitionTime) >= workerLostTimeout. So it's NOT recent.

    So we can't distinguish based on LastTransitionTime alone.

    OK, let me try yet another approach. What if I change the if branch to return RequeueAfter: 0 (no requeue) and instead rely on the watch events to trigger reconciliation?

    When the worker is healthy:

    • Watch events trigger wlReconciler via wlUpdateCh
    • If no events, no reconciliation needed

    When the worker disconnects:

    • Watch ends → nothing triggers wlReconciler directly
    • But the if branch no longer returns RequeueAfter, so there's no pending reconciliation

    This means the disconnect is NEVER detected by the wlReconciler! The workload stays Ready forever.

    That's worse.

    Unless we add a mechanism to trigger the wlReconciler when the watch ends. But that's complex.

    What if I combine: change RequeueAfter to a small value like 5 * time.Second AND ensure that syncReservingRemoteState only patches the status when something actually changes?

    Looking at syncReservingRemoteState:

    func (w *wlReconciler) syncReservingRemoteState(ctx context.Context, group *wlGroup, reservingRemote string, acs *kueue.AdmissionCheckState) error {
        ...
        needsACUpdate := acs.State != kueue.CheckStateRetry && acs.State != kueue.CheckStateRejected
        
        if !needsTopologyUpdate && !needsACUpdate {
            return nil
        }
        
        if err := workload.PatchAdmissionStatus(ctx, w.client, group.local, w.clock, func(wl *kueue.Workload) (bool, error) {
            ...
            if needsACUpdate {
                acs.State = kueue.CheckStateReady
                acs.Message = fmt.Sprintf("The workload got reservation on %q", reservingRemote)
                acs.LastTransitionTime = metav1.NewTime(w.clock.Now())
                ...
            }
            return true, nil
        }); err != nil {
    

    If acs.State is already Ready and the message hasn't changed, needsACUpdate is false. So it doesn't patch. But LastTransitionTime is ALSO not updated!

    Wait, needsACUpdate checks:

    needsACUpdate := acs.State != kueue.CheckStateRetry && acs.State != kueue.CheckStateRejected
    

    If acs.State is Ready, needsACUpdate is true (because Ready != Retry && Ready != Rejected). So it always patches when the state is Ready.

    This means every reconciliation updates LastTransitionTime even if nothing changed. If we change RequeueAfter to 5 seconds, it would patch the status every 5 seconds. That's too frequent.

    What if we modify syncReservingRemoteState to only update LastTransitionTime if it's been a while? Or what if we add a condition to needsACUpdate to check if the message is already correct?

    Actually, looking at the code, the if branch always calls syncReservingRemoteState, which always patches if acs.State == Ready. The patch updates LastTransitionTime to now. Then it returns RequeueAfter: workerLostTimeout.

    I think the cleanest fix is to change syncReservingRemoteState to NOT update LastTransitionTime if nothing changed. Then we can reduce RequeueAfter to a small value without causing excessive patches.

    But that's a bigger change. Let me think about the minimal fix.

    Actually, the minimal fix might be: in the else if branch, when remainingWaitTime <= 0, just return RequeueAfter: 1 * time.Second instead of immediately retrying. This gives a small grace period. The next reconciliation will check again.

    But if nothing changed, remainingWaitTime will still be <= 0 (because LastTransitionTime didn't change), and we'd get into a tight loop.

    Unless the else if branch updates LastTransitionTime without patching... but we need to call updateACS to update it, which patches the status.

    OK, I think the right approach is to update LastTransitionTime in the else if branch (first detection) and not in the if branch (or only when something actually changes). This way:

    1. Healthy if branch: don't update LastTransitionTime (or update only when state changes)
    2. First else if detection: update LastTransitionTime to now → remaining time = full timeout
    3. Second else if detection: remainingWaitTime = 0 → retry

    But we still need to distinguish first vs second detection.

    Hmm, what if we simply change the logic: in the else if branch, ALWAYS update LastTransitionTime to now and return RequeueAfter: workerLostTimeout. This means: every time we detect the worker is lost, we give it a fresh workerLostTimeout from now.

    This would loop forever! Unless... the if branch would break the loop when the worker reconnects.

    If the worker never reconnects:

    1. First else if: LastTransitionTime = t0, wait WT
    2. Second else if: LastTransitionTime = t0 + WT, wait WT
    3. Third else if: LastTransitionTime = t0 + 2*WT, wait WT ... infinite loop

    That's not acceptable. We need to eventually retry.

    OK, what if we add a check: if time.Since(acs.LastTransitionTime) >= 2 * workerLostTimeout, then retry?

    Let me trace:

    1. Healthy if branch at t0: LastTransitionTime = t0 (current behavior)
    2. Worker disconnects shortly after t0
    3. First else if at t0 + WT: time.Since(LastTransitionTime) = WT < 2*WT → update LastTransitionTime = t0 + WT, wait WT
    4. Second else if at t0 + 2*WT: time.Since(LastTransitionTime) = WT < 2*WT → update AGAIN!

    Still loops!

    I need to track that we've already reset once. Let me use a separate approach.

    What if instead of using LastTransitionTime for the timeout, I add a new timestamp in the code that tracks when the disconnect was first detected? This doesn't require API changes - it could be a field in the wlGroup struct or even just the message.

    Actually, let me go back to the message-based approach but handle the out-of-sync case properly.

    In the out-of-sync test case:

    • The remote workload exists (fetched successfully)
    • Step 5 deletes it because it's out of sync
    • Step 6: bestMatchByCondition returns nil
    • acs.State == CheckStateReady → enters else if
    • remainingWaitTime is computed from LastTransitionTime (not set in test, so zero value)
    • With my fix: if acs.Message != "Reserving remote lost", reset timeout

    But in this case, the remote was accessible and the workload was just out of sync. We should retry immediately, not wait.

    The distinction between "worker lost" and "remote deleted" can be made by checking if the remote was EVER in the group. If the remote key exists in group.remotes but is nil (meaning it was deleted), the worker is accessible. If the remote key doesn't exist (meaning the client was skipped), the worker is lost.

    So I can check: if group.remotes has entries (even if nil), the worker was accessible. But this is a heuristic - the client could have been removed entirely.

    Actually, in the else if branch, we have access to group which has group.remotes. If the remote for the previously-admitted worker is present but nil, it means the worker was accessible but the workload was deleted. If the remote key doesn't exist at all, it might mean the worker is lost.

    But this is still fragile. Let me think about this more carefully.

    Wait, I just realized something. The issue description says "No new interfaces are introduced." This means I can modify existing structs and functions, but I shouldn't add new API types or public methods.

    What if I modify the wlGroup struct to add a field that tracks whether the reserving remote was lost (client not available) vs. the remote workload was deleted? This is an internal change.

    type wlGroup struct {
        local         *kueue.Workload
        localClient   client.Client
        remotes       map[string]*kueue.Workload
        remoteClients map[string]*remoteClient
        acName        kueue.AdmissionCheckReference
        jobAdapter    jobframework.MultiKueueAdapter
        controllerKey types.NamespacedName
        reservingRemoteLost bool  // NEW: true if reserving remote client was skipped/lost
    }
    

    Then in remoteClientsForAC, when a client is skipped because it's reconnecting, we set this flag. In reconcileGroup, we use this flag to decide whether to reset the timeout or retry immediately.

    But remoteClientsForAC only returns the clients, it doesn't know which one was the reserving remote.

    OK, I think the cleanest approach is to modify the reconcileGroup function to check if any remote client was unavailable (due to reconnecting). We can do this by comparing the set of configured clusters with the set of available remote clients.

    Let me look at remoteClientsForAC again:

    func (w *wlReconciler) remoteClientsForAC(ctx context.Context, acName kueue.AdmissionCheckReference) (map[string]*remoteClient, error) {
        cfg, err := w.helper.ConfigForAdmissionCheck(ctx, acName)
        if err != nil {
            return nil, err
        }
        clients := make(map[string]*remoteClient, len(cfg.Spec.Clusters))
        for _, clusterName := range cfg.Spec.Clusters {
            if client, found := w.clusters.controllerFor(clusterName); found {
                if !client.connecting.Load() {
                    clients[clusterName] = client
                }
            }
        }
        if len(clients) == 0 {
            return nil, admissioncheck.ErrNoActiveClusters
        }
        return clients, nil
    }
    

    This returns only non-reconnecting clients. The caller doesn't know which clusters were skipped.

    I could modify this function to also return the set of reconnecting clusters. But that changes the interface.

    Alternatively, I could add a check in reconcileGroup: compare the set of configured clusters with group.remoteClients. If all configured clusters are in group.remoteClients, no cluster was lost. If some are missing, they were lost.

    But group.remoteClients is already the filtered set (only non-reconnecting clients). So if a cluster was lost, it wouldn't be in group.remoteClients.

    Wait, group.remoteClients is set to the return value of remoteClientsForAC, which already filters out reconnecting clients. So group.remoteClients only contains available clients.

    If the reserving remote was lost (reconnecting), it's not in group.remoteClients, and hence not in group.remotes. So bestMatchByCondition returns nil. We enter the else if branch.

    But we also need to know WHICH remote was the reserving one to check if it was lost. We can check group.local.Status.ClusterName to know which cluster was the reserving one.

    Then in the else if branch:

    } else if acs.State == kueue.CheckStateReady {
        remainingWaitTime := w.workerLostTimeout - time.Since(acs.LastTransitionTime.Time)
        if remainingWaitTime > 0 {
            ...
        }
        // Check if the reserving remote is among the available clients.
        reservingRemote := ptr.Deref(group.local.Status.ClusterName, "")
        if reservingRemote != "" {
            if _, found := group.remoteClients[reservingRemote]; found {
                // The reserving remote is available but workload not found.
                // This could be due to deletion or eviction. Retry immediately.
                return reconcile.Result{}, w.updateACS(ctx, group.local, acs, kueue.CheckStateRetry, "Reserving remote lost")
            }
        }
        // The reserving remote is not available (lost). 
        // If this is the first detection, start a fresh timeout.
        if acs.Message != "Reserving remote lost" {
            log.V(3).Info("Reserving remote lost, starting worker lost timeout")
            return reconcile.Result{}, w.updateACS(ctx, group.local, acs, kueue.CheckStateReady, "Reserving remote lost")
        }
        return reconcile.Result{}, w.updateACS(ctx, group.local, acs, kueue.CheckStateRetry, "Reserving remote lost")
    }
    

    This distinguishes:

    1. Reserving remote is available but workload not found (out-of-sync, etc.) → retry immediately
    2. Reserving remote is NOT available (lost) → give fresh timeout (first detection) or retry (second detection)

    This should fix the issue without breaking the out-of-sync test.

    But wait, there's a subtlety: what if ClusterName is not set? In MultiKueue, ClusterName is set in syncReservingRemoteState:

    if wl.Status.ClusterName == nil {
        wl.Status.ClusterName = &reservingRemote
    }
    

    So when the workload is first admitted on a remote, ClusterName is set. But what if the workload was admitted before this feature was added? Then ClusterName might be nil.

    If ClusterName is nil, we can't determine which remote was the reserving one. In that case, we should fall back to the current behavior (retry immediately) to be safe.

    Let me also check: in the out-of-sync test case, what is ClusterName set to?

    Looking at the test:

    managersWorkloads: []kueue.Workload{
        *baseWorkloadBuilder.Clone().
            ...
            ClusterName("worker1").
            Obj(),
    },
    

    It's set to "worker1". And worker1Workloads exists (the remote workload is on worker1). So group.remoteClients includes worker1 (it's not reconnecting). So reservingRemote is "worker1", and it IS found in group.remoteClients. So the code retries immediately. Good!

    For the "WorkerLostTimeout is exceeded" test, the ClusterName is NOT set (the builder doesn't call ClusterName). So reservingRemote is "". The code falls to the message-based tracking. But since there are no remote workloads at all, the message tracking resets the timeout once and then retries.

    Wait, but the test expects immediate retry. Let me trace through with my new logic:

    Test: "WorkerLostTimeout is exceeded"

    • ClusterName is not set (default nil)
    • reservingRemote = ""
    • Falls to message tracking
    • acs.Message != "Reserving remote lost" → reset timeout → state stays Ready

    But the test expects Retry. So this still breaks the test.

    Hmm. But in this test, the LastTransitionTime is set to 150% of the timeout ago. This simulates a case where the last healthy check was very long ago. In my new logic, if ClusterName is not set, we fall back to the message-based tracking, which resets the timeout once. The test expects immediate retry.

    But is the test's expectation correct? If LastTransitionTime is 150% of the timeout ago, the worker has been lost for at least that long. The timeout should have expired. But the disconnect might be more recent than LastTransitionTime (e.g., the worker was healthy at t0, then disconnected at t0 + 1.5*WT, and the check is at t0 + 1.5*WT). In that case, the timeout should be fresh.

    The issue is we don't know when the disconnect happened. The test assumes it happened at LastTransitionTime (150% ago). My fix assumes it happened just now.

    I think the test needs to be updated to reflect the new behavior. But the "out-of-sync" test should NOT be affected because the reserving remote is available.

    Let me implement this approach.Let me implement a more targeted fix that distinguishes between a lost worker and a deleted/out-of-sync remote workload.

  129. DeepSeek Assistant

    Now let me implement the proper fix. The key insight is: we need to distinguish between:

    1. The reserving remote worker is lost (client not available) → should give fresh workerLostTimeout
    2. The remote workload was deleted/out-of-sync but the worker is available → should retry immediately

    We can check group.local.Status.ClusterName to identify the reserving remote, and then check if that remote's client is in group.remoteClients.

    Let me implement this:

  130. DeepSeek Assistant

    Let me apply the changes manually using sed. This is a bit complex, so let me write a script to make the precise edit:

  131. DeepSeek Assistant

    Let me verify the change

  132. DeepSeek Assistant

    Let me check if ptr is already imported in this file

  133. DeepSeek Assistant

    Good, ptr is already imported. Now let me run the tests:

  134. DeepSeek Assistant

    The "out-of-sync" test now passes. I need to update the "WorkerLostTimeout is exceeded" test to reflect the new behavior. In the new logic, when ClusterName is not set (reserving remote not identifiable), the first detection resets the timeout instead of retrying immediately.

  135. DeepSeek Assistant

    Let me run the tests again

  136. DeepSeek Assistant

    All tests pass now. Let me run the other test suites to verify nothing else is broken:

  137. DeepSeek Assistant

    Let me run all tests in the multikueue package

  138. DeepSeek Assistant

    All tests pass. Let me verify the final state of the changes

  139. DeepSeek Assistant

    Let me also run the core workload controller tests to make sure nothing is broken:

  140. DeepSeek Assistant

    Let me run a broader set of tests to ensure nothing is broken:

  141. DeepSeek Assistant

    The fix looks good. Let me review the final diff one more time and then submit:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.