Skip to content
TheChatArchiveThe Chat Archive
GLMz-ai/glm-5.2· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Admitted workloads are requeued too early when a worker disconnects An…

  1. User

    ISSUE DESCRIPTION

    Admitted workloads are requeued too early when a worker disconnects

    An admitted workload must remain ready during a temporary worker disconnect and be retried only after the configured worker-loss timeout. Reconnection with the same configuration must restore processing without affecting other connected workers.

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /kueue, current file: none) bash-$

  2. GLM Assistant

    Let me start by exploring the repository structure to understand what we're working with.

  3. GLM Assistant

    Let me explore the repository structure.

  4. GLM Assistant

    This is a Go project - Kueue, a Kubernetes queueing system. Let me explore the structure more.

  5. GLM Assistant

    I need to make only one tool call at a time.

  6. GLM Assistant

    Let me explore the main package structure.

  7. GLM Assistant

    Let me search for worker-related code and "worker-loss" timeout.

  8. GLM Assistant

    Let me run one command at a time.

  9. GLM Assistant

    This looks like it's about MultiKueue. Let me look at the MultiKueue cluster controller since that deals with worker connections.

  10. GLM Assistant

    Let me look at the workload.go file which handles workload admission in multikueue.

  11. GLM Assistant

    Let me see the middle part of the file that was truncated, especially the reconcileGroup function and any worker-loss logic.

  12. GLM Assistant

    Let me look at the reconcileGroup and syncToSingleCluster/nominateAndSynchronizeWorkers functions in detail.

  13. GLM Assistant

    Now let me understand the issue. The issue says "Admitted workloads are requeued too early when a worker disconnects". Let me look at the logic around the workerLostTimeout and the reserving remote lost handling.

    Looking at the code in reconcileGroup:

    if remoteCond, reservingRemote := group.bestMatchByCondition(conditionToCheck); remoteCond != nil {
        // ... admitted path
        return reconcile.Result{RequeueAfter: w.workerLostTimeout}, nil
    } else if acs.State == kueue.CheckStateReady {
        // If there is no reserving and the AC is ready, the connection with the reserving remote might
        // be lost, keep the workload admitted for keepReadyTimeout and put it back in the queue after that.
        remainingWaitTime := w.workerLostTimeout - time.Since(acs.LastTransitionTime.Time)
        if remainingWaitTime > 0 {
            log.V(3).Info("Reserving remote lost, retry", "retryAfter", remainingWaitTime)
            return reconcile.Result{RequeueAfter: remainingWaitTime}, nil
        } else {
            return reconcile.Result{}, w.updateACS(ctx, group.local, acs, kueue.CheckStateRetry, "Reserving remote lost")
        }
    }
    

    So in the admitted path (when the remote workload is still admitted), we requeue after w.workerLostTimeout. But what happens when the worker disconnects? The remote workload might temporarily not be found (because the connection is lost). So bestMatchByCondition returns nil for the WorkloadAdmitted condition, and we fall into the else if acs.State == kueue.CheckStateReady branch.

    Wait, but let me understand the flow. When a worker disconnects, the informer for that cluster stops. So the remote workload object in the cache becomes stale - it might still show the workload as admitted, or it might be removed from the cache.

    Let me look at how readGroup works and how remote workloads are fetched.

  14. GLM Assistant

    Let me look at the remoteClient structure and how the connection state is tracked.

  15. GLM Assistant

    Let me find the remoteClient struct definition.

  16. GLM Assistant

    Let me look at the watch ending and reconnection logic.

  17. GLM Assistant

    Let me look at the git log and recent changes to understand the context.

  18. GLM Assistant

    Now I understand the issue. Let me look at the test file to understand the expected behavior.

  19. GLM Assistant

    Let me look at the clustersReconciler and how clients are accessed.

  20. GLM Assistant

    Let me look at the test setup to understand how the tests work.

  21. GLM Assistant

    Now I need to understand the test cases for the "WorkerLostTimeout not exceeded" and "exceeded" tests. Let me look at what worker1Workloads look like in those tests. The LastTransitionTime is set to 50% or 150% of the timeout. Let me check if there are remote workloads in those test cases.

  22. GLM Assistant

    Let me look at the specific test cases for WorkerLostTimeout to see if they have worker1Workloads (remote workloads).

  23. GLM Assistant

    Let me look at how the clock is set on remoteClient and the setConfig function.

  24. GLM Assistant

    Let me look at integration tests for multikueue worker disconnect scenarios.

  25. GLM Assistant

    Let me look at the BreakConnection utility and the other integration test about connection loss.

  26. GLM Assistant

    Let me look at the full clustersReconciler Reconcile and retry logic.

  27. GLM Assistant

    Let me find all callers of stopAndRemoveCluster.

  28. GLM Assistant

    Let me look at the newClustersReconciler and newWlReconciler functions.

  29. GLM Assistant

    Let me search for how the remoteClient clock is used in tests.

  30. GLM Assistant

    Let me look at SetAdmissionCheckState and how LastTransitionTime is managed.

  31. GLM Assistant

    Let me look at the other integration test about connection loss.

  32. GLM Assistant

    Let me look at getKubeConfig to understand the BreakConnection flow.

  33. GLM Assistant

    Let me check the multikueuecluster_test.go for tests related to loadClientConfig failure and stopAndRemoveCluster.

  34. GLM Assistant

    Let me look at the "update client with invalid path config" test case. It has:

    • A cluster with a path location type and empty path (invalid).
    • An existing remoteClient.
    • Expected: cluster status is BadKubeConfig, wantCancelCalled: 1 (the watchCancel is called).
    • Expected error.

    The test checks wantRemoteClients? Let me look at what wantRemoteClients is for this case. Actually, looking at the test, it doesn't have wantRemoteClients for the "update client with invalid path config" case. Let me look at how the test verifies the remoteClients.

  35. GLM Assistant

    Let me look at how the test verifies wantRemoteClients and the test runner.

  36. GLM Assistant

    Let me look at the test verification logic for wantRemoteClients.

  37. GLM Assistant

    Let me find the test runner code that checks wantRemoteClients.

  38. GLM Assistant

    Let me look at the newTestClient helper and setReconnectState.

  39. GLM Assistant

    Let me check how clustersReconciler and wlReconciler are created in production (controllers.go).

  40. GLM Assistant

    Now I have a thorough understanding of the issue. Let me implement the fix. First, let me open the multikueuecluster.go file to make changes to the remoteClient struct.

  41. GLM Assistant

    Now let me make the changes. First, let me add the lastLostAt field to the remoteClient struct.

  42. GLM Assistant

    Now let me add the lastLostAt field to the struct:

  43. GLM Assistant

    Let me use the edit command instead.

  44. GLM Assistant

    Now let me update newRemoteClient to set lastLostAt:

  45. GLM Assistant

    Now let me add the helper methods for lastLostAt. Let me add them after resetFailedConnAttempt:

  46. GLM Assistant

    Now let me update setConfig success path to call markConnected():

  47. GLM Assistant

    The edit went wrong - it broke the code structure. Let me fix it:

  48. GLM Assistant

    Good, the setConfig is fixed. Now let me update the watch-end goroutine to call markDisconnected:

  49. GLM Assistant

    There's an extra } at line 427. Let me fix it:

  50. GLM Assistant

    Good. Now let me add the disconnectedAt map to the clustersReconciler struct. Let me find it.

  51. GLM Assistant

    There's a duplicate remoteClients line at 612. Let me fix that:

  52. GLM Assistant

    Good. Now let me update stopAndRemoveCluster to record the disconnect time:

  53. GLM Assistant

    Now let me update setRemoteClientConfig to clear disconnectedAt after success. Let me add a helper method and update the function:

  54. GLM Assistant

    Now let me add the clearDisconnectedAt and clusterLostSince methods after controllerFor:

  55. GLM Assistant

    Now let me initialize the disconnectedAt map in newClustersReconciler:

  56. GLM Assistant

    There's an issue - configNamespace is missing and wlUpdateCh is duplicated. Let me fix that:

  57. GLM Assistant

    Good, newClustersReconciler is now correct. Now let me update the workload reconciler's else if acs.State == kueue.CheckStateReady branch. Let me open the workload.go file.

  58. GLM Assistant

    Now let me update the else if acs.State == kueue.CheckStateReady branch to use the disconnect time:

  59. GLM Assistant

    There's an extra } at line 524. Let me fix it:

  60. GLM Assistant

    Let me verify workload.ClusterName exists and check the imports are fine. Let me build the project:

  61. GLM Assistant

    The build succeeds. Now let me run the existing unit tests to see if they pass:

  62. GLM Assistant

    All existing unit tests pass. Now let me create a temporary test to verify the fix works for the disconnect scenario. Let me first look at the existing test structure for the "WorkerLostTimeout" cases to understand how to set up a connecting (disconnected) client.

  63. GLM Assistant

    Let me check if the test workload builder has a Cluster method and understand the test setup for ClusterName.

  64. GLM Assistant

    Let me create a temporary test to verify the fix. First, let me understand the test helper for the Reconcile result check:

  65. GLM Assistant

    Let me check if the test captures the RequeueAfter from Reconcile:

  66. GLM Assistant

    Let me create a temporary test to verify the fix works for the disconnect scenario:

  67. GLM Assistant

    Let me check the correct import path for the fake clock used in the existing tests:

  68. GLM Assistant

    OK, the import path for the testing util is sigs.k8s.io/kueue/pkg/util/testing (aliased as utiltesting). And the API testing helpers are utiltestingapi (probably sigs.k8s.io/kueue/pkg/util/testing/v1beta2 or similar). Let me check.

  69. GLM Assistant

    Let me fix the temporary test file with the correct imports:

  70. GLM Assistant

    The test passes. The fix works:

    • "disconnect recent - stay Ready": The workload was admitted an hour ago (muchEarlier), but the worker disconnected recently (50% of timeout ago). The workload stays Ready with a requeue after 2m30s (remaining timeout).
    • "disconnect old - Retry": The worker disconnected 150% of the timeout ago. The workload is requeued (Retry).

    Without the fix, both cases would have been immediately requeued (Retry) because acs.LastTransitionTime was an hour ago.

    Let me verify this by reverting the fix temporarily and running the test:

  71. GLM Assistant

    Let me verify the test fails without the fix by temporarily reverting the workload.go change:

  72. GLM Assistant

    This confirms the bug exists without the fix. The "disconnect recent - stay Ready" case fails (gets "Retry" instead of "Ready") because the old code uses acs.LastTransitionTime (an hour ago) instead of the actual disconnect time. Now let me restore the fix:

  73. GLM Assistant

    The fix works. Now let me also test the scenario where the remoteClient is removed (like BreakConnection) — the disconnectedAt map should be used. Let me update the temporary test:

  74. GLM Assistant

    All test cases pass. Now let me clean up the temporary test file and run the full existing test suite:

  75. GLM Assistant

    Now let me run the full test to make sure I didn't break anything broader:

  76. GLM Assistant

    Let me also run the broader controller tests to make sure nothing is broken:

  77. GLM Assistant

    All tests pass. Now let me review the full diff to make sure everything is clean:

  78. GLM Assistant

    Let me restore the blank line that was removed in workload.go:

  79. GLM Assistant

    Now let me rebuild and run the tests once more to confirm everything is clean:

  80. GLM Assistant

    Let me run go vet to check for any issues:

  81. GLM Assistant

    Let me do a final review of the complete diff:

  82. GLM Assistant

    Let me verify there are no leftover temporary files:

  83. GLM Assistant

    The temporary files in /tmp are outside the repo. Let me clean them up:

  84. GLM Assistant

    Now let me run the full multikueue unit test suite one final time to confirm everything passes:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.