Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-sonnet-5· anthropicHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Standalone elastic RayCluster self-updates raycluster-generation…

  1. User

    ISSUE DESCRIPTION

    Standalone elastic RayCluster self-updates raycluster-generation annotation What happened:

    A standalone Kueue-managed elastic RayCluster is continuously updated by Kueue.

    The kueue.x-k8s.io/raycluster-generation annotation tracks the RayCluster's own metadata.generation, but remains one value behind. For example:

    metadata.generation: 273390
    kueue.x-k8s.io/raycluster-generation: "273389"
    

    A later read showed the same pattern:

    metadata.generation: 273462
    kueue.x-k8s.io/raycluster-generation: "273461"
    

    Kueue logs showed repeated RayCluster reconciles followed by RayCluster update webhook calls from the Kueue controller manager.

    This is a self-triggered loop in the RayCluster custom annotation path. For standalone RayClusters, GetCustomAnnotations calls GetWorkloadslicingRayClusterCustomAnnotations(ctx, c, j.Object(), podSets, j.Name). That helper fetches the same RayCluster and returns the fetched RayCluster generation as an annotation to write back onto the RayCluster. The metadata patch then advances the RayCluster generation, making the annotation stale again.

    What you expected to happen:

    Kueue should not continuously patch a standalone elastic RayCluster when its replica counts have not changed.

    The raycluster-generation annotation is useful for RayJob and RayService, where the top-level object needs to account for an underlying RayCluster generation. A standalone RayCluster should not write its own generation back onto itself.

    How to reproduce it (as minimally and precisely as possible):

    1. Install Kueue v0.16.5 with ElasticJobsViaWorkloadSlices=true.
    2. Enable the RayCluster integration.
    3. Create a standalone RayCluster with:
    metadata:
      annotations:
        kueue.x-k8s.io/elastic-job: "true"
      labels:
        kueue.x-k8s.io/queue-name: <local-queue>
    
    1. Inspect the RayCluster generation and annotation repeatedly:
    kubectl get raycluster.ray.io <name> -n <namespace> \
      -o jsonpath='{.metadata.generation}{"\t"}{.metadata.annotations.kueue\.x-k8s\.io/raycluster-generation}{"\n"}'
    

    Anything else we need to know?:

    This may be related to #10160, which added generation-based elastic workload names on the release-0.16 branch.

    A fix is to omit kueue.x-k8s.io/raycluster-generation when the jobObject is the same standalone RayCluster as rayClusterName.

    Environment:

    • Kubernetes version (use kubectl version):
      • Client Version: v1.33.9
      • Server Version: v1.33.10-gke.1115000
    • Kueue version (use git describe --tags --dirty --always):
      • Deployed controller image: kueue:v0.16.5
      • Helm chart: kueue-0.16.5
    • Cloud provider or hardware configuration: GKE
    • OS (e.g: cat /etc/os-release): not collected
    • Kernel (e.g. uname -a): not collected
    • Install tools: Helm
    • Others:
      • ElasticJobsViaWorkloadSlices=true
      • TopologyAwareScheduling=false

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /kueue, current file: none) bash-$

  2. Claude Assistant

    Let's look at this test suite more closely - it might already exercise standalone RayCluster elastic behavior and reveal expectations.

  3. Claude Assistant

    Now let's implement the fix:

  4. Claude Assistant

    Now let's build and run existing tests.

  5. Claude Assistant

    Build passes. Now let's implement the standalone RayCluster wiring so that this fix actually matters (implementing GetCustomAnnotations for RayCluster), since the issue explicitly describes that path existing. Let's check further whether we should add this to make the fix meaningful, or leave as defensive-only. Let's re-read issue text carefully.

  6. Claude Assistant

    Now let's verify that without the fix (reverting temporarily) the test fails, to confirm this actually reproduces the described bug.

  7. Claude Assistant

    Confirmed: reproduces the bug without fix, and fix resolves it. Let's restore the fix.

  8. Claude Assistant

    Now remove the temporary repro test file (per instructions not to add new tests to the repo).

  9. Claude Assistant

    Now let's run the full test suite for the raycluster, rayjob, rayservice packages to make sure nothing broke.

  10. Claude Assistant

    All pass. Let's also review the final diff.

  11. Claude Assistant

    Now let's double check the whole build compiles and run broader tests to be safe (e.g. integration test compile check).

  12. Claude Assistant

    All good. Let's also run the entire unit test suite for the jobs directory broadly to be thorough.

  13. Claude Assistant

    All tests pass. Let's also check gofmt formatting.

  14. Claude Assistant

    Formatting is clean (no output from gofmt -l means correctly formatted).

  15. Claude Assistant

    Summary

    Root cause: GetWorkloadslicingRayClusterCustomAnnotations in pkg/controller/jobs/raycluster/common.go always fetches the RayCluster named rayClusterName and writes its metadata.generation back onto jobObject as the kueue.x-k8s.io/raycluster-generation annotation. When jobObject is that same standalone RayCluster (as opposed to an owning RayJob/RayService that tracks a separately-named RayCluster), this creates a self-referential loop: writing the annotation patches the object, which bumps its own metadata.generation, making the just-written annotation stale on the next reconcile — causing Kueue to patch the RayCluster forever.

    Fix: Added a helper isSameStandaloneRayCluster(jobObject, rayClusterName) that detects when jobObject is itself a *rayv1.RayCluster whose name matches rayClusterName. In that case, GetWorkloadslicingRayClusterCustomAnnotations now omits the RayClusterGenerationAnnotation entirely from the returned annotation map (while still returning the RayClusterPodsetReplicaSizesAnnotation), preventing the self-triggered update loop. For RayJob/RayService (where jobObject is never the same object as the tracked RayCluster), behavior is unchanged.

    I verified this reproduces/fixes the issue with a temporary test (since-removed) simulating a standalone RayCluster calling this function with itself as both jobObject and target rayClusterName: without the fix, the generation annotation was returned (reproducing the bug); with the fix, it's omitted.

    Checklist review

    • Reproduced the reported error: Yes. I wrote a temporary Go test that calls GetWorkloadslicingRayClusterCustomAnnotations with a *rayv1.RayCluster as both the jobObject and the entity referenced by rayClusterName (simulating the standalone-RayCluster case described in the issue). Before the fix, it returned RayClusterGenerationAnnotation populated with the RayCluster's own generation — exactly the self-referential write described in the issue. I removed this temporary test file afterward per instructions (no new tests were added to the repo).
    • Confirmed the fix resolves it: Yes. After applying the fix, the same reproduction test showed the RayClusterGenerationAnnotation key omitted from the returned map, which means mergeAnnotations will not attempt to patch that annotation for a standalone RayCluster, breaking the self-triggered update loop. This matches the exact fix suggested in the issue ("omit kueue.x-k8s.io/raycluster-generation when the jobObject is the same standalone RayCluster as rayClusterName").
    • Edge cases considered:
      • RayJob/RayService (owning jobs) continue to get the generation annotation as before, since their jobObject is never the same type/name as the tracked RayCluster — verified via existing unit tests in rayjob and rayservice packages, which still pass.
      • The "RayCluster not found" and "other get error" paths are preserved for the non-standalone case.
      • PodsetReplicaSizesAnnotation is still always returned/updated regardless of the standalone case, so scale-up/scale-down tracking for standalone elastic RayClusters is unaffected.
      • The fix is generic (based on object identity/type, not hardcoded names), so it will correctly guard this behavior even if a future integration wires GetCustomAnnotations for standalone RayCluster.
    • Ran existing tests: Yes. Ran go build ./..., go vet ./..., and go test across pkg/controller/jobs/raycluster, pkg/controller/jobs/rayjob, pkg/controller/jobs/rayservice, pkg/controller/jobframework, and the full pkg/controller/jobs/... tree — all passed with no failures or regressions.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.