ISSUE DESCRIPTION Standalone elastic RayCluster self-updates raycluster-generation…
- User
ISSUE DESCRIPTION
Standalone elastic RayCluster self-updates raycluster-generation annotation What happened:
A standalone Kueue-managed elastic RayCluster is continuously updated by Kueue.
The
kueue.x-k8s.io/raycluster-generationannotation tracks the RayCluster's ownmetadata.generation, but remains one value behind. For example:metadata.generation: 273390 kueue.x-k8s.io/raycluster-generation: "273389"A later read showed the same pattern:
metadata.generation: 273462 kueue.x-k8s.io/raycluster-generation: "273461"Kueue logs showed repeated RayCluster reconciles followed by RayCluster update webhook calls from the Kueue controller manager.
This is a self-triggered loop in the RayCluster custom annotation path. For standalone RayClusters,
GetCustomAnnotationscallsGetWorkloadslicingRayClusterCustomAnnotations(ctx, c, j.Object(), podSets, j.Name). That helper fetches the same RayCluster and returns the fetched RayCluster generation as an annotation to write back onto the RayCluster. The metadata patch then advances the RayCluster generation, making the annotation stale again.What you expected to happen:
Kueue should not continuously patch a standalone elastic RayCluster when its replica counts have not changed.
The
raycluster-generationannotation is useful for RayJob and RayService, where the top-level object needs to account for an underlying RayCluster generation. A standalone RayCluster should not write its own generation back onto itself.How to reproduce it (as minimally and precisely as possible):
- Install Kueue v0.16.5 with
ElasticJobsViaWorkloadSlices=true. - Enable the RayCluster integration.
- Create a standalone RayCluster with:
metadata: annotations: kueue.x-k8s.io/elastic-job: "true" labels: kueue.x-k8s.io/queue-name: <local-queue>- Inspect the RayCluster generation and annotation repeatedly:
kubectl get raycluster.ray.io <name> -n <namespace> \ -o jsonpath='{.metadata.generation}{"\t"}{.metadata.annotations.kueue\.x-k8s\.io/raycluster-generation}{"\n"}'Anything else we need to know?:
This may be related to #10160, which added generation-based elastic workload names on the release-0.16 branch.
A fix is to omit
kueue.x-k8s.io/raycluster-generationwhen thejobObjectis the same standalone RayCluster asrayClusterName.Environment:
- Kubernetes version (use
kubectl version):- Client Version: v1.33.9
- Server Version: v1.33.10-gke.1115000
- Kueue version (use
git describe --tags --dirty --always):- Deployed controller image:
kueue:v0.16.5 - Helm chart:
kueue-0.16.5
- Deployed controller image:
- Cloud provider or hardware configuration: GKE
- OS (e.g:
cat /etc/os-release): not collected - Kernel (e.g.
uname -a): not collected - Install tools: Helm
- Others:
ElasticJobsViaWorkloadSlices=trueTopologyAwareScheduling=false
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /kueue, current file: none) bash-$
- Install Kueue v0.16.5 with
Assistant
Let me understand the issue:
- When a standalone RayCluster (not part of a RayJob or RayService) is managed by Kueue, Kueue writes a
kueue.x-k8s.io/raycluster-generationannotation to the RayCluster. - This annotation tracks the RayCluster's own
metadata.generation, but it's always one behind. - This causes an infinite loop: Kueue reads the RayCluster, gets its generation, writes it back as an annotation, which updates the generation, making the annotation stale again, causing another reconcile.
The fix should be: omit
kueue.x-k8s.io/raycluster-generationwhen thejobObjectis the same standalone RayCluster. In other words, for a standalone RayCluster, we should not write its own generation back onto itself.Let me explore the repository structure first.
- When a standalone RayCluster (not part of a RayJob or RayService) is managed by Kueue, Kueue writes a
Assistant
The grep returned a match but with an error. Let me search more carefully.
Assistant
Let me find all references to this annotation.
Assistant
Let me look at the
common.gofile inAssistant
Let me try to open the file using a different syntax.
Assistant
Now I can see the
GetWorkloadslicingRayClusterCustomAnnotationsfunction. Let me find where it's calledAssistant
Let me find the RayCluster controller's
GetCustomAnnotations:Assistant
Let me find the RayCluster controller file:
Assistant
Let me check the raycluster_controller.go to see if it has GetCustomAnnotations.
Assistant
Let me look at the reconciler code to understand how
GetCustomAnnotationsis used:Assistant
Let me search for where
GetCustomAnnotationsis called in the reconciler codeAssistant
Let me look at the area around line 962 to understand the context.
Assistant
Let me use sed to read lines around 962.
Assistant
Let me check if RayCluster implements
GetCustomAnnotationsAssistant
Let me look at the common_test.go to understand the test cases:
Assistant
Let me check the raycluster_webhook.go:
Assistant
Let me check the interface definition for
JobWithCustomAnnotations:Assistant
Let me look at the interface definition.
Assistant
Let me see which types implement
JobWithCustomAnnotations:Assistant
Let me look at the workloadslicing package and how it handles annotations
Assistant
Let me look at the
GetWorkloadNameExtraPartand how it's used:Assistant
Let me look at the reconciler code around line 1437 to see how
GetWorkloadNameExtraPartis usedAssistant
Let me search more broadly for where annotations are set on the RayCluster:
Assistant
Let me look at the workloadslicing package
Assistant
Let me look at the workloadslicing package:
Assistant
Let me look at the full reconcile flow to understand where annotations might get written
Assistant
Let me look at the full
ReconcileGenericJobAssistant
Let me continue reading the reconcile flow after
ensureOneWorkload:Assistant
Let me search more thoroughly for any
GetCustomAnnotationsimplementation related to RayCluster:Assistant
Let me check git status and recent commits:
Assistant
Let me search for
ElasticWorkloadNameProviderand check what RayCluster implements:Assistant
Let me look at the full
ensureOneWorkloadfunction:Assistant
Let me look at the RayJob and RayService controllers to understand their
GetCustomAnnotations:Assistant
Let me also look at the RayService controller:
Assistant
Let me search for any RayCluster GetCustomAnnotations implementation more broadly:
Assistant
Let me look at
JobPodSets:Assistant
Let me search more broadly for JobPodSets:
Assistant
Let me look at
JobPodSets:Assistant
Let me look at
startJobandstopJobAssistant
Let me see the rest of the reconcile flow after
ensureOneWorkloadAssistant
Let me check the git log for changes to the raycluster controller files
Assistant
Let me look at the commit that added generation-based elastic workload names:
Assistant
Let me see what was added to common.go in that commit specifically
Assistant
Let me check if there's a
JobWithCustomAnnotationsimplementation for RayCluster in any fileAssistant
Let me check all callers of
GetWorkloadslicingRayClusterCustomAnnotationsandGetWorkloadNameExtraPart:Assistant
Let me check if there's a release branch or tag that might have different
Assistant
Now I have a clear understanding of the issue. Let me look at the test file to understand the expected
Assistant
Now I have a thorough understanding of the issue. Let me summarize:
The Problem: When
GetWorkloadslicingRayClusterCustomAnnotationsis called with ajobObjectthat is the same standalone RayCluster asrayClusterName, it:- Fetches the RayCluster's generation
- Returns it as
RayClusterGenerationAnnotationto be written back onto the same RayCluster - The write increments the RayCluster's generation, making the annotation stale
- This triggers another reconcile, creating an infinite loop
The Fix: Omit
RayClusterGenerationAnnotationwhenjobObjectis the same RayCluster asrayClusterName.Let me implement this fix in `common.go
Assistant
Now I'll implement the fix. The key change is in
GetWorkloadslicingRayClusterCustomAnnotations- whenjobObjectis the same asrayClusterName, we should skip theRayClusterGenerationAnnotation.Assistant
Let me open the file first and then edit it
Assistant
Now I can edit the function. Let me replace lines 335-366:
Assistant
Let me try writing the replacement to a temporary file and then using sed or a different approach:
Assistant
Now let me use a script to replace lines 335-366:
Assistant
Let me verify the change was applied correctly:
Assistant
The change looks correct. Now let me verify the code compiles and tests pass.
Assistant
Let me set up the Go cache and try again:
Assistant
The build passes! Now let me run the tests for this package:
Assistant
All existing tests pass! Let me also run all tests in the raycluster package to make sure nothing else is broken:
Assistant
All tests pass. Let me also run the related tests for rayjob and rayservice:
Assistant
Let me run the rayservice tests too:
Assistant
Let me run the full test suites for both packages:
Assistant
All tests pass. Let me also run the jobframework reconciler tests and workloadslicing tests:
Assistant
Let me now verify the change with a quick sanity check - let me look at the final state of the modified function:
Assistant
Let me verify the git diff to confirm the change is correct:
Assistant
The diff looks correct. Let me now run the broader test suite one more time to ensure everything is clean:
Assistant
All tests pass. Let me force re-run them to be certain:
Assistant
All tests pass. Let me clean up the temporary file:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →