Skip to content
TheChatArchiveThe Chat Archive
MiniMaxminimax/minimax-m3· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…

  1. User

    ISSUE DESCRIPTION

    Woodpecker services appear to be running on UI & API after pipeline completes

    Component

    server

    Describe the bug

    Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.

    These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).

    Steps to reproduce

    • Install woodpecker next-d05c597374
    • Use kubernetes backend
    • Define pipeline with services running infinitely, like running postgres
    services:
      - name: postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        ports:
          - 5432
        environment:
          POSTGRES_PASSWORD: somepass
          POSTGRES_USER: someuser
          POSTGRES_DB: somedb
    steps:
      - name: wait-for-postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        environment:
          DB_HOST: postgres
          DB_PORT: 5432
          MAX_WAIT: 60
        commands:
          - ./scripts/wait-for-postgres.sh
    
    • Run the pipeline, observe pods start for the service and the step
    • once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
    pr-build > clone (#2):
    Step: clone
    Started: 1779195580
    Stopped: 1779195594
    Type: clone
    State: success
    
    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    
    pr-build > wait-for-postgres (#4):
    Step: wait-for-postgres
    Started: 1779195596
    Stopped: 1779195602
    Type: commands
    State: success
    

    Expected behavior

    Pipeline services are stopped once the pipeline steps complete, the pipeline completes.

    System Info

    next-d05c597374
    kubernetes backend, bitbucket datacenter forge
    

    Additional context

    No response

    Validations

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /woodpecker, current file: none) bash-$

  2. MiniMax Assistant

    Let me start by exploring the repository structure to understand the codebase.

  3. MiniMax Assistant

    This is a Go project. Let me explore the server folder structure.

  4. MiniMax Assistant

    Now let me look at how services are defined, how steps transition from running to success/failure, and how they get cleaned up. Let me searchfor the relevant code.

  5. MiniMax Assistant

    Now let me look at how steps/services are managed in the workflow_status logic:

  6. MiniMax Assistant

    Let me look at the agent code to understand how step updates are sent.

  7. MiniMax Assistant

    Now let me look at how workflow completion is detected:

  8. MiniMax Assistant

    Let me look at the completeChildrenIfParentCompleted function:

  9. MiniMax Assistant

    Let me look at the git history to understand recent changes:

  10. MiniMax Assistant

    This is the commit that introduced the fix for the same issue! Let me look at it.

  11. MiniMax Assistant

    Wait, but according to git log, the commit says "Set workflow services stuck in running state to finished" was already merged. Let me check the current state of rpc.go and the step_status.go to see if it actually has this fix.

    The current state showed:

    func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) {
    	for _, c := range completedWorkflow.Children {
    		if c.Running() {
    			if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {
    

    Hmm. This is calling with model.StatusKilled, not model.StatusSkipped. Let me re-check.

    Actually, let me re-look at what I see:

    func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) {
    	for _, c := range completedWorkflow.Children {
    		if c.Running() {
    			if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {
    

    So the issue is:

    1. model.StatusKilled is passed as status.
    2. But UpdateStepToStatusSkipped overrides the state to StatusSuccess if Started != 0:
    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status  // Set to StatusKilled
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // Override to StatusSuccess because Started != 0
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    Hmm wait. Let me re-look at the code path:

    1. Service step has state = running, started != 0
    2. completeChildrenIfParentCompleted is called.
    3. For the running service, it calls UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled).
    4. UpdateStepToStatusSkipped sets step.State = StatusKilled (the passed value).
    5. Since step.Started != 0, it overrides step.State = StatusSuccess.
    6. Returns the updated step.
    7. The in-memory state c.State = updated.State = StatusSuccess.

    So the in-memory state would be StatusSuccess. This is correct.

    But the issue says:

    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    

    So the state is still running. But based on the analysis above, after completeChildrenIfParentCompleted is called, the state should be set to Success in memory and then persisted.

    Wait, let me check the current state of the file again to make sure I have the latest version.

    Let me look at the current state of rpc.go more carefully:

  12. MiniMax Assistant

    So the current state of the file has the completeChildrenIfParentCompleted function. Now let me check UpdateStepToStatusSkipped and understand why the issue is happening.

  13. MiniMax Assistant

    Wait, looking again at the current state of step_status.go:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    The step.State = model.StatusSuccess // for daemons that are killed is overriding the passed status. This is wrong!Let me verify the current state of this function.

  14. MiniMax Assistant

    Let me look at the git history for UpdateStepToStatusSkipped and the comment "for daemons that are killed":

  15. MiniMax Assistant

    Let me trace through the workflow structure:

  16. MiniMax Assistant

    So services and steps are added as stages to the same config. Now let me look at how workflows are split up - it seems like each "stage" is a workflow.

  17. MiniMax Assistant

    Let me look at the compiler more carefully:

  18. MiniMax Assistant

    Let me look at how the pipeline runtime handles services and steps:

  19. MiniMax Assistant

    Let me look at the runtime.go file to understand how workflows are dispatched:

  20. MiniMax Assistant

    Let me look at how the agent executes workflows:

  21. MiniMax Assistant

    Let me look at the order of operations in Done:

  22. MiniMax Assistant

    Let me check how the failure is set for service steps:

  23. MiniMax Assistant

    So for services:

    • The default failure is metadata.FailureFail
    • But if forceIgnoreServiceFailure is true and the step is detached, then failure is setto metadata.FailureIgnore.

    Let me check how detached is determined:

  24. MiniMax Assistant

    So services have detached = true. But whether they have FailureIgnore or FailureFail depends on whether forceIgnoreServiceFailure is set.

    Let me look at the model.Step.Failure more carefully:

  25. MiniMax Assistant

    Let me now look at the runtime step.go to see what happens with services (detached steps):

  26. MiniMax Assistant

    Let me look at the kubernetes backend to see how services are destroyed:

  27. MiniMax Assistant

    So when DestroyWorkflow is called, it stops all the pods. After the pods are stopped, the WaitStep calls should return.

    Now let me look at stopPod to see what happens to the pod:

  28. MiniMax Assistant

    Let me re-read the workflow.go carefully:

  29. MiniMax Assistant

    Let me check what forceIgnoreServiceFailure is and whether it's set by default:

  30. MiniMax Assistant

    Let me look at IsThereRunningStage:

  31. MiniMax Assistant

    Let me run the existing tests for the relevant package:

  32. MiniMax Assistant

    Let me look at checkWorkflowState:

  33. MiniMax Assistant

    Now let me look at the sanitize_test.go to see how the scenarios are tested:

  34. MiniMax Assistant

    Let me check how the workflow gets dispatched:

  35. MiniMax Assistant

    Let me look at the scheduler to understand how workflows are dispatched:

  36. MiniMax Assistant

    Let me look at the API that shows pipeline state:

  37. MiniMax Assistant

    Let me look at the full RPC.Done code:

  38. MiniMax Assistant

    Let me search for how workflows get split into separate DB workflows:

  39. MiniMax Assistant

    Looking back at the issue description, it shows different workflow IDs (#2, #3, #4). Maybe the workflow structure is different from what I assumed. Let me look at the scheduler and queue to see if stages can be split into separate workflows. Let me also look at recent issues with stages being separated.

  40. MiniMax Assistant

    Let me look at how the items are stored and how they relate to workflows:

  41. MiniMax Assistant

    Let me check the recent git changes that might have introduced this issue:

  42. MiniMax Assistant

    This commit (60df1c618) is interesting. It fixed the workflow hang issue on services. It calls destroyWorkflowFunc() BEFORE waiting for log uploads. This is needed because services run forever in detached goroutines.

    Now let me look at the current state of the workflow.go to see if this fix is still in place:

  43. MiniMax Assistant

    Now let me look at the actual step behavior in the runtime:

  44. MiniMax Assistant

    Now let me look at the tracer to see how the step status is reported:

  45. MiniMax Assistant

    Now I understand the flow. Let me look at how traceStep calls the tracer:

  46. MiniMax Assistant

    Let me re-look at the agent runner.go to confirm:

  47. MiniMax Assistant

    Let me look at the order of operations in Done more carefully:

  48. MiniMax Assistant

    Let me look at the order of operations in the workflow more carefully:

  49. MiniMax Assistant

    Let me search for how workflows get split:

  50. MiniMax Assistant

    Let me re-examine the issue. Let me look at the agent runner and check the actual run flow:

  51. MiniMax Assistant

    Let me look at the stopPod function:

  52. MiniMax Assistant

    Let me look at the kubernetes WaitStep:

  53. MiniMax Assistant

    Let me check the kubernetes tests for wait step:

  54. MiniMax Assistant

    Let me re-check the tests and look at the existing test:

  55. MiniMax Assistant

    Let me check the recent changes that might have caused this issue:

  56. MiniMax Assistant

    Let me look at the prior state of completeChildrenIfParentCompleted:

  57. MiniMax Assistant

    Let me look at the kubernetes WaitStep more carefully:

  58. MiniMax Assistant

    Let me check the kubernetes pod delete behavior:

  59. MiniMax Assistant

    Let me look at the cancel flow:

  60. MiniMax Assistant

    Let me look at how the agent handles the queue's ErrorAtOnce:

  61. MiniMax Assistant

    Let me check the recent git changes that might have split workflows:

  62. MiniMax Assistant

    OK so I think the workflows #2, #3, #4 must be from a different yaml structure. Let me focus on the actual issue.

    The issue says:

    • Services remain as state=running in the API even after the pods are destroyed.
    • The state is cleaned up only after the pipeline times out.

    The key insight from the code review:

    1. The UpdateStepToStatusSkipped function has a bug: it overrides the passed status to StatusSuccess when Started != 0, regardless of the passed status. This means when Done is called for a workflow with running services, the service is updated to Success (not Killed).

    2. The current implementation calls completeChildrenIfParentCompleted in Done. This function calls UpdateStepToStatusSkipped with model.StatusKilled. But due to the bug, the state is overridden to StatusSuccess.

    3. There's also a potential issue with the order of operations: the agent's Update from the service's goroutine might happen BEFORE or AFTER the agent's Done.

    Let me think about this more carefully. The user says the state is "running" with Stopped: 0. This means the step was never updated to a final state.

    Let me look at what happens when the workflow is canceled:

    1. User cancels the pipeline.
    2. Cancel is called. It evicts running workflows from the queue.
    3. ErrorAtOnce is called, which closes the task's done channel.
    4. The agent's Wait returns with ErrCancel.
    5. The agent cancels the workflowCtx.
    6. The runtime's WaitStep returns with ctx.Err() == context.Canceled.
    7. The service's goroutine completes and traces the step.
    8. The tracer sends Update to the server with Canceled: true.
    9. The server updates the service state to StatusKilled (because Canceled: true).
    10. The runtime's Run returns.
    11. The agent sends Done to the server.
    12. The server's Done is processed.
    13. completeChildrenIfParentCompleted is called. The service is in StatusKilled state. Not running. So not updated.
    14. UpdateWorkflowStatusToDone is called. The workflow is updated based on the service state (StatusKilled).

    But the issue says the state is "running". So the chain is breaking somewhere.

    Wait, I just realized something. The service's goroutine is in completeStep -> WaitStep. When the context is canceled, WaitStep returns. But wait, the kubernetes WaitStep uses r.ctx:

    waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID)
    

    But r.ctx is the workflowCtx (set via WithContext). When the workflowCtx is canceled, WaitStep returns.

    But wait, destroyWorkflowFunc() is also called when the workflow is canceled (via defer destroyWorkflowFunc()). Let me check:

    destroyWorkflowFunc := sync.OnceFunc(func() {
        ctx := runnerCtx //nolint:contextcheck
        if ctx.Err() != nil {
            // runnerCtx itself is done — fall back to a short-lived shutdown context.
            ctx = GetShutdownCtx()
        }
        if err := r.engine.DestroyWorkflow(ctx, r.spec, r.taskUUID); err != nil {
            logger.Error().Err(err).Msg("could not destroy workflow")
        }
    })
    
    // we make sure cleanup always happens
    defer destroyWorkflowFunc()
    

    So when the workflow is canceled:

    • cancelWorkflowCtx is called via the agent's defer.
    • This cancels r.ctx.
    • WaitStep returns.
    • The service's goroutine completes.
    • The runtime's Run returns.
    • The agent's defer destroyWorkflowFunc() runs. But wait, this is the runtime's defer, not the agent's. Let me re-check.

    Actually, the destroyWorkflowFunc is in the runtime's Run function. It's called via defer when Run returns.

    So when the workflow is canceled:

    1. The agent's Wait returns with ErrCancel.
    2. The agent cancels the workflowCtx (which is r.ctx).
    3. The runtime's WaitStep returns with ctx.Err() == context.Canceled.
    4. The service's goroutine completes and traces the step.
    5. r.uploadWait.Wait() returns.
    6. The runtime's Run returns.
    7. The runtime's defer destroyWorkflowFunc() runs. This destroys the pods.
    8. The agent's Run returns.
    9. The agent sends Done to the server.

    So the pods are destroyed AFTER the service's goroutine completes. Hmm.

    Actually, let me re-check the order. The runtime's defer runs after the function returns. So:

    1. r.uploadWait.Wait() returns (because the service's goroutine completed).
    2. The function returns.
    3. The defer runs. destroyWorkflowFunc() is called. The pods are destroyed.

    So the pods are destroyed AFTER the service's goroutine completed. But the service's goroutine completed because WaitStep returned. Why did WaitStep return? Because the context was canceled.

    So the sequence is:

    1. The agent cancels the workflowCtx (r.ctx).
    2. WaitStep returns.
    3. The service's goroutine completes.
    4. destroyWorkflowFunc() is called. The pods are destroyed.

    So the pods are destroyed AFTER the workflow is canceled. This is correct.

    OK so my analysis suggests the service state should be StatusKilled. But the issue says it's "running". So something else is happening.

    Let me check if the issue is with how the kubernetes WaitStep returns. Maybe it doesn't return when the context is canceled.

    Actually, let me just focus on the fix. The issue is about services remaining in running state. Let me look at what changes have been made recently that might affect this.

    Looking at the recent commits:

    • 855eae143: Set workflow services stuck in running state to finished - This added the completeChildrenIfParentCompleted function and moved it before UpdateWorkflowStatusToDone.
    • 7efe8bf51: sanitize agent should allow done updates to canceled pipelines - This refactored sanitize functions.

    Let me check the state of the issue at the time of these commits. The issue mentions "next-d05c597374" which is a specific commit. Let me see what that commit is.

    Let me check what the issue's commit is:

  63. MiniMax Assistant

    Looking at the recent commits for workflow splitting:

  64. MiniMax Assistant

    Let me look at how the actual workflow scheduling works. Maybe there's a recent change that I'm missing:

  65. MiniMax Assistant

    The user's commit is d05c59737. Let me see what that commit is:

  66. MiniMax Assistant

    Let me look at the actual issue more carefully. The user reports that even though the pods are destroyed by the kubernetes backend, the woodpecker API shows the service as "running".

    Let me check if the issue can be reproduced with a simple test:

  67. MiniMax Assistant

    Now let me look at the rest of the test to understand:

  68. MiniMax Assistant

    Let me look at the actual workflow.go to see the full picture:

  69. MiniMax Assistant

    Let me now analyze the actual issue and make the fix. Based on my analysis, the bug is that the service workflow hangs because:

    1. The runtime's destroyWorkflowFunc() destroys the pods.
    2. But the service's goroutine is in WaitStep() which doesn't return when the pod is deleted (only when the phase changes).
    3. So the runtime hangs in r.uploadWait.Wait().
    4. The agent never sends Done for the service workflow.
    5. The pipeline is not marked as done until the workflow timeout.

    Let me look at the runtime to understand the fix better:

  70. MiniMax Assistant

    Now let me first run the existing tests to make sure they pass:

  71. MiniMax Assistant

    The test file has a //go:build test tag. So I need to enable the test build tag to run them.

  72. MiniMax Assistant

    The existing tests pass. They use the dummy backend which has a stop channel mechanism. Now let me check the kubernetes backend tests:

  73. MiniMax Assistant

    Let me look at the dummy backend test more carefully to understand the WaitStep behavior:

  74. MiniMax Assistant

    Now I understand the issue completely. Let me write the fix.

    The fix is to:

    1. Modify the runtime to have a cancel function for r.ctx.
    2. Modify WithContext to wrap the user's context with a cancellable context.
    3. Modify destroyWorkflowFunc to call the cancel function.

    Let me make the changes:

  75. MiniMax Assistant

    Let me first modify the runtime to add the cancel function:

  76. MiniMax Assistant

    The edit duplicated the comment. Let me clean it up:

  77. MiniMax Assistant

    Let me fix the comment to be properly formatted:

  78. MiniMax Assistant

    Now let me modify the WithContext option to wrap the context:

  79. MiniMax Assistant

    Now let me modify the destroyWorkflowFunc in workflow.go to also cancel the context:

  80. MiniMax Assistant

    Now let me build to make sure my changes compile:

  81. MiniMax Assistant

    The error is about web/dist which is unrelated. Let me build only the relevant packages:

  82. MiniMax Assistant

    This error is unrelated to my changes. Let me check:

  83. MiniMax Assistant

    This is unrelated to my changes. Now let me run the existing tests:

  84. MiniMax Assistant

    All tests pass with my fix. Let me now run all the runtime tests to make sure I didn't break anything:

  85. MiniMax Assistant

    Some tests failed. Let me see which ones:

  86. MiniMax Assistant

    Let me check all the tests that fail:

  87. MiniMax Assistant

    Only TestWithOptions is failing. This test is checking that r.ctx == ctx. With my fix, r.ctx is wrapped, so it's not exactly equal.

    Let me check if I should update the test or work around this. Let me look at the test:

  88. MiniMax Assistant

    Looking at the test more carefully, it's specifically checking that r.ctx == ctx. With my fix wrapping the context, this fails. The instructions say to ignore test failures that need to be updated to reflect changes I made, so this is acceptable.

    However, let me take a different approach to maintain backward compatibility. Let me NOT wrap the context, and instead add a separate WithCancel option that the agent can pass:

  89. MiniMax Assistant

    Now let me revert WithContext and add WithCancel:

  90. MiniMax Assistant

    Now let me build and run tests:

  91. MiniMax Assistant

    The test now passes. Now I need to update the agent to pass the cancel function:

  92. MiniMax Assistant

    Now let me build and test:

  93. MiniMax Assistant

    I need to use the cancel function in a way that matches context.CancelFunc. Letme wrap it.

  94. MiniMax Assistant

    All tests pass now. Let me run the rest of the tests to make sure I didn't break anything:

  95. MiniMax Assistant

    The dummy backend test fails because it requires the test build tag. Let me run with that tag:

  96. MiniMax Assistant

    The server/router failure is due to web/dist missing - unrelated to my fix. Let me make sure all the relevant tests pass:

  97. MiniMax Assistant

    All tests pass. Let me also write a test to reproduce the actual issue scenario and verify the fix works:

  98. MiniMax Assistant

    The step was skipped because OnSuccess is false. Let me update the test to set OnSuccess=true.

  99. MiniMax Assistant

    The mock backend's TailStep returns nil, which causes a nil pointer dereference. Let me fix the mock:

  100. MiniMax Assistant

    Excellent! My fix works. Let me verify the bug exists without my fix by removing the WithCancel:

  101. MiniMax Assistant

    The bug is reproduced. Without my fix, the run times out. With my fix, the run completes successfully.

    Let me clean up the test file and verify all the existing tests still pass:

  102. MiniMax Assistant

    Now let me check the current state of my changes:

  103. MiniMax Assistant

    Let me run all the tests once more to make sure everything is working:

  104. MiniMax Assistant

    All tests pass. Let me also check the cli code which uses pipeline_runtime.New to make sure it still works:

  105. MiniMax Assistant

    The cli/exec/exec.go doesn't use WithCancel. That's fine because my fix is optional (ctxCancel is nil if not provided). Let me verify by building cli:

  106. MiniMax Assistant

    Let me also check if there are other places where the agent creates workflow contexts that need the cancel function:

  107. MiniMax Assistant

    Let me also look at the e2e tests to see if there are any related tests:

  108. MiniMax Assistant

    Let me also see what tests exist for the runtime:

  109. MiniMax Assistant

    All tests pass. Let me check if there are any other tests in the agent package that I should be aware of:

  110. MiniMax Assistant

    Let me also look at the complete diff once more:

  111. MiniMax Assistant

    Let me also check that I haven't introduced any lint issues:

  112. MiniMax Assistant

    These are pre-existing warnings (not from my changes). My fix doesn't introduce new lint issues.

    Let me also check the workflow.go one more time to make sure my changes are correct:

  113. MiniMax Assistant

    Let me also look at the agent runner.go:

  114. MiniMax Assistant

    Let me also check the option.go file:

  115. MiniMax Assistant

    Let me also verify the runtime.go change:

  116. MiniMax Assistant

    Let me also run the full test suite one more time to confirm everything works:

  117. MiniMax Assistant

    All tests pass. Let me also run the integration tests:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.