ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…
- User
ISSUE DESCRIPTION
Woodpecker services appear to be running on UI & API after pipeline completes
Component
server
Describe the bug
Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.
These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).
Steps to reproduce
- Install woodpecker next-d05c597374
- Use kubernetes backend
- Define pipeline with services running infinitely, like running postgres
services: - name: postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 ports: - 5432 environment: POSTGRES_PASSWORD: somepass POSTGRES_USER: someuser POSTGRES_DB: somedb steps: - name: wait-for-postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 environment: DB_HOST: postgres DB_PORT: 5432 MAX_WAIT: 60 commands: - ./scripts/wait-for-postgres.sh- Run the pipeline, observe pods start for the service and the step
- once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
pr-build > clone (#2): Step: clone Started: 1779195580 Stopped: 1779195594 Type: clone State: success pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: running pr-build > wait-for-postgres (#4): Step: wait-for-postgres Started: 1779195596 Stopped: 1779195602 Type: commands State: successExpected behavior
Pipeline services are stopped once the pipeline steps complete, the pipeline completes.
System Info
next-d05c597374 kubernetes backend, bitbucket datacenter forgeAdditional context
No response
Validations
- Read the docs.
- Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
- Checked that the bug isn't fixed in the
nextversion already [https://woodpecker-ci.org/versions]
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /woodpecker, current file: none) bash-$
Assistant
Let me start by exploring the repository structure to understand the codebase.
Assistant
This is a Go project. Let me explore the server folder structure.
Assistant
Now let me look at how services are defined, how steps transition from running to success/failure, and how they get cleaned up. Let me searchfor the relevant code.
Assistant
Now let me look at how steps/services are managed in the workflow_status logic:
Assistant
Let me look at the agent code to understand how step updates are sent.
Assistant
Now let me look at how workflow completion is detected:
Assistant
Let me look at the
completeChildrenIfParentCompletedfunction:Assistant
Let me look at the git history to understand recent changes:
Assistant
This is the commit that introduced the fix for the same issue! Let me look at it.
Assistant
Wait, but according to git log, the commit says "Set workflow services stuck in running state to finished" was already merged. Let me check the current state of
rpc.goand thestep_status.goto see if it actually has this fix.The current state showed:
func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) { for _, c := range completedWorkflow.Children { if c.Running() { if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {Hmm. This is calling with
model.StatusKilled, notmodel.StatusSkipped. Let me re-check.Actually, let me re-look at what I see:
func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) { for _, c := range completedWorkflow.Children { if c.Running() { if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {So the issue is:
model.StatusKilledis passed as status.- But
UpdateStepToStatusSkippedoverrides the state toStatusSuccessifStarted != 0:
func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status // Set to StatusKilled if step.Started != 0 { step.State = model.StatusSuccess // Override to StatusSuccess because Started != 0 step.Finished = finished } return &step, store.StepUpdate(&step) }Hmm wait. Let me re-look at the code path:
- Service step has state = running, started != 0
completeChildrenIfParentCompletedis called.- For the running service, it calls
UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled). UpdateStepToStatusSkippedsetsstep.State = StatusKilled(the passed value).- Since
step.Started != 0, it overridesstep.State = StatusSuccess. - Returns the updated step.
- The in-memory state
c.State = updated.State=StatusSuccess.
So the in-memory state would be
StatusSuccess. This is correct.But the issue says:
pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: runningSo the state is still
running. But based on the analysis above, aftercompleteChildrenIfParentCompletedis called, the state should be set toSuccessin memory and then persisted.Wait, let me check the current state of the file again to make sure I have the latest version.
Let me look at the current state of
rpc.gomore carefully:Assistant
So the current state of the file has the
completeChildrenIfParentCompletedfunction. Now let me checkUpdateStepToStatusSkippedand understand why the issue is happening.Assistant
Wait, looking again at the current state of
step_status.go:func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }The
step.State = model.StatusSuccess // for daemons that are killedis overriding the passed status. This is wrong!Let me verify the current state of this function.Assistant
Let me look at the git history for
UpdateStepToStatusSkippedand the comment "for daemons that are killed":Assistant
Let me trace through the workflow structure:
Assistant
So services and steps are added as stages to the same config. Now let me look at how workflows are split up - it seems like each "stage" is a workflow.
Assistant
Let me look at the compiler more carefully:
Assistant
Let me look at how the pipeline runtime handles services and steps:
Assistant
Let me look at the runtime.go file to understand how workflows are dispatched:
Assistant
Let me look at how the agent executes workflows:
Assistant
Let me look at the order of operations in
Done:Assistant
Let me check how the failure is set for service steps:
Assistant
So for services:
- The default failure is
metadata.FailureFail - But if
forceIgnoreServiceFailureis true and the step is detached, then failure is settometadata.FailureIgnore.
Let me check how
detachedis determined:- The default failure is
Assistant
So services have
detached = true. But whether they haveFailureIgnoreorFailureFaildepends on whetherforceIgnoreServiceFailureis set.Let me look at the model.Step.Failure more carefully:
Assistant
Let me now look at the runtime step.go to see what happens with services (detached steps):
Assistant
Let me look at the kubernetes backend to see how services are destroyed:
Assistant
So when
DestroyWorkflowis called, it stops all the pods. After the pods are stopped, theWaitStepcalls should return.Now let me look at
stopPodto see what happens to the pod:Assistant
Let me re-read the workflow.go carefully:
Assistant
Let me check what
forceIgnoreServiceFailureis and whether it's set by default:Assistant
Let me look at
IsThereRunningStage:Assistant
Let me run the existing tests for the relevant package:
Assistant
Let me look at
checkWorkflowState:Assistant
Now let me look at the sanitize_test.go to see how the scenarios are tested:
Assistant
Let me check how the workflow gets dispatched:
Assistant
Let me look at the scheduler to understand how workflows are dispatched:
Assistant
Let me look at the API that shows pipeline state:
Assistant
Let me look at the full RPC.Done code:
Assistant
Let me search for how workflows get split into separate DB workflows:
Assistant
Looking back at the issue description, it shows different workflow IDs (#2, #3, #4). Maybe the workflow structure is different from what I assumed. Let me look at the scheduler and queue to see if stages can be split into separate workflows. Let me also look at recent issues with stages being separated.
Assistant
Let me look at how the items are stored and how they relate to workflows:
Assistant
Let me check the recent git changes that might have introduced this issue:
Assistant
This commit (60df1c618) is interesting. It fixed the workflow hang issue on services. It calls
destroyWorkflowFunc()BEFORE waiting for log uploads. This is needed because services run forever in detached goroutines.Now let me look at the current state of the workflow.go to see if this fix is still in place:
Assistant
Now let me look at the actual step behavior in the runtime:
Assistant
Now let me look at the tracer to see how the step status is reported:
Assistant
Now I understand the flow. Let me look at how
traceStepcalls the tracer:Assistant
Let me re-look at the agent runner.go to confirm:
Assistant
Let me look at the order of operations in
Donemore carefully:Assistant
Let me look at the order of operations in the workflow more carefully:
Assistant
Let me search for how workflows get split:
Assistant
Let me re-examine the issue. Let me look at the agent runner and check the actual run flow:
Assistant
Let me look at the
stopPodfunction:Assistant
Let me look at the kubernetes WaitStep:
Assistant
Let me check the kubernetes tests for wait step:
Assistant
Let me re-check the tests and look at the existing test:
Assistant
Let me check the recent changes that might have caused this issue:
Assistant
Let me look at the prior state of
completeChildrenIfParentCompleted:Assistant
Let me look at the kubernetes WaitStep more carefully:
Assistant
Let me check the kubernetes pod delete behavior:
Assistant
Let me look at the cancel flow:
Assistant
Let me look at how the agent handles the queue's ErrorAtOnce:
Assistant
Let me check the recent git changes that might have split workflows:
Assistant
OK so I think the workflows #2, #3, #4 must be from a different yaml structure. Let me focus on the actual issue.
The issue says:
- Services remain as state=running in the API even after the pods are destroyed.
- The state is cleaned up only after the pipeline times out.
The key insight from the code review:
-
The
UpdateStepToStatusSkippedfunction has a bug: it overrides the passed status toStatusSuccesswhenStarted != 0, regardless of the passed status. This means whenDoneis called for a workflow with running services, the service is updated toSuccess(notKilled). -
The current implementation calls
completeChildrenIfParentCompletedinDone. This function callsUpdateStepToStatusSkippedwithmodel.StatusKilled. But due to the bug, the state is overridden toStatusSuccess. -
There's also a potential issue with the order of operations: the agent's
Updatefrom the service's goroutine might happen BEFORE or AFTER the agent'sDone.
Let me think about this more carefully. The user says the state is "running" with
Stopped: 0. This means the step was never updated to a final state.Let me look at what happens when the workflow is canceled:
- User cancels the pipeline.
Cancelis called. It evicts running workflows from the queue.ErrorAtOnceis called, which closes the task's done channel.- The agent's
Waitreturns withErrCancel. - The agent cancels the workflowCtx.
- The runtime's
WaitStepreturns withctx.Err() == context.Canceled. - The service's goroutine completes and traces the step.
- The tracer sends
Updateto the server withCanceled: true. - The server updates the service state to
StatusKilled(becauseCanceled: true). - The runtime's
Runreturns. - The agent sends
Doneto the server. - The server's
Doneis processed. completeChildrenIfParentCompletedis called. The service is inStatusKilledstate. Not running. So not updated.UpdateWorkflowStatusToDoneis called. The workflow is updated based on the service state (StatusKilled).
But the issue says the state is "running". So the chain is breaking somewhere.
Wait, I just realized something. The service's goroutine is in
completeStep->WaitStep. When the context is canceled, WaitStep returns. But wait, the kubernetes WaitStep usesr.ctx:waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID)But
r.ctxis the workflowCtx (set viaWithContext). When the workflowCtx is canceled, WaitStep returns.But wait,
destroyWorkflowFunc()is also called when the workflow is canceled (viadefer destroyWorkflowFunc()). Let me check:destroyWorkflowFunc := sync.OnceFunc(func() { ctx := runnerCtx //nolint:contextcheck if ctx.Err() != nil { // runnerCtx itself is done — fall back to a short-lived shutdown context. ctx = GetShutdownCtx() } if err := r.engine.DestroyWorkflow(ctx, r.spec, r.taskUUID); err != nil { logger.Error().Err(err).Msg("could not destroy workflow") } }) // we make sure cleanup always happens defer destroyWorkflowFunc()So when the workflow is canceled:
cancelWorkflowCtxis called via the agent's defer.- This cancels
r.ctx. - WaitStep returns.
- The service's goroutine completes.
- The runtime's
Runreturns. - The agent's
defer destroyWorkflowFunc()runs. But wait, this is the runtime's defer, not the agent's. Let me re-check.
Actually, the
destroyWorkflowFuncis in the runtime'sRunfunction. It's called via defer whenRunreturns.So when the workflow is canceled:
- The agent's
Waitreturns withErrCancel. - The agent cancels the workflowCtx (which is
r.ctx). - The runtime's
WaitStepreturns withctx.Err() == context.Canceled. - The service's goroutine completes and traces the step.
r.uploadWait.Wait()returns.- The runtime's
Runreturns. - The runtime's
defer destroyWorkflowFunc()runs. This destroys the pods. - The agent's
Runreturns. - The agent sends
Doneto the server.
So the pods are destroyed AFTER the service's goroutine completes. Hmm.
Actually, let me re-check the order. The runtime's defer runs after the function returns. So:
r.uploadWait.Wait()returns (because the service's goroutine completed).- The function returns.
- The defer runs.
destroyWorkflowFunc()is called. The pods are destroyed.
So the pods are destroyed AFTER the service's goroutine completed. But the service's goroutine completed because
WaitStepreturned. Why didWaitStepreturn? Because the context was canceled.So the sequence is:
- The agent cancels the workflowCtx (r.ctx).
- WaitStep returns.
- The service's goroutine completes.
destroyWorkflowFunc()is called. The pods are destroyed.
So the pods are destroyed AFTER the workflow is canceled. This is correct.
OK so my analysis suggests the service state should be
StatusKilled. But the issue says it's "running". So something else is happening.Let me check if the issue is with how the kubernetes WaitStep returns. Maybe it doesn't return when the context is canceled.
Actually, let me just focus on the fix. The issue is about services remaining in running state. Let me look at what changes have been made recently that might affect this.
Looking at the recent commits:
- 855eae143: Set workflow services stuck in running state to finished - This added the
completeChildrenIfParentCompletedfunction and moved it beforeUpdateWorkflowStatusToDone. - 7efe8bf51: sanitize agent should allow done updates to canceled pipelines - This refactored sanitize functions.
Let me check the state of the issue at the time of these commits. The issue mentions "next-d05c597374" which is a specific commit. Let me see what that commit is.
Let me check what the issue's commit is:
Assistant
Looking at the recent commits for workflow splitting:
Assistant
Let me look at how the actual workflow scheduling works. Maybe there's a recent change that I'm missing:
Assistant
The user's commit is
d05c59737. Let me see what that commit is:Assistant
Let me look at the actual issue more carefully. The user reports that even though the pods are destroyed by the kubernetes backend, the woodpecker API shows the service as "running".
Let me check if the issue can be reproduced with a simple test:
Assistant
Now let me look at the rest of the test to understand:
Assistant
Let me look at the actual workflow.go to see the full picture:
Assistant
Let me now analyze the actual issue and make the fix. Based on my analysis, the bug is that the service workflow hangs because:
- The runtime's
destroyWorkflowFunc()destroys the pods. - But the service's goroutine is in
WaitStep()which doesn't return when the pod is deleted (only when the phase changes). - So the runtime hangs in
r.uploadWait.Wait(). - The agent never sends
Donefor the service workflow. - The pipeline is not marked as done until the workflow timeout.
Let me look at the runtime to understand the fix better:
- The runtime's
Assistant
Now let me first run the existing tests to make sure they pass:
Assistant
The test file has a
//go:build testtag. So I need to enable thetestbuild tag to run them.Assistant
The existing tests pass. They use the dummy backend which has a stop channel mechanism. Now let me check the kubernetes backend tests:
Assistant
Let me look at the dummy backend test more carefully to understand the WaitStep behavior:
Assistant
Now I understand the issue completely. Let me write the fix.
The fix is to:
- Modify the runtime to have a cancel function for
r.ctx. - Modify
WithContextto wrap the user's context with a cancellable context. - Modify
destroyWorkflowFuncto call the cancel function.
Let me make the changes:
- Modify the runtime to have a cancel function for
Assistant
Let me first modify the runtime to add the cancel function:
Assistant
The edit duplicated the comment. Let me clean it up:
Assistant
Let me fix the comment to be properly formatted:
Assistant
Now let me modify the
WithContextoption to wrap the context:Assistant
Now let me modify the
destroyWorkflowFuncinworkflow.goto also cancel the context:Assistant
Now let me build to make sure my changes compile:
Assistant
The error is about web/dist which is unrelated. Let me build only the relevant packages:
Assistant
This error is unrelated to my changes. Let me check:
Assistant
This is unrelated to my changes. Now let me run the existing tests:
Assistant
All tests pass with my fix. Let me now run all the runtime tests to make sure I didn't break anything:
Assistant
Some tests failed. Let me see which ones:
Assistant
Let me check all the tests that fail:
Assistant
Only
TestWithOptionsis failing. This test is checking thatr.ctx == ctx. With my fix,r.ctxis wrapped, so it's not exactly equal.Let me check if I should update the test or work around this. Let me look at the test:
Assistant
Looking at the test more carefully, it's specifically checking that
r.ctx == ctx. With my fix wrapping the context, this fails. The instructions say to ignore test failures that need to be updated to reflect changes I made, so this is acceptable.However, let me take a different approach to maintain backward compatibility. Let me NOT wrap the context, and instead add a separate
WithCanceloption that the agent can pass:Assistant
Now let me revert
WithContextand addWithCancel:Assistant
Now let me build and run tests:
Assistant
The test now passes. Now I need to update the agent to pass the cancel function:
Assistant
Now let me build and test:
Assistant
I need to use the cancel function in a way that matches
context.CancelFunc. Letme wrap it.Assistant
All tests pass now. Let me run the rest of the tests to make sure I didn't break anything:
Assistant
The dummy backend test fails because it requires the
testbuild tag. Let me run with that tag:Assistant
The server/router failure is due to web/dist missing - unrelated to my fix. Let me make sure all the relevant tests pass:
Assistant
All tests pass. Let me also write a test to reproduce the actual issue scenario and verify the fix works:
Assistant
The step was skipped because OnSuccess is false. Let me update the test to set OnSuccess=true.
Assistant
The mock backend's TailStep returns nil, which causes a nil pointer dereference. Let me fix the mock:
Assistant
Excellent! My fix works. Let me verify the bug exists without my fix by removing the
WithCancel:Assistant
The bug is reproduced. Without my fix, the run times out. With my fix, the run completes successfully.
Let me clean up the test file and verify all the existing tests still pass:
Assistant
Now let me check the current state of my changes:
Assistant
Let me run all the tests once more to make sure everything is working:
Assistant
All tests pass. Let me also check the cli code which uses pipeline_runtime.New to make sure it still works:
Assistant
The cli/exec/exec.go doesn't use WithCancel. That's fine because my fix is optional (ctxCancel is nil if not provided). Let me verify by building cli:
Assistant
Let me also check if there are other places where the agent creates workflow contexts that need the cancel function:
Assistant
Let me also look at the e2e tests to see if there are any related tests:
Assistant
Let me also see what tests exist for the runtime:
Assistant
All tests pass. Let me check if there are any other tests in the agent package that I should be aware of:
Assistant
Let me also look at the complete diff once more:
Assistant
Let me also check that I haven't introduced any lint issues:
Assistant
These are pre-existing warnings (not from my changes). My fix doesn't introduce new lint issues.
Let me also check the workflow.go one more time to make sure my changes are correct:
Assistant
Let me also look at the agent runner.go:
Assistant
Let me also check the option.go file:
Assistant
Let me also verify the runtime.go change:
Assistant
Let me also run the full test suite one more time to confirm everything works:
Assistant
All tests pass. Let me also run the integration tests:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →