ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…
- User
ISSUE DESCRIPTION
Woodpecker services appear to be running on UI & API after pipeline completes
Component
server
Describe the bug
Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.
These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).
Steps to reproduce
- Install woodpecker next-d05c597374
- Use kubernetes backend
- Define pipeline with services running infinitely, like running postgres
services: - name: postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 ports: - 5432 environment: POSTGRES_PASSWORD: somepass POSTGRES_USER: someuser POSTGRES_DB: somedb steps: - name: wait-for-postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 environment: DB_HOST: postgres DB_PORT: 5432 MAX_WAIT: 60 commands: - ./scripts/wait-for-postgres.sh- Run the pipeline, observe pods start for the service and the step
- once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
pr-build > clone (#2): Step: clone Started: 1779195580 Stopped: 1779195594 Type: clone State: success pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: running pr-build > wait-for-postgres (#4): Step: wait-for-postgres Started: 1779195596 Stopped: 1779195602 Type: commands State: successExpected behavior
Pipeline services are stopped once the pipeline steps complete, the pipeline completes.
System Info
next-d05c597374 kubernetes backend, bitbucket datacenter forgeAdditional context
No response
Validations
- Read the docs.
- Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
- Checked that the bug isn't fixed in the
nextversion already [https://woodpecker-ci.org/versions]
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /woodpecker, current file: none) bash-$
- Assistant
I'll start by exploring the repository structure to understand the codebase and locate relevant files for the issue. The bug is about services not being cleaned up after pipeline completes, showing as running in API/UI. This seems to be a server-side issue with the pipeline execution state management.
Let me first get an overview of the repository.
- Assistant
Let me explore the repository structure more broadly.
- Assistant
Let me search for files related to pipeline workflow/step execution and state management.
- Assistant
Let me explore the pipeline and server directories more to understand how workflows and services are managed.
- Assistant
Let me look at the key areas of the code. The issue is about services not being cleaned up after the pipeline completes. The services continue to show as "running" in the API/UI even after the pipeline's steps complete. Let me look at:
- How services/workflows are managed in the pipeline
- The agent's workflow execution logic
- How step completion is tracked and services are stopped
Let me start by looking at the pipeline directory structure and the agent's workflow handling.
- Assistant
Let me explore the runtime directory and look at workflow execution.
- Assistant
Let me look at the workflow.go file to understand how workflows execute services.
- Assistant
Now let's look at step execution to understand how service steps are handled.
- Assistant
Let me examine the tracer implementation to understand how step states are reported.
- Assistant
Let me check the agent-side code to see how the tracer communicates state back to the server.
- Assistant
Now let me understand the execution flow. Let me check how services are structured in the backend types:
- Assistant
Let me look at the pipeline backend types to understand step types:
- Assistant
Let me look at the compiler to understand how services are compiled into stages:
- Assistant
Let me look at how the compiler creates stages from services and steps:
- Assistant
Now let me look at how services are organized into stages and how the workflow stages are set up:
- Assistant
Let me look at the compiler to understand how services are organized into stages:
- Assistant
Let me look at the runtime structure and how context is managed:
- Assistant
Now let me look at the agent runner to understand the full lifecycle:
- Assistant
Now let me look at how the detached steps (services) are handled. I need to understand the lifecycle. Let me check the
WaitStepimplementation for kubernetes backend: - Assistant
Let me understand the flow more clearly. Let me check the server-side RPC handler for step updates:
- Assistant
Let me look at the server-side step status handling:
- Assistant
Let me look at the checkWorkflowAllowsStepUpdate function that sanitizes agent input:
- Assistant
Now let me look at the
Donefunction in detail to see what happens when the workflow completes: - Assistant
Now let me look at the kubernetes backend's WaitStep implementation:
- Assistant
Let me look at how
DestroyWorkflowworks for kubernetes backend: - Assistant
Now let me look at the
stopPodfunction to understand how pods are deleted: - Assistant
Let me search for stopPod function.
- Assistant
Let me look at the stopPod function and how delete options are configured:
- Assistant
Let me look at the default delete options and how they affect the pod lifecycle:
- Assistant
Now I need to understand the key issue. Let me look at the
WaitStepfor kubernetes more carefully: - Assistant
Now let me check how
destroyWorkflowFuncand service cleanup interact. Let me look at the DestroyStep implementation for kubernetes: - Assistant
Let me look more carefully at the
WaitStepfunction - particularly what happens when a pod is deleted (not just updated): - Assistant
Let me look at the default delete options:
- Assistant
Now I understand the issue. Let me trace the flow carefully:
- Service steps run as
DetachedinrunDetachedStep runDetachedStepspawns a background goroutine callingcompleteStep→WaitStep(r.ctx, ...)and returns immediately- After all stages complete,
destroyWorkflowFunc()deletes all pods - But
WaitSteponly registersUpdateFuncon the kubernetes informer, notDeleteFunc - When the pod is deleted, the informer may not see a terminal phase update, so
WaitStepblocks until workflow timeout
The key issue is in
WaitStep— it only handles podUpdateFunc, missingDeleteFunc. When pods are force-deleted or deleted before reaching a terminal state, the service step'sWaitStepnever returns. - Service steps run as
- Assistant
Let me check the StopTimeout configuration:
- Assistant
Now let me look at what the
backend_types.Statelooks like: - Assistant
Let me look at how the Docker backend handles
WaitStepfor comparison: - Assistant
Now I understand the full picture. Let me look at how the stages are organized with services and regular steps:
- Assistant
Now I understand the full picture. Let me trace the execution flow:
- Service stage runs → services start as
Detachedsteps → stage completes immediately (background goroutine for each service keeps runningWaitStep) - Steps stage runs → regular steps complete → stage completes
destroyWorkflowFunc()is called →DestroyWorkflowdeletes ALL pods (including service pods)r.uploadWait.Wait()blocks waiting for service goroutine to finish
The problem: In step 3, when the service pod is deleted, the
WaitStepfunction in the service goroutine only has anUpdateFunchandler on the informer - noDeleteFunc. When a pod is deleted (especially with short/zero grace period), it may be removed from Kubernetes without transitioning through Succeeded/Failed phase. TheUpdateFuncnever fires, soWaitStepblocks until the workflow context timeout.This is exactly the reported behavior: "services are not cleaned properly... only cleaned after the pipeline times out."
The fix: Add a
DeleteFunchandler to the informer inWaitStepto detect pod deletion. - Service stage runs → services start as
- Assistant
Now let me also check how
kubeerrorsis used to check for not-found errors: - Assistant
Let me check the imports in kubernetes.go:
- Assistant
Now let me also check the
errorsimport to make surekubeerrorsis available: - Assistant
Now I understand the full problem. The
WaitStepin the Kubernetes backend only watches forUpdateFuncon the pod informer. When a service pod is deleted bydestroyWorkflowFunc(), the pod may transition to a terminal phase (which triggersUpdateFunc), but there's also a race where the pod could be removed before the terminal state update reaches the informer, or the informer might not fireUpdateFuncwith the right phase. More importantly, whendestroyWorkflowFunc()is called, it deletes the pod. The pod will be deleted with a grace period (StopTimeout), during which SIGTERM is sent to the containers. However, there's a critical path issue:destroyWorkflowFunc()callsstopPod()with the delete options- The pod deletion happens, which eventually causes the container to terminate
WaitStepshould detect the pod reaching a terminal state via the informer- After
finishedis closed,WaitSteptries toGet()the pod - If the pod is already deleted by then, the
Get()call returns a 404 error
The key issue: When
destroyWorkflowFunc()callsDestroyWorkflow, it deletes all pods. The service pod is still running. After deletion, the pod goes through termination. ButWaitSteponly watches forUpdateFunc. When the pod is fully removed (DELETE event), there's noDeleteFunchandler to close thefinishedchannel.Additionally, there's a more subtle issue. When
completeStep()finishes for the service step (afterWaitStepreturns), it callsDestroyStep(). ButdestroyWorkflowFunc()already destroyed the workflow (including all pods). TheDestroyStep()call might fail because the pod is already gone, and the error fromDestroyStep()causescompleteStep()to returnnil, errwithout setting theStartedfield. This meanstraceStepgets called withprocessState=nilanderr != nil, which should still mark the step as exited.But the main issue is that
WaitStepdoesn't handle the pod being deleted. When the pod is deleted and there's noUpdateFunccallback with a terminal state, the informer never closes thefinishedchannel. TheWaitSteponly returns whenctx.Done()is triggered (i.e., context timeout or cancellation).Let me now implement the fix. I need to:
- Add a
DeleteFunchandler to the pod informer that also closes thefinishedchannel - Handle the case where the pod is deleted and
Get()returns a 404 error - instead of returning an error, return a state with the information we have
Actually, let me re-think. The main flow is:
- Services stage runs: service step starts,
runDetachedStepspawns a goroutine that callscompleteStep->WaitStep(r.ctx, step, r.taskUUID) - Steps stage runs: blocking steps complete
destroyWorkflowFunc()is called: deletes all pods (including service pods)r.uploadWait.Wait()waits for the service goroutine
When
destroyWorkflowFunc()deletes the service pod:- The pod gets a deletion timestamp
- SIGTERM is sent to containers
- The container should terminate, transitioning the pod to Succeeded/Failed
WaitStep's informer should see the Update and closefinished
But what if the container doesn't terminate cleanly within the grace period? Then SIGKILL is sent. But the pod should still transition to Failed.
Actually, I think the real issue might be different. Let me look at the timing more carefully.
When
destroyWorkflowFunc()callsstopPod(), it callsDelete()on the pod. The pod will be deleted asynchronously (with grace period). ButdestroyWorkflowFunc()doesn't wait for the pods to actually terminate - it just issues the delete request.Then
r.uploadWait.Wait()waits for the service goroutine. The service goroutine is inWaitStep, waiting for the pod to reach a terminal state.Now, the pod deletion should cause the container to terminate, which should cause the pod to reach a terminal state. But there's a timing issue: what if the pod is fully removed from Kubernetes before the informer sees the terminal state transition? In that case, the
UpdateFuncwould never fire with a terminal state.When a pod is deleted in Kubernetes:
- The pod gets a deletion timestamp (UPDATE event)
- The kubelet sends SIGTERM to containers
- After grace period, SIGKILL is sent
- The pod transitions to terminal state (Succeeded/Failed) (UPDATE event)
- The pod is removed from the API server (DELETE event)
Steps 2-4 happen asynchronously. The informer should see the UPDATE at step 4. But if the pod is removed quickly (e.g., if
StopTimeoutis 0), the pod might go directly from step 1 to step 5 without a terminal state update.Actually, looking at the default delete options:
func (c *config) newDefaultDeleteOptions() kube_meta_v1.DeleteOptions { propagationPolicy := kube_meta_v1.DeletePropagationBackground return kube_meta_v1.DeleteOptions{ GracePeriodSeconds: &c.StopTimeout, PropagationPolicy: &propagationPolicy, } }The
GracePeriodSecondsisc.StopTimeout. If this is 0, the pod is force-deleted immediately, which means the informer might only see a DELETE event, not an UPDATE with terminal state.Even if
StopTimeoutis not 0, there's still a race: the informer might not process the terminal state update before the pod is removed.I think the fix should be:
- Add a
DeleteFunchandler to close thefinishedchannel when the pod is deleted - Handle the case where the pod is deleted and we can't get its state (return a reasonable default state with Exited=true)
Let me implement this now.Now I have a clear understanding of the bug. Let me implement the fix. The issue is in
WaitStep- it only watches forUpdateFuncbut notDeleteFuncon the pod informer. WhendestroyWorkflowFunc()deletes the service pod, the pod might be fully removed from Kubernetes (DELETE event) without the informer seeing a terminal state update, causingWaitStepto block until the context timeout.Additionally, when the pod is deleted and
WaitSteptries toGet()it after detecting the event, it gets a 404 error, which propagates incorrectly.Let me fix both issues:
- Assistant
Now I'll implement the fix. I need to:
- Add a
DeleteFunchandler to the informer that also closes thefinishedchannel - Handle the case where the pod is deleted and
Get()returns a 404 error - instead of propagating the error, return a state indicating the step was exited
- Add a
- Assistant
Now I need to add the
kubeerrorsimport: - Assistant
I see there's a formatting issue in the imports. Let me fix it:
- Assistant
Good. Let me verify the complete
WaitStepfunction looks correct: - Assistant
Now let me verify the code compiles:
- Assistant
Let me try setting up the Go environment properly.
- Assistant
I need to set GOPATH and GOMODCACHE.
- Assistant
I need to set HOME and GOCACHE too.
- Assistant
Good, the code compiles. Let me also run the existing tests:
- Assistant
Let me run all the kubernetes backend tests:
- Assistant
All tests pass. Let me also run the runtime tests:
- Assistant
Let me also run the agent tests:
- Assistant
Let me also run the RPC server tests:
- Assistant
Let me also check the
completeChildrenIfParentCompletedfunction in the server's RPC handler, which is important for understanding how the server handles the workflow completion: - Assistant
Now let me check the existing tests for WaitStep to see if we should add a test for the delete case:
- Assistant
Let me also run a broader set of tests to make sure nothing is broken:
- Assistant
Let me check if the dummy test failure is pre-existing:
- Assistant
Good - the dummy backend build failure is pre-existing (build constraints exclude all Go files). All kubernetes backend tests pass.
Now let me verify the full file looks correct:
- Assistant
The fix looks correct. Let me also run the server pipeline tests to make sure nothing is broken:
- Assistant
Let me verify the complete diff of my changes:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →