Woodpecker services appear to be running on UI & API after pipeline completes Component…
- User
Woodpecker services appear to be running on UI & API after pipeline completes
Component
server
Describe the bug
Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.
These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).
Steps to reproduce
- Install woodpecker next-d05c597374
- Use kubernetes backend
- Define pipeline with services running infinitely, like running postgres
services: - name: postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 ports: - 5432 environment: POSTGRES_PASSWORD: somepass POSTGRES_USER: someuser POSTGRES_DB: somedb steps: - name: wait-for-postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 environment: DB_HOST: postgres DB_PORT: 5432 MAX_WAIT: 60 commands: - ./scripts/wait-for-postgres.sh- Run the pipeline, observe pods start for the service and the step
- once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
pr-build > clone (#2): Step: clone Started: 1779195580 Stopped: 1779195594 Type: clone State: success pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: running pr-build > wait-for-postgres (#4): Step: wait-for-postgres Started: 1779195596 Stopped: 1779195602 Type: commands State: successExpected behavior
Pipeline services are stopped once the pipeline steps complete, the pipeline completes.
System Info
next-d05c597374 kubernetes backend, bitbucket datacenter forgeAdditional context
No response
Validations
- Read the docs.
- Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
- Checked that the bug isn't fixed in the
nextversion already [https://woodpecker-ci.org/versions]
Interface
No new interfaces are introduced.
- Assistant
Investigating how Woodpecker cleans up pipeline services after steps complete — likely a regression in service state handling.
- Assistant
Identifying the issue:
DestroyWorkflowdeletes pods, but KubernetesWaitSteponly unblocks on terminal pod phases—not on deletion—so detached service goroutines hang and the API never gets the completion trace. Implementing a fix. - Assistant
Implementing the fix: Kubernetes
WaitStepdoesn't detect pod deletion, so detached service goroutines hang afterDestroyWorkflowremoves pods. - Assistant
Root Cause
This is a regression from the v3.14 change that waits for detached steps/services to report status before the workflow finishes (#6471, fixed partially in #6507).
The sequence for a long-running service (e.g. Postgres):
- All regular steps finish successfully.
destroyWorkflowFunc()runs and deletes Kubernetes pods viaDestroyWorkflow.- The detached service goroutine is still blocked in
WaitStep, waiting for a terminal pod phase (Succeeded/Failed/Unknown). - Deleted pods never reach those phases, so
WaitStepnever returns. uploadWait.Wait()blocks → agent never callsDone()→ API keeps the service step inrunninguntil pipeline timeout.
The server-side fallback in
completeChildrenIfParentCompletedonly runs whenDone()is called, so it never triggers here.Fix
Updated the Kubernetes backend
WaitStepto:- Listen for pod deletion via the informer's
DeleteFunc. - Treat a missing pod as a successful exit when
GetreturnsNotFound(pod was torn down byDestroyWorkflow).
func (e *kube) WaitStep(ctx context.Context, step *types.Step, taskUUID string) (*types.State, error) { // ... if _, err := si.Core().V1().Pods().Informer().AddEventHandler( cache.ResourceEventHandlerFuncs{ UpdateFunc: podUpdated, DeleteFunc: podDeleted, }, ); err != nil { return nil, err } // ... pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{}) if kube_errors.IsNotFound(err) { return &types.State{ ExitCode: 0, Exited: true, }, nil } // ... }Added
TestWaitStepReturnsOnPodDeletionto cover this path.After this change, when workflow teardown deletes service pods, the agent traces the service as finished, the workflow completes green, and deploys are no longer blocked.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- composer-2.5Compare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- fable-5Compare side by side →
- fable-5Compare side by side →