ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…
- User
ISSUE DESCRIPTION
Woodpecker services appear to be running on UI & API after pipeline completes
Component
server
Describe the bug
Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.
These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).
Steps to reproduce
- Install woodpecker next-d05c597374
- Use kubernetes backend
- Define pipeline with services running infinitely, like running postgres
services: - name: postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 ports: - 5432 environment: POSTGRES_PASSWORD: somepass POSTGRES_USER: someuser POSTGRES_DB: somedb steps: - name: wait-for-postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 environment: DB_HOST: postgres DB_PORT: 5432 MAX_WAIT: 60 commands: - ./scripts/wait-for-postgres.sh- Run the pipeline, observe pods start for the service and the step
- once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
pr-build > clone (#2): Step: clone Started: 1779195580 Stopped: 1779195594 Type: clone State: success pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: running pr-build > wait-for-postgres (#4): Step: wait-for-postgres Started: 1779195596 Stopped: 1779195602 Type: commands State: successExpected behavior
Pipeline services are stopped once the pipeline steps complete, the pipeline completes.
System Info
next-d05c597374 kubernetes backend, bitbucket datacenter forgeAdditional context
No response
Validations
- Read the docs.
- Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
- Checked that the bug isn't fixed in the
nextversion already [https://woodpecker-ci.org/versions]
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /woodpecker, current file: none) bash-$
Assistant
Let me start by understanding the repository structure and the issue. The bug is about Woodpecker CI services not being properly cleaned up - the pods are destroyed on Kubernetes but the API still shows them as "running" state. This seems like a regression in 3.14.x.
Let me explore the repository structure first, understand the codebase, and then find the relevant code that handles service completion/cleanup.
Assistant
Let me explore the relevant directories. I'll start with the pipeline directory and look for code related to service state
Assistant
I need to make only one tool call at a time.
Assistant
Let me look at the pipeline state directory and the backend directory since this is about service state management.
Assistant
Let me look at the backend files and also the kubernetes backend:
Assistant
Let me look at the kubernetes backend:
Assistant
Let me look at the main kubernetes backend file and the pod.go
Assistant
Let me look at the pipeline runtime code - this is likely where the issue is since it manages the pipeline execution
Assistant
Let me look at the runtime directory and the server pipeline code:
Assistant
Let me look at the runtime files
Assistant
Let me look at workflow.go and step.go:
Assistant
Let me look at step.go
Assistant
Now let me look at shutdown.go:
Assistant
Let me look at the server pipeline handling code:
Assistant
Let me search for files that handle pipeline steps and services:
Assistant
Let me look for the main pipeline step handling in the server:
Assistant
Let me also search for where the pipeline execution/running logic is in the server, and where service steps are handled:
Assistant
Let me look at the
Assistant
Let me look at how pipeline steps are managed in server code, especially the pipeline creation and step state updates
Assistant
Let me look at the pipeline step_status.go
Assistant
Let me look at the RPC handler to see how step states are communicated:
Assistant
Now let me look at the agent code to see how it handles service steps:
Assistant
Let me look at the agent runner.go
Assistant
Let me look at the tracer.go and state.go:
Assistant
Let me look at the pipeline frontend yaml to see how services are handled
Assistant
Let me look at the compiler to see how services are compiled into steps
Assistant
Let me look at the compiler.go to understand how services are compiled
Assistant
Let me look at the convert.go to see how createProcess works
Assistant
Let me look at the kubernetes pod.go to understand how pods are stopped
Assistant
Let me look at the git log for recent changes to the pipeline runtime files
Assistant
Let me look at the commit that fixed workflow hang on services
Assistant
Let me look at the full diff of this commit
Assistant
Now I understand the issue. The problem is in the Kubernetes backend's
WaitStepfunction. WhenDestroyWorkflowdeletes the service pod, theWaitStepfunction (which is running in a goroutine for the detached service step) has an informer that only registersUpdateFunc, notDeleteFunc. When a pod is deleted, the delete event is ignored, and thefinishedchannel is never closed. This causesWaitStepto block until the context times out.Let me also check the
TailStepfunction for the same issueAssistant
Let me look at the exact code around both handlers
Assistant
Let me look at the TailStep handler
Assistant
Now I understand the issue. Let me verify my understanding by looking at the full flow once more. The issue is in the Kubernetes backend's
WaitStepfunction. WhenDestroyWorkflowdeletes the service pod, theWaitStepfunction's informer only hasUpdateFuncregistered, notDeleteFunc. When a pod is deleted, the delete event is ignored, and thefinishedchannel is never closed. This causesWaitStepto block until the workflow context times out.But wait, let me also check if there's an issue with the
WaitStepbeing called after the pod is already deleted — the pod might not exist at all when the informer starts.Let me also look at the
TailStepfunction which has the same patternAssistant
Now I have a clear understanding of the bug. Let me implement the fix.
The issue is in the Kubernetes backend's
WaitStepfunction. WhenDestroyWorkflowdeletes the service pod, theWaitStepfunction's pod informer only hasUpdateFuncregistered. When a pod is deleted:- The delete event is sent to the informer
- But since only
UpdateFuncis registered, the delete event is ignored - The
finishedchannel is never closed WaitStepblocks until the workflow context times out
The fix is to add a
DeleteFunchandler in bothWaitStepandTailStepthat closes the signal channels when the pod is deletedAssistant
Let me use the open command correctly.
Assistant
Let me see the full
WaitStepfunctionAssistant
Now I understand the issue fully. Let me implement the fix. The
WaitStepfunction needs to handle the case where the pod is deleted (byDestroyWorkflow) by adding aDeleteFunchandler. I also need to handle the case where the pod no longer exists whenWaitStepstarts.Let me also look at the
TailStepfunction to see if it needs the same fix:Assistant
Now let me implement the fix. I'll edit both
WaitStepandTailStepto addDeleteFunchandlersAssistant
Let me use the replace command instead
Assistant
Let me check the raw content of the file around the area I want to edit
Assistant
I see the file uses tabs. Let me use the edit command with proper tab indentation
Assistant
Good, the
WaitStepfunction is updated. Now let me also update theTailStepfunction. Let me also check iferrorspackage hasIsNotFound:Assistant
Let me check how
errors.IsNotFoundis used elsewhere in the kubernetes backendAssistant
Let me check the imports in pod.go
Assistant
Let me check the imports in kubernetes.go
Assistant
I need to add the k8s errors import to
kubernetes.gosince I'm usingerrors.IsNotFound. But the standard libraryerrorsis already imported. I need to use an aliasAssistant
Now I need to update the reference from
errors.IsNotFoundtokube_errors.IsNotFoundin the WaitStep functionAssistant
Let me navigate to that line first
Assistant
Now I can replace
errors.IsNotFoundwithkube_errors.IsNotFoundAssistant
I need to also handle the case where the pod is already deleted before the informer starts. Let me add a pre-check and also add the
DeleteFunctoTailStep. Let me first add the pre-check inWaitStep:Assistant
Now let me also add the
DeleteFunctoTailStepfor consistency. Let me view the current state of TailStep firstAssistant
Let me also update
TailStepto add aDeleteFunchandlerAssistant
Let me use the heredoc approach
Assistant
Now let me verify the code compiles correctly
Assistant
Let me check the Go environment
Assistant
Let me try setting up the Go module cache
Assistant
Let me set up GOCACHE as well
Assistant
The code compiles. Let me now run the tests for the kubernetes backend and the pipeline runtime
Assistant
Let me continue looking at the test output
Assistant
All kubernetes tests pass. Let me also run the pipeline runtime tests
Assistant
Let me check the test files in the runtime directory
Assistant
Let me try running tests with the correct path
Assistant
Let me check the package declaration in the test files
Assistant
Let me check the package declaration.
Assistant
Let me check if there are build tags
Assistant
Let me run with the test build tag
Assistant
Both tests pass. Let me now review the full changes
Assistant
The diff looks good. Let me run the full runtime test suite to ensure nothing is broken
Assistant
All tests pass. Let me verify the file looks correct in its entirety
Assistant
Let me check if there are other places in the codebase with similar patterns that might need the
Assistant
Good - only the two places I already fixed use
AddEventHandler. Let me run the kubernetes tests one more time to confirm everything passesAssistant
All tests pass. Let me also run the runtime tests once
Assistant
All tests pass. Let me verify the final state of
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-lunaCompare side by side →