ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…
- User
ISSUE DESCRIPTION
Woodpecker services appear to be running on UI & API after pipeline completes
Component
server
Describe the bug
Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.
These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).
Steps to reproduce
- Install woodpecker next-d05c597374
- Use kubernetes backend
- Define pipeline with services running infinitely, like running postgres
services: - name: postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 ports: - 5432 environment: POSTGRES_PASSWORD: somepass POSTGRES_USER: someuser POSTGRES_DB: somedb steps: - name: wait-for-postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 environment: DB_HOST: postgres DB_PORT: 5432 MAX_WAIT: 60 commands: - ./scripts/wait-for-postgres.sh- Run the pipeline, observe pods start for the service and the step
- once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
pr-build > clone (#2): Step: clone Started: 1779195580 Stopped: 1779195594 Type: clone State: success pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: running pr-build > wait-for-postgres (#4): Step: wait-for-postgres Started: 1779195596 Stopped: 1779195602 Type: commands State: successExpected behavior
Pipeline services are stopped once the pipeline steps complete, the pipeline completes.
System Info
next-d05c597374 kubernetes backend, bitbucket datacenter forgeAdditional context
No response
Validations
- Read the docs.
- Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
- Checked that the bug isn't fixed in the
nextversion already [https://woodpecker-ci.org/versions]
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /woodpecker, current file: none) bash-$
- Assistant
I'll start by exploring the repository structure to understand the codebase and locate relevant files related to services and pipeline completion.
- Assistant
Let me look at the issue more carefully. The problem is that services are not being stopped after the pipeline completes. The services are showing as
state=runningin the API even though the pods are destroyed. Let me look at the pipeline execution and service cleanup logic. - Assistant
The issue is that after pipeline steps complete, service steps are not being marked as stopped/completed in the API/UI. The pods are destroyed but the state is still showing as "running".
Let me explore how the workflow/step execution works and how states are updated.
- Assistant
Let me look at the server pipeline code and the RPC service to understand how steps are updated.
- Assistant
Let me look at the tracer.go file, as it seems relevant for updating step states.
- Assistant
Let me look at the pipeline runtime workflow execution:
- Assistant
Let me look at
executeStepto understand how steps are executed and cleaned up. - Assistant
Now let me understand how services are configured. Let me look at the backend types:
- Assistant
Now let me understand the flow better. Let me look at how the compiler sets up services:
- Assistant
Let me look at the compiler to see how service steps are created and whether they're marked as detached.
- Assistant
Now let me look at how
createProcesshandles service steps and detached flag: - Assistant
Let me find the
createProcessmethod. - Assistant
Let me search more broadly for createProcess:
- Assistant
Let me look at how the service steps are compiled. Let me check the
createProcessfunction. - Assistant
Now let me look at the kubernetes backend to understand how services are handled:
- Assistant
Now let me look at the
runDetachedStepfunction and theWaitStepimplementation for kubernetes more carefully: - Assistant
Let me look at the Runtime struct and how the context is used:
- Assistant
Let me check the Runtime struct and its context.
- Assistant
Now let me look at how the runner works:
- Assistant
Let me now look at how service step states are tracked on the server side and how they are updated when the pipeline completes:
- Assistant
Let me look at the workflow status update logic to understand how the overall pipeline status is computed and updated.
- Assistant
Now let me look at how the RPC client handles the Update call:
- Assistant
Now let me look at the pipeline runtime to understand how service steps complete their lifecycle:
- Assistant
Let me look at how services are handled as "detached" steps and how the step state transitions work:
- Assistant
Let me look at the key issue. Let me trace the lifecycle more carefully. Let me look at how
destroyWorkflowFuncinteracts withWaitStep: - Assistant
Now I need to understand the exact issue. Let me look at the
WaitStepimplementation more carefully and trace through what happens when a service step's pod gets destroyed: - Assistant
Now I can see the problem! Let me trace the flow:
- Service steps are compiled as
Detached: truesteps runDetachedStepstarts the step, then spawns a goroutine callingcompleteStep(which callsWaitStep)WaitStepuses a kubernetes informer with only anUpdateFunchandler watching for pod phase changes toSucceeded/Failed/Unknown- When all pipeline stages finish,
destroyWorkflowFunc()callsDestroyWorkflowwhich callsstopPodto delete the pods - When a pod is deleted, the pod's phase doesn't necessarily change to
FailedorSucceeded- it can be deleted while still inRunningphase - The informer only has
UpdateFunc, noDeleteFunc- so when the pod is deleted,finishedchannel never gets closed WaitStepeventually returns whenr.ctxtimes out (pipeline timeout)
Let me verify this hypothesis by looking at
stopPod: - Service steps are compiled as
- Assistant
Let me look at the stopPod function to understand how pods are deleted.
- Assistant
Let me check recent git history for the relevant files to see if there was a regression:
- Assistant
Let me look at the recent "Fix workflow hang on services" commit and the one that moved wait for log uploads:
- Assistant
This is very relevant! Let me look at this commit in detail.
- Assistant
Now I understand the issue! Let me trace through the exact flow:
- Services are
Detached: truesteps, sorunDetachedStepstarts them and spawns a background goroutine callingcompleteStep→WaitStep WaitStepin kubernetes only watches forUpdateFuncevents, checking for pod phasesSucceeded/Failed/Unknown- After all stages complete,
destroyWorkflowFunc()callsDestroyWorkflow→stopPodwhich deletes the pod - When the pod is deleted in kubernetes, it can be removed without going through
Succeeded/Failed/Unknownphase (pod gets deleted from etcd directly) - The informer's
UpdateFunchandler doesn't fire with a terminal phase, and there's noDeleteFunchandler WaitStepblocks until the context times out- The service step is stuck in "running" state until pipeline timeout
Let me verify this by looking at what happens after the pod deletion:
- Services are
- Assistant
Let me search for those functions in the kubernetes package.
- Assistant
Let me check the backend interface to understand what WaitStep should return:
- Assistant
Let me look at the StopTimeout configuration:
- Assistant
Let me also check how the workflow context is set and what triggers its cancellation:
- Assistant
Let me now understand the full picture. Let me look at the workflow runner to understand the timing:
- Assistant
Now I have a clear understanding of the bug. Let me trace through the exact issue:
- Services are
Detached: truesteps →runDetachedStepspawns a goroutine that callscompleteStep→WaitStep WaitStepin kubernetes creates an informer that only handlesUpdateFunc(pod status changes)- After all stages complete,
destroyWorkflowFunc()callsDestroyWorkflow→stopPodwhich deletes the pods - When a pod is deleted, the informer receives a
Deleteevent, NOT anUpdateevent - Since there's no
DeleteFunchandler, thefinishedchannel never gets closed WaitStepblocks untilctx.Done()(pipeline timeout)- The service step remains in "running" state until the pipeline timeout
Let me fix this by adding a
DeleteFunchandler to the informer inWaitStep: - Services are
- Assistant
Now I understand the issue! Let me look at how the service step state flows through the system:
- Assistant
Let me now look at the exact sequence of events to confirm my understanding. Let me also check how the TailStep informer is implemented for comparison:
- Assistant
Now I have a clear understanding of the bug. Let me trace through the exact flow:
- Service steps are
Detached: true, sorunDetachedStepstarts them and spawns a goroutine callingcompleteStep→WaitStep WaitStepin kubernetes creates an informer that only hasUpdateFunchandler- When all stages complete,
destroyWorkflowFunc()callsDestroyWorkflow→stopPodwhich deletes pods - The critical bug: When a pod is deleted from the API server, the informer receives a
Deleteevent, NOT anUpdateevent. There's noDeleteFunchandler, so thefinishedchannel never gets closed WaitStepfalls through toctx.Done()which only fires on pipeline timeout- The service step remains in "running" state until the pipeline timeout
The fix is to add a
DeleteFunchandler to the informer inWaitStep, and handle the case where the pod has already been deleted when we try to fetch its status afterwards.Let me implement the fix:
- Service steps are
- Assistant
Now let me implement the fix. I need to add a
DeleteFunchandler and handle the "pod not found" case: - Assistant
Now let me implement the fix. I need to:
- Add a
DeleteFunchandler to close thefinishedchannel when the pod is deleted - Handle the "not found" case when fetching the pod status after deletion
- Add a
- Assistant
Let me verify the change by viewing the complete function:
- Assistant
Now let me check if the kubernetes API error handling is correct. The "not found" error from kubernetes client-go should be handled properly:
- Assistant
Let me check how the codebase handles "not found" errors from the k8s API:
- Assistant
I need to use the kubernetes API error handling for "not found" errors. Let me update the fix:
- Assistant
Now I need to add the import for
kube_errors. Let me check the current imports: - Assistant
I need to add the
kube_errorsimport fork8s.io/apimachinery/pkg/api/errors. Let me add it. - Assistant
Now let me verify the fix compiles:
- Assistant
Let me check the go environment.
- Assistant
The GOPATH and GOMODCACHE are empty. Let me set them.
- Assistant
Let me set up the environment properly:
- Assistant
Good, the build succeeds! Now let me also run the existing tests to make sure nothing is broken:
- Assistant
Let me see the rest of the test output:
- Assistant
All tests pass. Now let me also check if there are any existing tests that specifically test the WaitStep behavior:
- Assistant
All tests pass. Now let me also run the pipeline runtime tests to make sure nothing is broken there:
- Assistant
Let me check if the tests are in a different package structure:
- Assistant
Let me check why the tests didn't run.
- Assistant
Let me check the package declaration of the test files:
- Assistant
The tests are in the
runtimepackage, not theruntime_testpackage. Let me try running them differently. - Assistant
Let me run the tests from the right directory.
- Assistant
The test files use
package runtime(notpackage runtime_test), andgo testwith a path that doesn't have external test files will show "no test files". Let me check why... - Assistant
The test files are internal (package runtime, not runtime_test), so they should be found by
go test. Let me check if the build tag is the issue. - Assistant
Let me look at the file more carefully:
- Assistant
The test files have
//go:build testbuild tag. Let me run the tests with that tag. - Assistant
The test passes. Let me also run all the pipeline runtime tests to make sure nothing is broken:
- Assistant
All tests pass. Let me verify the changes by reviewing the final state of the file:
- Assistant
Let me now check the git diff to verify the changes:
- Assistant
Let me now review the full file to make sure the import for standard
errorsis still used (and not orphaned): - Assistant
Good, the
errorsimport is still used. Let me also verify there are no unused imports: - Assistant
All clean. Let me also look at the existing WaitStep tests to see if there's a test for pod deletion that I should verify:
- Assistant
The existing tests only test context cancellation, not pod deletion. Let me verify the existing tests still pass:
- Assistant
All tests pass. Let me run a final comprehensive check on all tests in the repository that could be affected:
- Assistant
All tests pass. Let me now verify the complete diff one final time:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-lunaCompare side by side →