ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…
- User
ISSUE DESCRIPTION
Woodpecker services appear to be running on UI & API after pipeline completes
Component
server
Describe the bug
Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.
These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).
Steps to reproduce
- Install woodpecker next-d05c597374
- Use kubernetes backend
- Define pipeline with services running infinitely, like running postgres
services: - name: postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 ports: - 5432 environment: POSTGRES_PASSWORD: somepass POSTGRES_USER: someuser POSTGRES_DB: somedb steps: - name: wait-for-postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 environment: DB_HOST: postgres DB_PORT: 5432 MAX_WAIT: 60 commands: - ./scripts/wait-for-postgres.sh- Run the pipeline, observe pods start for the service and the step
- once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
pr-build > clone (#2): Step: clone Started: 1779195580 Stopped: 1779195594 Type: clone State: success pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: running pr-build > wait-for-postgres (#4): Step: wait-for-postgres Started: 1779195596 Stopped: 1779195602 Type: commands State: successExpected behavior
Pipeline services are stopped once the pipeline steps complete, the pipeline completes.
System Info
next-d05c597374 kubernetes backend, bitbucket datacenter forgeAdditional context
No response
Validations
- Read the docs.
- Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
- Checked that the bug isn't fixed in the
nextversion already [https://woodpecker-ci.org/versions]
Interface
No new interfaces are introduced.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /woodpecker, current file: none) bash-$
Assistant
Let me start by exploring the repository structure to understand the codebase and find the relevant files related to the issue.
The issue describes that Woodpecker services (containers) are not being cleaned up properly after a pipeline completes when using the Kubernetes backend. The pods are destroyed but the API still shows the service as running.
Let me first explore the repository structure.
Assistant
Let me explore the server directory since the issue mentions "server" as a component and the problem is with how services are tracked after pipeline completion.
<tool_call> <function=bash_terminal> <parameter=command> ls -la server/ </tool_call>
Assistant
Let me explore the server directory to understand its structure.
Assistant
The issue is about services not being cleaned up properly after pipeline completion. Let me explore the pipeline directory in server since that's where pipeline execution logic would be.
<tool_call> <function=bash_terminal> <parameter=command> ls -la server/pipeline/ </tool_call>
Assistant
Let me explore the pipeline directory in server to understand how pipelines are executed and how services are handled.
Assistant
Let me look at the cancel.go file since the issue is about services not being stopped properly.
Assistant
Now let me look at the step_status.go file which seems to handle step status updates, including how services should be stopped.
<tool_call> <function=bash_terminal> <parameter=command> cat server/pipeline/step_status.go </tool_call>
Assistant
Let me look at the step_status.go file which should contain the logic for updating step status, including how services should be stopped.
Assistant
Now let me look at the agent's pipeline execution code to see how services are handled. The issue mentions that pods are destroyed properly, but the API still shows the service as running. Let me explore the pipeline directory in the main pipeline folder.
<tool_call> <function=bash_terminal> <parameter=command> ls -la pipeline/ </tool_call>
Assistant
Let me explore the pipeline directory to find where services are handled.
Assistant
Let me explore the pipeline/backend directory, which likely contains the Kubernetes backend code that manages services.
<tool_call> <function=bash_terminal> <parameter=command> ls -la pipeline/backend/ </tool_call>
Assistant
The output says "ERROR: No tool call was invoked" but it also shows the output. Let me try again with a simpler command.
Assistant
Let me explore the kubernetes backend directory to understand how services are cleaned up there.
Assistant
The kubernetes backend has files for managing pods, services, etc. Let me look at the main kubernetes.go file to understand how cleanup is handled when a pipeline completes.
Assistant
Now I see the key code. Looking at
DestroyStepandDestroyWorkflow, they are the methods that destroy pods and other resources. Let me also check the agent code to see how steps report completion and whether service steps are properly handled.Let me look at the agent's pipeline runtime to understand how steps (including services) are tracked and reported.
<tool_call> <function=bash_terminal> <parameter=command> ls -la agent/ </tool_call>
Assistant
The output was shown but the error message was confusing. Let me try running a command with proper structure.
Assistant
Let me look at the agent runner to understand how steps are run and completed:
Assistant
Now let me look at the pipeline runtime to understand how steps are executed, especially services:
<tool_call> <function=bash_terminal> <parameter=command> ls -la pipeline/runtime/ </tool_call>
Assistant
Let me explore the pipeline/runtime directory to understand how the runtime executes steps including services.
Assistant
Let me look at the runtime.go file to understand how the runtime executes steps:
<tool_call> <function=bash_terminal> <parameter=command> cat pipeline/runtime/runtime.go </tool_call>
Assistant
I see the issue - the bash_terminal tool is returning an error message saying "No tool call was invoked" but it's also showing the output. This is strange. Let me try running the command again.
Assistant
Let me look at the step.go file to see how steps are executed, especially services:
Assistant
Now I understand the flow better. Let me look at the workflow.go to see how stages and steps are executed, especially to understand when services are destroyed:
<tool_call> <function=bash_terminal> <parameter=command> cat pipeline/runtime/workflow.go </tool_call>
Assistant
Let me look at the workflow.go to understand how stages and steps are executed:
Assistant
Now I understand the issue better. Looking at the flow:
Run()in workflow.go iterates through all stages- For each stage,
runStage()is called which executes steps in parallel using errgroup - After all stages complete,
DestroyWorkflow()is called at the end of the defer DestroyStep()is called insidecompleteStep()which destroys individual steps (containers/pods)
However, I notice that the issue is specifically about services (which are "detached" steps). Looking at
runDetachedStepin step.go, it starts the step and returns immediately. The problem is that when a non-detached step (likewait-for-postgres) completes and returns, the workflow continues to the next stage, but service steps that are still running may not be properly reported as completed.Let me search for how service steps are handled differently, especially how they get marked as "running" vs "success" after being destroyed.
Let me look at the server-side code that receives step status updates and updates the step state:
<tool_call> <function=bash_terminal> <parameter=command> find server -type f -name "*.go" | xargs grep -l "service" | head -20 </tool_call>
Assistant
The command seems to have some issue. Let me try again with a simpler search command.
Assistant
Now I see that services can also be "detached" (
container.Detached). Let me understand the flow better. Looking at the code:- Services can be either detached or non-detached (blocking)
- For detached steps,
runDetachedStepis called which returns immediately - The problem is that after all main steps complete, the service pods are destroyed, but the step state might not be updated correctly.
Let me look at the step type handling in step.go and understand how service steps are handled:
<tool_call> <function=bash_terminal> <parameter=command> grep -n "StepTypeService" pipeline/frontend/yaml/compiler/convert.go </tool_call>
Assistant
I'm having an issue with the tool calls. Let me check if the tool is actually working correctly by trying a simpler command.
Assistant
The tool is working. Let me see the context around line 95 in the convert.go file:
Assistant
This shows that services are marked as detached. Now let me understand the issue more. Looking at the code flow:
- In
step.go,runDetachedStepis called for services - The service step is started and logs are streamed in the background
- When the workflow context is done,
DestroyWorkflowis called
But the issue is that the service step's trace may not be updated with the final state after the pod is destroyed. Let me look at
traceStepagain and understand what happens:In
traceStep, whenprocessState != nil, it setss.CurrStepState = *processState. This happens incompleteStepafter waiting for the step and destroying it.But wait - in
runDetachedStep, thecompleteStepis called in a goroutine:go func() { defer r.uploadWait.Done() processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime) ... if traceErr := r.traceStep(processState, err, step); traceErr != nil { ... } }()The issue might be timing - when
DestroyWorkflowis called, it might be called before the goroutine completes and traces the final state.Let me check if
uploadWait.Wait()is called beforeDestroyWorkflowreturns:Looking at
workflow.go:for _, stage := range r.spec.Stages { ... } // Now we can shutdown the workflow destroyWorkflowFunc() // Ensure all logs/traces are uploaded before finishing logger.Debug().Msg("waiting for logs and traces upload") r.uploadWait.Wait()So
DestroyWorkflow()is called first, thenuploadWait.Wait()is called. This means the services are destroyed before the goroutines have finished tracing their final state!But wait, the issue says the pods ARE destroyed properly, but the API still shows the service as "running". Let me re-read the issue:
"The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
So the issue is not about the state not being traced, but about the state not being updated to the server correctly. The server receives the state trace but doesn't update it properly.
Let me look at how the server handles step status updates:
<tool_call> <function=bash_terminal> <parameter=command> find server -type f -name "*.go" | xargs grep -l "UpdateStepStatus" </tool_call>
- In
Assistant
Let me try again with the grep command to find files containing UpdateStepStatus.
Assistant
Let me look at the server's RPC handler to see how step status is received:
<tool_call> <function=bash_terminal> <parameter=command> sed -n '200,230p' server/rpc/rpc.go </tool_call>
Assistant
The command is failing to produce output. Let me try a different approach.
Assistant
Now I found the issue! Looking at the
Donemethod inserver/rpc/rpc.go:if !model.IsThereRunningStage(currentPipeline.Workflows) { if currentPipeline, err = pipeline.UpdateStatusToDone(s.store, *currentPipeline, pipeline.PipelineStatus(currentPipeline.Workflows), workflow.Finished); err != nil { logger.Error().Err(err).Msgf("pipeline.UpdateStatusToDone: cannot update workflows final state") } }This only updates the final pipeline status when there's no running stage, but it doesn't ensure that all step statuses are properly updated before the workflow completes.
The problem is:
- When the
wait-for-postgresstep completes, the step status is updated correctly - The service step (
postgres) is still running and hasn't reported completion yet - The pipeline status is checked and updated
- Then
DestroyWorkflowis called which destroys the service pods - But the service step's final state is never reported because the goroutine hasn't finished tracing yet
Looking at
completeStepinpipeline/runtime/step.go:func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) { // Drain the log stream before waiting on the process exit. waitForLogs() waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck ... // Use runnerCtx here: the workflow context may already be canceled but we // still need to reach the backend to stop/remove the container. if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil { return nil, err } ... return waitState, nil }The issue is that
DestroyStepis called BEFORE the final state is traced. After destroying the step, the code should still trace the final state, but there might be an issue with how the state is being reported.Wait, I see that after
DestroyStep, the code continues to check errors and return the state. But then inrunDetachedStep:go func() { defer r.uploadWait.Done() logger := r.makeLogger() processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime) logger.Debug().Str("step", step.Name).Msg("complete") if errors.Is(err, context.Canceled) { err = pipeline_errors.ErrCancel } if err != nil { logger.Error().Err(err).Str("step", step.Name).Msg("detached step failed while running") } if traceErr := r.traceStep(processState, err, step); traceErr != nil { logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result") } }()After
completeStepreturns,traceStepshould be called. But the problem might be that when the pipeline workflow context is done, thetraceStepmight not be able to send the state to the server properly.Let me look more carefully at
traceStepin step.go:func (r *Runtime) traceStep(processState *backend_types.State, err error, step *backend_types.Step) error { s := new(state.State) s.Workflow.Started = r.started s.CurrStep = step s.Workflow.Error = r.err.Get() switch { case processState == nil && err != nil: // Step failed to start — create an dummy exited process state. s.CurrStepState = backend_types.State{ Error: err, Exited: true, OOMKilled: false, } case processState != nil: s.CurrStepState = *processState // processState == nil && err == nil: step just started, leave s.CurrStepState zero-valued. } if traceErr := r.tracer.Trace(s); traceErr != nil { logger := r.makeLogger() logger.Error().Err(traceErr).Msg("could not trace step state change") } return err }The
tracer.Trace(s)is called. The tracer should be the agent's tracer which sends the state to the server via RPC. If this fails, the error is logged but not propagated.The issue might be that when the server's
UpdateRPC handler receives the step update, something goes wrong with updating the service step state.Looking back at
CalcStepStatusinserver/pipeline/step_status.go:func CalcStepStatus(step model.Step, state rpc.StepState) (_ *model.Step, cancelPipelineFromStep bool, _ error) { switch step.State { case model.StatusPending: // Handle skip before anything else ... case model.StatusRunning: // Already running, check if it finished if state.Exited || state.Error != "" { step.Finished = state.Finished ... if state.ExitCode == 0 && state.Error == "" { step.State = model.StatusSuccess } else { step.State = model.StatusFailure ... } } default: return nil, false, fmt.Errorf("step has state %s and does not expect rpc state updates", step.State) } ... }This looks correct - if the step is running and the state says it exited, it should update to success or failure. But wait, there's a check:
if state.Exited || state.Error != "" {This means if
state.Exitedis false ANDstate.Erroris empty, the step state won't be updated even if the container was destroyed!Now let me look at what state the agent sends. Looking at
completeStep:bs := &backend_types.State{ ExitCode: int(cs.State.Terminated.ExitCode), Exited: true, OOMKilled: false, }The state has
Exited: true, so this should work...Wait, I need to re-read the issue more carefully. The issue says:
"The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
So the pods ARE destroyed but the step state is still "running". This means the agent isn't reporting the final state, or the server isn't updating it correctly.
Let me look at the
WaitStepmethod in the kubernetes backend:func (e *kube) WaitStep(ctx context.Context, step *types.Step, taskUUID string) (*types.State, error) { podName, err := stepToPodName(step) ... select { case <-finished: case <-ctx.Done(): return nil, ctx.Err() } pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{}) if err != nil { return nil, err } // ... checks for image pull issues ... if len(pod.Status.ContainerStatuses) == 0 { return nil, fmt.Errorf("no container statuses found for pod %s", podName) } cs := pod.Status.ContainerStatuses[0] if cs.State.Terminated == nil { err := fmt.Errorf("no terminated state found for container %s/%s", podName, cs.Name) log.Error().Str("taskUUID", taskUUID).Str("pod", podName).Str("container", cs.Name).Interface("state", cs.State).Msg(err.Error()) return nil, err } ... }This looks correct - it waits for the pod to terminate and returns the exit code.
But wait, I think I see the issue now! Let me look at the
DestroyStepmethod:func (e *kube) DestroyStep(ctx context.Context, step *types.Step, taskUUID string) error { var errs []error if needsRegistrySecret(step) { err := stopRegistrySecret(ctx, e, step, e.config.newDefaultDeleteOptions()) if err != nil { errs = append(errs, err) } } if needsStepSecret(step) { err := stopStepSecret(ctx, e, step, e.config.newDefaultDeleteOptions()) if err != nil { errs = append(errs, err) } } err := stopPod(ctx, e, step, e.config.newDefaultDeleteOptions()) if err != nil { errs = append(errs, err) } return errors.Join(errs...) }This calls
stopPodto destroy the pod. Let me check whatstopPoddoes:<tool_call> <function=bash_terminal> <parameter=command> grep -n "func stopPod" pipeline/backend/kubernetes/pod.go </tool_call>
- When the
Assistant
Let me search for the stopPod function:
Assistant
Let me look at the stopPod function and understand what's happening:
<tool_call> <function=bash_terminal> <parameter=command> sed -n '736,780p' pipeline/backend/kubernetes/pod.go </tool_call>
Assistant
The tool is having issues. Let me try reading the file directly with cat.
Assistant
The
stopPodfunction just deletes the pod. The issue might be a timing problem - ifDestroyStepis called incompleteStep, andDestroyStepcallsstopPod, the pod is deleted. But what happens to theWaitStepthat was called earlier?Looking at
completeStep:waitForLogs() waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) ... if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil { return nil, err }The order is:
- Wait for logs to drain
- Wait for the step to complete (WaitStep)
- Destroy the step (DestroyStep)
So
WaitStepshould complete beforeDestroyStepis called. But wait - the issue says the pods ARE destroyed properly. This meansDestroyStepis being called. But the state is not being updated.Let me check the kubernetes
WaitStepfunction again more carefully. It waits for the pod to reach a terminal state:func (e *kube) WaitStep(ctx context.Context, step *types.Step, taskUUID string) (*types.State, error) { ... finished := make(chan struct{}) var finishedOnce sync.Once podUpdated := func(_, newPod any) { pod, ok := newPod.(*kube_core_v1.Pod) if !ok { log.Error().Msgf("could not parse pod: %v", newPod) return } if pod.Name == podName { if isImagePullBackOffState(pod) || isInvalidImageName(pod) { finishedOnce.Do(func() { close(finished) }) } switch pod.Status.Phase { case kube_core_v1.PodSucceeded, kube_core_v1.PodFailed, kube_core_v1.PodUnknown: finishedOnce.Do(func() { close(finished) }) } } } ... }It waits for the pod to reach
Succeeded,Failed, orUnknownphase. If the pod is deleted byDestroyStepbefore this wait completes, the informer might not see the final state.But wait - the issue says the wait-for-postgres step completes successfully, but the postgres service step shows as "running" with
Stopped: 0. This means the service step's state was never updated to "success" or "failure".Let me re-read the issue output:
pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: runningThe
Stopped: 0means the Finished time is 0, which meansstep.Finishedwas never set. Looking atCalcStepStatus:case model.StatusRunning: // Already running, check if it finished if state.Exited || state.Error != "" { step.Finished = state.Finished if step.Finished == 0 { step.Finished = time.Now().Unix() } ... }So if
state.Exitedis false andstate.Erroris empty, the step state is never updated. This means the agent is not sending the correct state.Looking at
completeStepin step.go:waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID)If
WaitStepfails (e.g., returns an error because the pod was deleted), the function returns early without callingDestroyStep. But the issue says the pods ARE destroyed, soDestroyStepis being called.Wait - let me look at the order again:
func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) { // Drain the log stream before waiting on the process exit. waitForLogs() waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck if err != nil { if errors.Is(err, context.Canceled) { if waitState == nil { waitState = &backend_types.State{} } waitState.Error = pipeline_errors.ErrCancel } else { return nil, err } } // Use runnerCtx here: the workflow context may already be canceled but we // still need to reach the backend to stop/remove the container. if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil { return nil, err } ... }If
WaitStepreturnscontext.Canceled, the code setswaitState.Errorbut doesn't setwaitState.Exited. ThenDestroyStepis called. After that:waitState.Started = startTime // Re-check context cancellation: the wait may have raced with cancellation. if ctxErr := r.ctx.Err(); ctxErr != nil && errors.Is(ctxErr, context.Canceled) { waitState.Error = pipeline_errors.ErrCancel }The
waitState.Exitedis not set in this case!But wait - if the pod was destroyed by
DestroyStep,WaitStepshould fail with some error, not necessarilycontext.Canceled.Let me check what
WaitStepreturns when the pod is already deleted:pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{}) if err != nil { return nil, err }If the pod is deleted,
Getwill return a "not found" error. But the code checks for this case?No, it doesn't check for "not found" error. So
WaitStepwill return an error like "pod not found", andcompleteStepwill return that error without settingwaitState.Exited.Wait, but the code flow is:
WaitStepis called - this waits for the pod to reach a terminal state- If the context is canceled,
WaitStepreturns withcontext.Canceled DestroyStepis called - this deletes the pod
But if the pod is deleted BEFORE
WaitStepcompletes (e.g., by some other mechanism), thenWaitStepwill return an error.Actually, I think I finally understand the issue! Let me trace through the execution:
- The
postgresservice step is started - The
wait-for-postgresstep completes - The
wait-for-postgresstep's goroutine inrunDetachedStepfinishes - The main loop continues to the next stage (there's no next stage in this case)
- After all stages complete,
DestroyWorkflowis called DestroyWorkflowcallsDestroyStepfor ALL steps, including the service
But the problem is:
DestroyStepis called BEFORE the service step's goroutine has finished waiting for the pod and reporting the final state!Looking at
workflow.go:for _, stage := range r.spec.Stages { ... } // Now we can shutdown the workflow destroyWorkflowFunc() // Ensure all logs/traces are uploaded before finishing logger.Debug().Msg("waiting for logs and traces upload") r.uploadWait.Wait()So
DestroyWorkflow()is called BEFOREuploadWait.Wait(). This means the service pods are destroyed BEFORE the service step's goroutine has a chance to:- Wait for the pod to complete
- Destroy the step (which is already done by DestroyWorkflow)
- Trace the final state
But wait -
DestroyWorkflowcallsDestroyStepfor all steps. So when the service step's goroutine eventually callsDestroyStep, the pod is already gone. And whenWaitStepis called, it might get an error because the pod is already deleted.Let me check the
completeStepfunction again:waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) if err != nil { if errors.Is(err, context.Canceled) { if waitState == nil { waitState = &backend_types.State{} } waitState.Error = pipeline_errors.ErrCancel } else { return nil, err // <- Returns without setting Exited! } }If
WaitStepreturns an error that's NOTcontext.Canceled(e.g., "pod not found"), the function returns immediately without settingwaitState.Exited. Then the error is traced, but the state doesn't haveExited: true, so the server doesn't update the step state.Actually wait, let me re-read
completeStep:func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) { // Drain the log stream before waiting on the process exit. waitForLogs() waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck if err != nil { if errors.Is(err, context.Canceled) { if waitState == nil { waitState = &backend_types.State{} } waitState.Error = pipeline_errors.ErrCancel } else { return nil, err // Returns error and nil state } } // Use runnerCtx here: the workflow context may already be canceled but we // still need to reach the backend to stop/remove the container. if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil { return nil, err } ... }If
WaitStepreturns an error (not context.Canceled), the function returnsnil, err. This meansprocessStateinrunDetachedStepisnil, andtraceStepis called withprocessState == nil.Looking at
traceStep:switch { case processState == nil && err != nil: // Step failed to start — create an dummy exited process state. s.CurrStepState = backend_types.State{ Error: err, Exited: true, OOMKilled: false, } case processState != nil: s.CurrStepState = *processState ... }If
processState == nilanderr != nil, a dummy state withExited: trueis created. So the server should receiveExited: trueeven ifWaitStepfails.But the issue says the pods ARE destroyed. This means
DestroyStepis being called, which meansWaitStepmust have succeeded. So the flow is:WaitStepsucceeds (pod reaches terminal state)DestroyStepis called (pod is deleted)- The state is traced with
Exited: true
But the issue says the server still shows the service as "running". This means either:
- The server didn't receive the state update
- The server received it but didn't update the step state
Let me check the server's
Updatemethod again. Looking at the code around line 212:if err := pipeline.UpdateStepStatus(c, s.store, step, state); err != nil { log.Error().Err(err).Msg("rpc.update: cannot update step") }If
UpdateStepStatusfails, the error is logged. But maybe the error is not being logged or is being swallowed?Actually, I just realized something. Let me look at the
DestroyWorkflowmethod inkubernetes.go:func (e *kube) DestroyWorkflow(ctx context.Context, conf *types.Config, taskUUID string) error { log.Trace().Str("taskUUID", taskUUID).Msg("deleting Kubernetes primitives") for _, stage := range conf.Stages { for _, step := range stage.Steps { err := stopPod(ctx, e, step, e.config.newDefaultDeleteOptions()) if err != nil { return err } } } ... }This iterates over ALL steps and destroys them, including services. This is called in
DestroyWorkflowwhich is called BEFOREuploadWait.Wait()inworkflow.go.So the problem is:
- The main pipeline steps (like
wait-for-postgres) complete DestroyWorkflowis called, which destroys ALL pods, including the service pods- The service step's goroutine (running
runDetachedStep) is still waiting for the pod to complete - When
WaitStepis called, the pod is already deleted, so it returns an error - The error is returned from
completeStep, but it's notcontext.Canceled, soprocessStateisnil traceStepis called withprocessState == nilanderr != nil, which creates a dummy state withExited: true- But wait - this should still result in the state being updated!
Actually, let me re-read the flow more carefully. In
runDetachedStep:func (r *Runtime) runDetachedStep(runnerCtx context.Context, step *backend_types.Step) error { waitForLogs, startTime, err := r.startStep(step) if err != nil { return r.traceStep(nil, err, step) } // Container is up and logging is streaming — hand off to background. r.uploadWait.Add(1) go func() { defer r.uploadWait.Done() logger := r.makeLogger() processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime) logger.Debug().Str("step", step.Name).Msg("complete") if errors.Is(err, context.Canceled) { err = pipeline_errors.ErrCancel } if err != nil { logger.Error().Err(err).Str("step", step.Name).Msg("detached step failed while running") } if traceErr := r.traceStep(processState, err, step); traceErr != nil { logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result") } }() return nil }The goroutine runs
completeStep, then callstraceStep. But ifDestroyStepis called byDestroyWorkflowbefore the goroutine finishes,WaitStepwill fail.But the goroutine should still call
traceStepwith the error state. Let me check what happens intraceStepwhenprocessState == nilanderr != nil:case processState == nil && err != nil: // Step failed to start — create an dummy exited process state. s.CurrStepState = backend_types.State{ Error: err, Exited: true, OOMKilled: false, }Wait, this says "Step failed to start" but it's also used when the step fails during execution. This might be the issue - the error message is misleading, but the code should still create a state with
Exited: true.Actually, I think I found the issue! Let me look at
completeStepagain:func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) { waitForLogs() waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck if err != nil { if errors.Is(err, context.Canceled) { if waitState == nil { waitState = &backend_types.State{} } waitState.Error = pipeline_errors.ErrCancel } else { return nil, err } } if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil { return nil, err } waitState.Started = startTime if ctxErr := r.ctx.Err(); ctxErr != nil && errors.Is(ctxErr, context.Canceled) { waitState.Error = pipeline_errors.ErrCancel } ... }If
WaitStepsucceeds,waitStateis not nil and hasExited: true. ThenDestroyStepis called. AfterDestroyStep, the code continues to setwaitState.Startedand check for context cancellation.But what if
WaitStepsucceeds withwaitState.Exited = true, but thenDestroyStepis called byDestroyWorkflowbefore the goroutine finishes? TheDestroyStepcall incompleteStepmight fail because the pod is already deleted, but the error is returned without updating the state.Actually, I just realized the order of operations in
completeStep:WaitStepis called - waits for pod to completeDestroyStepis called - destroys the pod
But if
DestroyWorkflowis called before step 1 completes, the pod will be deleted byDestroyWorkflow. Then whenWaitStepis called in step 1, it will get an error (pod not found), andcompleteStepwill return an error.Wait, but the issue says the pods ARE destroyed properly. This implies that
DestroyWorkflowis being called. But if the pods are destroyed byDestroyWorkflow, thenWaitStepshould fail.Let me re-read the issue more carefully:
"The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
So the pods ARE destroyed, but the API shows the service as "running". This means:
- The pod was destroyed (either by
DestroySteporDestroyWorkflow) - But the step state was not updated to "success" or "failure"
This could happen if:
WaitStepis called but returns an error (pod not found)- The error is returned from
completeStep traceStepis called withprocessState == nilanderr != nil- A dummy state with
Exited: trueis created - The server receives the state update
- But the server doesn't update the step state correctly
Actually, wait. Let me check the server's
UpdateStepStatusfunction again:func UpdateStepStatus(ctx context.Context, store store.Store, step *model.Step, state rpc.StepState) error { updatedStep, shouldCancelPipelineFromStep, err := CalcStepStatus(*step, state) if err != nil { return err } *step = *updatedStep if shouldCancelPipelineFromStep { if err := cancelPipelineFromStep(ctx, store, step); err != nil { return err } } return store.StepUpdate(step) }And
CalcStepStatus:func CalcStepStatus(step model.Step, state rpc.StepState) (_ *model.Step, cancelPipelineFromStep bool, _ error) { switch step.State { case model.StatusPending: ... case model.StatusRunning: if state.Exited || state.Error != "" { step.Finished = state.Finished if step.Finished == 0 { step.Finished = time.Now().Unix() } step.ExitCode = state.ExitCode step.Error = state.Error if state.ExitCode == 0 && state.Error == "" { step.State = model.StatusSuccess } else { step.State = model.StatusFailure ... } } default: return nil, false, fmt.Errorf("step has state %s and does not expect rpc state updates", step.State) } ... }The key check is:
if state.Exited || state.Error != "". If both are false, the step state is not updated.So if the agent sends a state with
Exited: falseandError: "", the server won't update the step state. This would happen ifWaitStepreturns without an error but also doesn't setExited: true.Let me check what
WaitStepreturns when the pod is in a terminal state:cs := pod.Status.ContainerStatuses[0] if cs.State.Terminated == nil { err := fmt.Errorf("no terminated state found for container %s/%s", podName, cs.Name) log.Error().Str("taskUUID", taskUUID).Str("pod", podName).Str("container", cs.Name).Interface("state", cs.State).Msg(err.Error()) return nil, err } bs := &types.State{ ExitCode: int(cs.State.Terminated.ExitCode), Exited: true, OOMKilled: false, } return bs, nilThis looks correct - if the container is terminated,
Exited: trueis set.So the issue must be that the agent is sending a state with
Exited: falseandError: "". Let me check what state the agent sends intraceStep:case processState == nil && err != nil: s.CurrStepState = backend_types.State{ Error: err, Exited: true, OOMKilled: false, } case processState != nil: s.CurrStepState = *processState ...If
processState != nilandprocessState.Exited == true, the agent should send a state withExited: true. But what ifprocessState.Exitedis not set?Let me check the flow again. In
completeStep:waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) if err != nil { if errors.Is(err, context.Canceled) { if waitState == nil { waitState = &backend_types.State{} } waitState.Error = pipeline_errors.ErrCancel } else { return nil, err } } if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil { return nil, err } waitState.Started = startTime if ctxErr := r.ctx.Err(); ctxErr != nil && errors.Is(ctxErr, context.Canceled) { waitState.Error = pipeline_errors.ErrCancel }If
WaitStepreturnscontext.CanceledwithwaitState == nil, a new state is created butExitedis not set! ThenDestroyStepis called, and the function returnswaitState(which hasExited: false).But wait,
waitStateis created as&backend_types.State{}, which meansExitedis the zero valuefalse. ThenwaitState.Erroris set, butwaitState.Exitedis never set totrue.Then in
traceStep:case processState != nil: s.CurrStepState = *processStateThe state is copied as-is, with
Exited: false. The server receives this and seesExited: falseandError: pipeline_errors.ErrCancel, so the checkif state.Exited || state.Error != ""is true, and the step state should be updated.Hmm, but the issue says the service is shown as "running" with
Stopped: 0. This meansstep.Finishedis 0, which means the state was never updated.Wait, let me re-read the issue output:
pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: runningThe
Startedtime is set, butStoppedis 0. This means the step started but never finished. And the state is "running".This could happen if:
- The agent sent a state with
Startedtime set - But never sent a state with
Exited: true
Or:
- The agent sent a state with
Exited: trueandFinishedtime - But the server didn't update the step correctly
Let me check what state the agent sends when the step starts:
In
step.go,executeStep:if err := r.traceStep(nil, nil, step); err != nil { return err }This calls
traceStepwithprocessState == nilanderr == nil. Looking attraceStep:switch { case processState == nil && err != nil: ... case processState != nil: s.CurrStepState = *processState // processState == nil && err == nil: step just started, leave s.CurrStepState zero-valued. }When
processState == nilanderr == nil,s.CurrStepStateis left as the zero value. Buts.CurrStepis set to the step.So the tracer sends:
s.Workflow.Started = r.starteds.CurrStep = step(which hasStartedandFinishedtimes)s.CurrStepStateis zero-valued
The server receives this and updates the step's
Startedtime and sets the state to "running".But the issue is that the step's
Finishedtime is never updated. This means the agent never sends a state withExited: true.Wait, I think I understand the issue now! The problem is that
DestroyStepis called BEFOREWaitStepcompletes in the goroutine. So the flow is:- Main loop finishes all stages
DestroyWorkflowis called, which destroys ALL pods (including services)- The service step's goroutine is still waiting for
WaitStepto complete WaitStepreturns an error because the pod was deletedcompleteStepreturns with an errortraceStepis called withprocessState == nilanderr != nil- A dummy state with
Exited: trueis created and sent to the server
But wait, if the dummy state is sent with
Exited: true, the server should update the step state. Let me check the server'sUpdateStepStatusfunction again:func UpdateStepStatus(ctx context.Context, store store.Store, step *model.Step, state rpc.StepState) error { updatedStep, shouldCancelPipelineFromStep, err := CalcStepStatus(*step, state) if err != nil { return err } ... }And
CalcStepStatus:case model.StatusRunning: if state.Exited || state.Error != "" { step.Finished = state.Finished ... }If
state.Error != "", the step state should be updated. But the issue says the step is still "running".Hmm, maybe the issue is that the server doesn't receive the state update at all. Let me check the
Donemethod inrpc.go:func (s *RPC) Done(c context.Context, strWorkflowID string, state rpc.WorkflowState) error { ... // Check if all steps are done if !model.IsThereRunningStage(currentPipeline.Workflows) { if currentPipeline, err = pipeline.UpdateStatusToDone(s.store, *currentPipeline, pipeline.PipelineStatus(currentPipeline.Workflows), workflow.Finished); err != nil { logger.Error().Err(err).Msgf("pipeline.UpdateStatusToDone: cannot update workflows final state") } } ... }The
Donemethod is called when the workflow is complete. At this point, it checks if all stages are done. But if a service step is still "running", the condition!model.IsThereRunningStage(currentPipeline.Workflows)will be false, and the pipeline won't be marked as done.Let me check
IsThereRunningStage:<tool_call> <function=bash_terminal> <parameter=command> grep -n "IsThereRunningStage" server/model/ </tool_call>
Assistant
The command failed to produce output. Let me try again with grep.
Assistant
Let me look at this function:
<tool_call> <function=bash_terminal> <parameter=command> sed -n '50,65p' server/model/workflow.go </tool_call>
Assistant
The command is failing to produce output. Let me try a different approach - using cat to read the file.
Assistant
Now I understand the issue better.
IsThereRunningStagechecks if any workflow (stage) is running or pending. But the issue is about steps, not workflows.Let me trace through the flow again:
- The
wait-for-postgresstep completes successfully - The main loop in
workflow.gocontinues - After all stages complete,
DestroyWorkflowis called DestroyWorkflowdestroys ALL pods, including services- The service step's goroutine is still waiting for
WaitStepto complete WaitStepreturns an error because the pod was deleted- The error is returned from
completeStep traceStepis called withprocessState == nilanderr != nil- A dummy state with
Exited: trueis created and sent to the server - The server updates the step state to "failure" (because there's an error)
But wait - the issue says the service is shown as "running". This means the server didn't receive the state update, or the update was ignored.
Actually, I think I finally understand the issue! Let me check the order of operations in
runDetachedStepmore carefully:go func() { defer r.uploadWait.Done() processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime) ... if traceErr := r.traceStep(processState, err, step); traceErr != nil { logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result") } }()The goroutine calls
traceStep, which callsr.tracer.Trace(s). The tracer sends the state to the server via RPC. But what if the server closes the RPC connection before the tracer finishes sending the state?Actually, I think the issue might be with how
DestroyWorkflowis called. Looking atworkflow.go:for _, stage := range r.spec.Stages { stageChan := r.runStage(runnerCtx, stage.Steps) select { case <-r.ctx.Done(): <-stageChan return pipeline_errors.ErrCancel case err := <-stageChan: if err != nil { r.err.Set(err) } } } // Now we can shutdown the workflow destroyWorkflowFunc()The
destroyWorkflowFunc()is called AFTER all stages complete. But at this point, the service step's goroutine might still be running. The goroutine will calltraceStepeventually, but by the time it does, the server might have already marked the pipeline as done.Actually, I think I found the real issue! Let me look at the
traceStepfunction again:func (r *Runtime) traceStep(processState *backend_types.State, err error, step *backend_types.Step) error { s := new(state.State) s.Workflow.Started = r.started s.CurrStep = step s.Workflow.Error = r.err.Get() switch { case processState == nil && err != nil: s.CurrStepState = backend_types.State{ Error: err, Exited: true, OOMKilled: false, } case processState != nil: s.CurrStepState = *processState ... } if traceErr := r.tracer.Trace(s); traceErr != nil { logger := r.makeLogger() logger.Error().Err(traceErr).Msg("could not trace step state change") } return err }The tracer is
r.tracer, which is set viaWithTraceroption. Let me check what tracer is used:Looking at
agent/runner.go:}).Run(runnerCtx)And
pipeline_runtime.WithTracer(r.createTracer(ctxMeta, logger, workflow)).The tracer sends the state to the server via RPC. But what if the RPC connection is closed before the tracer finishes sending the state?
Actually, I think the issue might be simpler. Let me check what happens when
WaitStepis called after the pod has been deleted. TheWaitStepfunction uses an informer to watch for pod updates:func (e *kube) WaitStep(ctx context.Context, step *types.Step, taskUUID string) (*types.State, error) { podName, err := stepToPodName(step) ... finished := make(chan struct{}) var finishedOnce sync.Once podUpdated := func(_, newPod any) { pod, ok := newPod.(*kube_core_v1.Pod) ... switch pod.Status.Phase { case kube_core_v1.PodSucceeded, kube_core_v1.PodFailed, kube_core_v1.PodUnknown: finishedOnce.Do(func() { close(finished) }) } } ... select { case <-finished: case <-ctx.Done(): return nil, ctx.Err() } ... }The informer is watching for pod updates. If the pod is deleted by
DestroyWorkflow, the informer might receive a "delete" event. But thepodUpdatedfunction only handles "update" events. So if the pod is deleted, thefinishedchannel will never be closed, andWaitStepwill wait until the context is canceled.But the context for
WaitStepisr.ctx, which is the workflow context. The workflow context is canceled when the workflow is done. SoWaitStepwill return withcontext.Canceled.Looking at
completeStep:waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) if err != nil { if errors.Is(err, context.Canceled) { if waitState == nil { waitState = &backend_types.State{} } waitState.Error = pipeline_errors.ErrCancel } else { return nil, err } }If
WaitStepreturnscontext.Canceled, a new state is created withwaitState.Error = pipeline_errors.ErrCancel. ButwaitState.Exitedis not set! It's the zero valuefalse.Then
DestroyStepis called. The pod might already be deleted byDestroyWorkflow, but thestopPodfunction handles "not found" errors:err = engine.client.CoreV1().Pods(engine.config.GetNamespace(step.OrgID)).Delete(ctx, podName, deleteOpts) if errors.IsNotFound(err) { return nil } return errSo
DestroyStepreturns nil even if the pod is already deleted.Then
waitState.Started = startTimeis set.Then:
if ctxErr := r.ctx.Err(); ctxErr != nil && errors.Is(ctxErr, context.Canceled) { waitState.Error = pipeline_errors.ErrCancel }The
waitState.Erroris set topipeline_errors.ErrCancel.Then the function returns
waitState, nil.So the state returned has:
waitState.Started = startTimewaitState.Error = pipeline_errors.ErrCancelwaitState.Exited = false(zero value)
Then in
traceStep:case processState != nil: s.CurrStepState = *processStateThe state is copied as-is, with
Exited: falseandError: pipeline_errors.ErrCancel.The server receives this state and checks:
if state.Exited || state.Error != "" { step.Finished = state.Finished ... }Since
state.Error != "", the step state should be updated. But the issue says the step is still "running".Wait, I just realized something. The
WaitStepfunction might not returncontext.Canceled. It might return an error from the informer or theGetcall.Let me check the
WaitStepfunction again. After theselectstatement:select { case <-finished: case <-ctx.Done(): return nil, ctx.Err() } pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{})If the
finishedchannel is closed (pod reached terminal state), the code continues to get the pod. If the pod is deleted after the informer sees the terminal state but before theGetcall, theGetwill return an error.But the informer only sees update events, not delete events. So if the pod is deleted, the informer won't see it, and the
finishedchannel won't be closed.WaitStepwill wait until the context is canceled.Hmm, but the issue says the pods ARE destroyed properly. This means
DestroyWorkflowis being called. But the service step's goroutine might not have finished yet.Actually, I think the issue might be a race condition. Let me trace through the flow more carefully:
- The
wait-for-postgresstep completes successfully - The main loop in
workflow.gocontinues to the next iteration - After all stages complete,
destroyWorkflowFunc()is called destroyWorkflowFunc()callse.engine.DestroyWorkflow(ctx, r.spec, r.taskUUID)DestroyWorkflowiterates over all steps and callsstopPodfor each- The service step's goroutine is still waiting in
WaitStep WaitStepis waiting for the informer to see a terminal state- The pod is deleted by
DestroyWorkflow - The informer might see the delete event, but the
podUpdatedfunction only handles update events - The
finishedchannel is never closed WaitStepwaits until the context is canceled- The context is canceled (by
destroyWorkflowFunc()or by the workflow completion) WaitStepreturnscontext.CanceledcompleteStepcreates a new state withError = pipeline_errors.ErrCanceltraceStepis called with the state- The server receives the state and updates the step
But wait - if
WaitStepreturnscontext.Canceled, the context is canceled. But the goroutine is still running. ThetraceStepshould be called aftercompleteStepreturns.Actually, I think I might be overcomplicating this. Let me re-read the issue:
"The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
The pods ARE destroyed, but the API shows the service as "running". This means:
- The pod was destroyed (confirmed)
- The step state is "running" (the issue)
This could happen if:
- The agent never sent a state update after the pod was destroyed
- The agent sent a state update, but the server didn't process it
- The agent sent a state update, but the server updated the wrong step
Let me check if there's any issue with how the step UUID is matched. In the server's
Updatemethod:step, err := s.store.StepByUUID(state.StepUUID) if err != nil { log.Error().Err(err).Msgf("cannot find step with uuid %s", state.StepUUID) return err }If the step is not found, the error is logged and returned. But what if the step was already updated by a previous trace?
Actually, I think the issue might be with the order of operations. Let me check the flow again:
- The
wait-for-postgresstep completes - The main loop finishes
destroyWorkflowFunc()is calledDestroyWorkflowdestroys all pods- The service step's goroutine calls
WaitStep WaitStepreturnscontext.Canceledbecause the pod was deletedcompleteStepcreates a state withError = pipeline_errors.ErrCanceltraceStepis called with the state- The server receives the state and updates the step to "failure"
But the issue says the step is "running", not "failure". This means the server didn't receive the state update, or the update was ignored.
Wait, I just realized something. The
Donemethod inrpc.gois called when the workflow is complete:func (s *RPC) Done(c context.Context, strWorkflowID string, state rpc.WorkflowState) error { ... if !model.IsThereRunningStage(currentPipeline.Workflows) { if currentPipeline, err = pipeline.UpdateStatusToDone(s.store, *currentPipeline, pipeline.PipelineStatus(currentPipeline.Workflows), workflow.Finished); err != nil { logger.Error().Err(err).Msgf("pipeline.UpdateStatusToDone: cannot update workflows final state") } } ... }The
Donemethod is called by the agent after all steps complete. At this point, the service step's goroutine might not have finished yet. TheDonemethod checks if all stages are done, but the service step is still "running".Actually,
IsThereRunningStagechecks workflows (stages), not steps. So even if a step is "running", the workflow might be "success" or "failure".Wait, but the issue says the service step is "running" with
Stopped: 0. This means the step'sFinishedtime is 0, which means the step was never updated to "success" or "failure".Let me think about this differently. The issue might be that the service step's goroutine is blocked in
WaitStepwhenDestroyWorkflowis called. TheDestroyWorkflowfunction destroys the pod, but the goroutine is still waiting. The goroutine will eventually return fromWaitStep(when the context is canceled), but by that time, the pipeline might have already been marked as done.Actually, I think I need to look at the actual timing. Let me check when
Doneis called:In
agent/runner.go:err = pipeline_runtime.New(...).Run(runnerCtx) state.Finished = time.Now().Unix() ... if err := r.client.Done(doneCtx, workflow.ID, state); err != nil { logger.Error().Err(err).Msg("failed to update workflow status") }The
Donemethod is called afterpipeline_runtime.New(...).Run(runnerCtx)returns. TheRunmethod inworkflow.gois:func (r *Runtime) Run(runnerCtx context.Context) error { ... defer destroyWorkflowFunc() if err := r.engine.SetupWorkflow(r.ctx, r.spec, r.taskUUID); err != nil { ... return err } for _, stage := range r.spec.Stages { ... } destroyWorkflowFunc() r.uploadWait.Wait() return r.err.Get() }So
DestroyWorkflowis called insideRun, thenuploadWait.Wait()is called, thenRunreturns. AfterRunreturns, the agent callsDone.The
uploadWait.Wait()waits for all goroutines to finish (including the service step's goroutine). So the service step's goroutine should finish beforeRunreturns.But wait - what if the service step's goroutine is blocked indefinitely? The
WaitStepfunction waits for the pod to reach a terminal state or the context to be canceled. If the context is not canceled,WaitStepwill wait forever.But the context for
WaitStepisr.ctx, which is set inNew:r.ctx = context.Background()And
WithContextoption can override this:pipeline_runtime.WithContext(workflowCtx)The
workflowCtxis created inagent/runner.go:workflowCtx, _ := context.WithTimeout(ctxMeta, timeout) workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx) defer cancelWorkflowCtx(nil)The
workflowCtxis canceled when the workflow completes or times out. SoWaitStepshould return when the context is canceled.But what if the context is canceled before
WaitStepstarts? ThenWaitStepwill return immediately withcontext.Canceled.Actually, I think I understand the issue now. The problem is that
DestroyWorkflowis called INSIDERun, beforeuploadWait.Wait(). So the flow is:- All stages complete
destroyWorkflowFunc()is called (insideRun)DestroyWorkflowdestroys all pods- The service step's goroutine is waiting in
WaitStep - The goroutine continues after
WaitStepreturns DestroyStepis called (but the pod is already deleted)- The error is handled
traceStepis calleduploadWait.Done()is calleduploadWait.Wait()returnsRunreturns- The agent calls
Done
But the issue is that step 6 might fail, and the error might be returned from
completeStep. ThentraceStepis called with the error state.Actually, looking at
completeStepagain:if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil { return nil, err }If
DestroyStepfails, the error is returned. ButwaitStateis already set fromWaitStep. So the state is returned with the error fromDestroyStep.Wait, but if
DestroyStepfails, the function returnsnil, err, notwaitState, err. So the state is not returned.Let me re-read the issue more carefully. The issue says:
"The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
So the pods ARE destroyed. This means
DestroyWorkflowis being called. But the step state is not updated.I think the issue might be that the service step's goroutine is not finishing before
Doneis called. TheuploadWait.Wait()should wait for all goroutines to finish, but what if the goroutine is blocked somewhere?Actually, let me look at the
startStepfunction:func (r *Runtime) startStep(step *backend_types.Step) (func(), int64, error) { if err := r.engine.StartStep(r.ctx, step, r.taskUUID); err != nil { return nil, 0, err } startTime := time.Now().Unix() rc, err := r.engine.TailStep(r.ctx, step, r.taskUUID) if err != nil { return nil, 0, err } var wg sync.WaitGroup wg.Go(func() { logger := r.makeLogger() if err := r.logger(step, rc); err != nil { logger.Error().Err(err).Str("step", step.Name).Msg("step log streaming failed") } _ = rc.Close() }) return wg.Wait, startTime, nil }The
waitForLogsfunction iswg.Wait, which waits for the log streaming goroutine to finish. IncompleteStep:waitForLogs() waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID)waitForLogswaits for the log streaming to finish. But the log streaming might be blocked if the pod is deleted before the logs are fully read.Actually, I think the issue might be with the log streaming. Let me check the
TailStepfunction in the kubernetes backend:func (e *kube) TailStep(ctx context.Context, step *types.Step, taskUUID string) (io.ReadCloser, error) { ... select { case <-up: case <-ctx.Done(): return nil, ctx.Err() } ... }The
TailStepfunction waits for the pod to be in a terminal state before returning the log stream. If the pod is deleted before it reaches a terminal state, the informer might not see the update, and theupchannel will never be closed.But the informer should see the "delete" event when the pod is deleted. Let me check the informer setup:
podUpdated := func(_, newPod any) { pod, ok := newPod.(*kube_core_v1.Pod) ... switch pod.Status.Phase { case kube_core_v1.PodRunning, kube_core_v1.PodSucceeded, kube_core_v1.PodFailed: upOnce.Do(func() { close(up) }) } } if _, err := si.Core().V1().Pods().Informer().AddEventHandler( cache.ResourceEventHandlerFuncs{ UpdateFunc: podUpdated, }, ); err != nil { return nil, err }The informer only has an
UpdateFunc, not aDeleteFunc. So when the pod is deleted, the informer won't trigger thepodUpdatedfunction. Theupchannel will never be closed, andTailStepwill wait until the context is canceled.This could be the issue! The
TailStepis waiting for the pod to reach a terminal state, but if the pod is deleted before that, theupchannel is never closed. TheTailStepreturns withcontext.Canceled, and the error is returned fromstartStep.Then in
runDetachedStep:waitForLogs, startTime, err := r.startStep(step) if err != nil { return r.traceStep(nil, err, step) }If
startStepfails,traceStepis called withprocessState == nilanderr != nil. A dummy state withExited: trueis created and sent to the server.But wait - this means the step state should be updated to "failure" (because there's an error). But the issue says the step is "running".
Actually, I think the issue might be different. Let me re-read the issue again:
"With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
The key phrase is "pods from kubernetes backend are destroyed properly". This means the pods ARE destroyed. But the step state is still "running".
This suggests that:
- The pod was destroyed (by
DestroyWorkflow) - The step state was not updated to "success" or "failure"
If the step state is still "running", it means:
- The agent never sent a state update after the pod was destroyed
- Or the agent sent a state update, but the server didn't process it
I think the issue might be that the agent is blocked in
WaitSteporTailStepwhenDestroyWorkflowis called. TheDestroyWorkflowdestroys the pod, but the agent is still waiting. The agent will eventually unblock (when the context is canceled), but by that time, the state update might not be processed correctly.Actually, let me look at the informer setup more carefully. The informer is created with:
si := informers.NewSharedInformerFactoryWithOptions(e.client, defaultResyncDuration, informers.WithNamespace(e.config.GetNamespace(step.OrgID)))And the event handler is:
if _, err := si.Core().V1().Pods().Informer().AddEventHandler( cache.ResourceEventHandlerFuncs{ UpdateFunc: podUpdated, }, ); err != nil { return nil, err }But there's no
DeleteFunc! So when the pod is deleted, the informer won't trigger any event. TheWaitStepfunction will wait forever for the pod to reach a terminal state.But the context is not canceled... Wait, the context IS canceled when the workflow completes. So
WaitStepshould return withcontext.Canceled.Hmm, but what if the context is canceled before
WaitStepstarts? ThenWaitStepwill return immediately withcontext.Canceled.Actually, I think I need to trace through the flow more carefully with actual timing:
wait-for-postgresstep completes successfully (time T1)- The main loop finishes all stages (time T2)
destroyWorkflowFunc()is called (time T3)DestroyWorkflowiterates over all steps and destroys them (time T4)- At time T4, the
postgresservice pod is deleted - The
postgresservice step's goroutine is inWaitStep, waiting for the pod to reach a terminal state - The informer sees the delete event but doesn't trigger any handler (no
DeleteFunc) WaitStepwaits for the context to be canceled- The context is canceled when the workflow completes (time T5)
WaitStepreturnscontext.Canceled(time T6)completeStepcreates a state withError = pipeline_errors.ErrCancelDestroyStepis called (but the pod is already deleted, so it returns nil)traceStepis called with the state- The server receives the state and updates the step to "failure" (because there's an error)
But the issue says the step is "running", not "failure". This means the server didn't receive the state update, or the update was ignored.
Actually, I think the issue might be with how the state is sent. Let me check the tracer implementation:
<tool_call> <function=bash_terminal> <parameter=command> cat agent/tracer.go </tool_call>
- The
Assistant
The command failed to produce output. Let me try using grep to find the tracer implementation.
Assistant
Let me look at the agent/tracer.go file to see how the tracer works.
Assistant
I found the issue! In
agent/tracer.go, thecreateTracerfunction creates aStepStatewith:stepState := rpc.StepState{ StepUUID: state.CurrStep.UUID, Exited: state.CurrStepState.Exited, ExitCode: state.CurrStepState.ExitCode, Started: state.CurrStepState.Started, Canceled: errors.Is(state.CurrStepState.Error, pipeline_errors.ErrCancel), Skipped: state.CurrStepState.Skipped, } if state.CurrStepState.Error != nil { stepState.Error = state.CurrStepState.Error.Error() } if state.CurrStepState.Exited { stepState.Finished = time.Now().Unix() }The key line is:
if state.CurrStepState.Exited { stepState.Finished = time.Now().Unix() }If
Exitedisfalse,Finishedis not set! This means the server receives a state withExited: falseandFinished: 0.Looking at the server's
CalcStepStatus:case model.StatusRunning: if state.Exited || state.Error != "" { step.Finished = state.Finished ... }If
Exited: falseandError: "", the step state is not updated. But ifExited: falseandError != "", the step state SHOULD be updated.So the issue might be that the agent is sending
Exited: falseandError: "". Let me check when this happens.Actually, I think I found the real issue! Looking at the flow:
WaitStepis called and waits for the pod to reach a terminal state- The pod is deleted by
DestroyWorkflowbefore it reaches a terminal state WaitStepreturnscontext.CanceledcompleteStepcreates a state withError = pipeline_errors.ErrCancelbutExited: falsetraceStepcopies the state withExited: falseandError != ""- The server receives the state with
Exited: falseandError != "" - The server updates the step state because
state.Error != ""
So the step state SHOULD be updated. But the issue says the step is still "running".
Wait, let me check what happens when
WaitStepis called and the pod is already deleted. TheGetcall might fail with "not found":pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{}) if err != nil { return nil, err }If the pod is not found,
WaitStepreturns an error. Then incompleteStep:waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) if err != nil { if errors.Is(err, context.Canceled) { if waitState == nil { waitState = &backend_types.State{} } waitState.Error = pipeline_errors.ErrCancel } else { return nil, err // Returns without setting waitState.Exited! } }If the error is NOT
context.Canceled(e.g., "pod not found"), the function returnsnil, err. This meansprocessStateinrunDetachedStepisnil, andtraceStepis called withprocessState == nilanderr != nil.Looking at
traceStep:case processState == nil && err != nil: s.CurrStepState = backend_types.State{ Error: err, Exited: true, OOMKilled: false, }A dummy state with
Exited: trueis created. So the agent sends a state withExited: trueandError != "".But wait - the dummy state is created in
traceStep, not incompleteStep. SocompleteStepreturnsnil, err, andprocessStateisnilinrunDetachedStep. ThentraceStepis called withprocessState == nilanderr != nil.Looking at
runDetachedStep:processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime) ... if traceErr := r.traceStep(processState, err, step); traceErr != nil { logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result") }If
completeStepreturnsnil, err, thenprocessStateisnil. ThentraceStepis called withprocessState == nilanderr != nil, which creates a dummy state withExited: true.So the agent SHOULD send a state with
Exited: true. But the issue says the step is "running".Hmm, let me think about this differently. Maybe the issue is that the server's
Updatemethod is not being called at all. Let me check if there's any error handling that might prevent the state from being sent.Actually, I think I need to look at the actual flow more carefully. Let me check what happens when
DestroyWorkflowis called before the service step's goroutine finishes.Looking at
workflow.go:for _, stage := range r.spec.Stages { stageChan := r.runStage(runnerCtx, stage.Steps) select { case <-r.ctx.Done(): <-stageChan return pipeline_errors.ErrCancel case err := <-stageChan: if err != nil { r.err.Set(err) } } } destroyWorkflowFunc()After all stages complete,
destroyWorkflowFunc()is called. TheDestroyWorkflowfunction iterates over all steps and destroys them:func (e *kube) DestroyWorkflow(ctx context.Context, conf *types.Config, taskUUID string) error { for _, stage := range conf.Stages { for _, step := range stage.Steps { err := stopPod(ctx, e, step, e.config.newDefaultDeleteOptions()) if err != nil { return err } } } ... }This destroys ALL steps, including services. So when
DestroyWorkflowis called, the service pods are destroyed.But the service step's goroutine might be blocked in
WaitStep. TheWaitStepfunction waits for the pod to reach a terminal state. If the pod is deleted, the informer won't see it (noDeleteFunc), so thefinishedchannel is never closed. The goroutine waits until the context is canceled.The context is not explicitly canceled here. The context is
r.ctx, which is set inNew:r.ctx = context.Background()And
WithContextoption can override this:pipeline_runtime.WithContext(workflowCtx)The
workflowCtxis created inagent/runner.go:workflowCtx, _ := context.WithTimeout(ctxMeta, timeout) workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx) defer cancelWorkflowCtx(nil)The
workflowCtxis canceled when the workflow completes or times out. But when is the workflow considered "complete"?Looking at
agent/runner.go:err = pipeline_runtime.New(...).Run(runnerCtx) state.Finished = time.Now().Unix() ... if err := r.client.Done(doneCtx, workflow.ID, state); err != nil { logger.Error().Err(err).Msg("failed to update workflow status") }The
Donemethod is called afterRunreturns. TheRunmethod returns when all stages complete anduploadWait.Wait()completes.So the flow is:
- All stages complete
destroyWorkflowFunc()is called, which destroys all podsuploadWait.Wait()is called, which waits for all goroutines to finish- The service step's goroutine is blocked in
WaitStep - The goroutine waits for the context to be canceled
- But the context is not canceled! The context is only canceled when
cancelWorkflowCtx(nil)is called in the defer ofagent/runner.go - So the goroutine waits forever!
Wait, but the issue says the pods ARE destroyed. So the goroutine must finish eventually. Let me check when
cancelWorkflowCtx(nil)is called.In
agent/runner.go:workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx) defer cancelWorkflowCtx(nil)The
defer cancelWorkflowCtx(nil)is called whenRunreturns. ButRunis inside theagent/runner.go:Runfunction. So the context is canceled when theRunfunction returns.But
Runwaits foruploadWait.Wait(), which waits for the service step's goroutine to finish. So there's a deadlock!Actually, no. The
cancelWorkflowCtx(nil)is deferred inagent/runner.go, not inpipeline/runtime/workflow.go. So the context is canceled whenagent/runner.go:Runreturns, which is afterpipeline/runtime/workflow.go:Runreturns.Let me trace through the flow more carefully:
agent/runner.go:RunstartsworkflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx)is calleddefer cancelWorkflowCtx(nil)is registeredpipeline_runtime.New(...).Run(runnerCtx)is calledpipeline/runtime/workflow.go:Runstarts- All stages complete
destroyWorkflowFunc()is called, which destroys all podsuploadWait.Wait()is called- The service step's goroutine is blocked in
WaitStep - The goroutine waits for the context to be canceled
- The context is NOT canceled yet! The defer in
agent/runner.gois not executed yet - So the goroutine waits forever!
But wait, the issue says the pods ARE destroyed. So the goroutine must finish eventually. Let me check if there's a timeout somewhere.
Actually, I think the issue is that the goroutine is blocked in
WaitStepwhenDestroyWorkflowis called. TheDestroyWorkflowdestroys the pod, but the goroutine is still waiting. The goroutine will eventually finish when the context is canceled, but the context is only canceled afteruploadWait.Wait()completes.This is a deadlock! The
uploadWait.Wait()waits for the goroutine to finish, but the goroutine is waiting for the context to be canceled, and the context is only canceled afteruploadWait.Wait()completes.But the issue says the pods ARE destroyed. So the goroutine must finish. Let me check if there's another way the context is canceled.
Actually, I think the issue might be with how the context is handled. Let me check the
Runmethod inworkflow.go:func (r *Runtime) Run(runnerCtx context.Context) error { ... defer destroyWorkflowFunc() if err := r.engine.SetupWorkflow(r.ctx, r.spec, r.taskUUID); err != nil { ... } for _, stage := range r.spec.Stages { stageChan := r.runStage(runnerCtx, stage.Steps) select { case <-r.ctx.Done(): <-stageChan return pipeline_errors.ErrCancel case err := <-stageChan: if err != nil { r.err.Set(err) } } } destroyWorkflowFunc() r.uploadWait.Wait() return r.err.Get() }The
r.ctxis used for the step operations (likeWaitStep). If the context is canceled,WaitStepwill return. But when isr.ctxcanceled?Looking at
pipeline/runtime/option.go:func WithContext(ctx context.Context) Option { return func(r *Runtime) { r.ctx = ctx } }And in
agent/runner.go:pipeline_runtime.New( workflow.Config, r.backend, pipeline_runtime.WithContext(workflowCtx), ... ).Run(runnerCtx)The
workflowCtxis the context passed toRun. TheworkflowCtxis canceled whencancelWorkflowCtx(nil)is called in the defer ofagent/runner.go.But wait - looking at
agent/runner.go:workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx) defer cancelWorkflowCtx(nil)The
defer cancelWorkflowCtx(nil)is called when theagent/runner.go:Runfunction returns. Butagent/runner.go:Runwaits forpipeline/runtime/workflow.go:Runto return, andpipeline/runtime/workflow.go:Runwaits foruploadWait.Wait()to complete.So if the service step's goroutine is blocked in
WaitStep, theuploadWait.Wait()will wait forever, and the context will never be canceled.But the issue says the pods ARE destroyed. So the goroutine must finish eventually. Let me check if there's a timeout in
WaitStep.Actually, I just realized something. The
WaitStepfunction usesselectto wait for either thefinishedchannel or the context to be canceled:select { case <-finished: case <-ctx.Done(): return nil, ctx.Err() }If the context is canceled,
WaitStepreturnsctx.Err(). The context is canceled whencancelWorkflowCtx(nil)is called. ButcancelWorkflowCtx(nil)is only called afteruploadWait.Wait()completes.So there's a deadlock! Unless... the
finishedchannel is closed somehow.Wait, I think I understand now. The
finishedchannel is closed when the pod reaches a terminal state (Succeeded, Failed, Unknown). But if the pod is deleted byDestroyWorkflow, the informer might not see the terminal state. Thefinishedchannel is never closed.But the context IS canceled eventually. When
cancelWorkflowCtx(nil)is called, the context is canceled. ButcancelWorkflowCtx(nil)is called afteruploadWait.Wait()completes. AnduploadWait.Wait()waits for the goroutine to finish. And the goroutine is waiting for the context to be canceled.This is a deadlock! Unless there's a timeout somewhere.
Actually, let me re-read the
agent/runner.go:Runfunction more carefully:workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx) defer cancelWorkflowCtx(nil) // Periodically extend the workflow lease while running go func() { for { select { case <-workflowCtx.Done(): logger.Debug().Msg("workflow context done") return case <-time.After(constant.TaskTimeout / 3): ... } } }()There's a goroutine that extends the workflow lease. But this doesn't cancel the context.
Actually, I think the issue might be that the
workflowCtxhas a timeout. Let me check:timeout := time.Hour if minutes := workflow.Timeout; minutes != 0 { timeout = time.Duration(minutes) * time.Minute } workflowCtx, _ := context.WithTimeout(ctxMeta, timeout) workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx)Yes! The
workflowCtxhas a timeout oftime.Hour(orminutes * time.Minuteif specified). So even if the goroutine is blocked inWaitStep, the context will be canceled when the timeout expires.But the timeout is 1 hour, which is way longer than the issue describes. The issue says the service is shown as "running" immediately after the pipeline completes.
OK, I think I need to step back and think about this more carefully. Let me re-read the issue:
"The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
The key phrase is "pods from kubernetes backend are destroyed properly". This means the pods ARE destroyed. But the step state is still "running".
If the pods are destroyed, the service step's goroutine must have finished. If the goroutine finished,
traceStepwas called, and the server received a state update. If the server received a state update withExited: true, the step state should be updated.So the question is: why is the step state still "running"?
Possible reasons:
- The agent never sent a state update (goroutine never finished)
- The agent sent a state update, but the server didn't process it
- The agent sent a state update with
Exited: falseandError: "" - The server received the state update but didn't update the step correctly
I think the issue might be #1 - the goroutine never finishes. But the issue says the pods ARE destroyed, which implies the goroutine finished.
Actually, wait. Let me re-read the issue description more carefully:
"once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running"
So the wait-for-postgres step completes, and the pods are destroyed. But the service is still "running". This suggests that the service step's goroutine is still waiting when the pods are destroyed.
Let me think about this flow:
wait-for-postgresstep completes- The main loop finishes
destroyWorkflowFunc()is calledDestroyWorkflowdestroys all pods, including the service- The service step's goroutine is still waiting in
WaitStep - The goroutine waits for the context to be canceled
- The context is canceled when
cancelWorkflowCtx(nil)is called - But
cancelWorkflowCtx(nil)is only called afteruploadWait.Wait()completes uploadWait.Wait()waits for the goroutine to finish- Deadlock!
But the issue says the pods ARE destroyed. So the goroutine must finish. How?
Actually, I think I might be missing something. Let me check if the
finishedchannel is closed when the pod is deleted.Looking at
WaitStep:podUpdated := func(_, newPod any) { pod, ok := newPod.(*kube_core_v1.Pod) ... switch pod.Status.Phase { case kube_core_v1.PodSucceeded, kube_core_v1.PodFailed, kube_core_v1.PodUnknown: finishedOnce.Do(func() { close(finished) }) } }The
podUpdatedfunction only handles update events. If the pod is deleted, the informer won't trigger this function. But maybe the informer triggers a delete event that causes thefinishedchannel to be closed?Actually, no. The informer only has an
UpdateFunc, not aDeleteFunc. So when the pod is deleted, the informer won't trigger any event.But wait - the
finishedchannel might be closed by theDeleteFuncif it's set. Let me check:if _, err := si.Core().V1().Pods().Informer().AddEventHandler( cache.ResourceEventHandlerFuncs{ UpdateFunc: podUpdated, }, ); err != nil { return nil, err }There's no
DeleteFunc. So when the pod is deleted, the informer won't trigger any event.Hmm, but the issue says the pods ARE destroyed. So something must be happening.
Actually, I think I need to check if the informer also handles delete events by default. Let me check the Kubernetes informer documentation.
Actually, the Kubernetes informer's
AddEventHandlertakes acache.ResourceEventHandlerwhich hasAddFunc,UpdateFunc, andDeleteFunc. IfDeleteFuncis not set, the informer won't call any function on delete events.So when the pod is deleted, the
finishedchannel is never closed. TheWaitStepfunction waits forever for the context to be canceled.But the context is only canceled after
uploadWait.Wait()completes. AnduploadWait.Wait()waits for the goroutine to finish. This is a deadlock!Unless... the context is canceled somewhere else.
Actually, I just realized something. The
workflowCtxhas a timeout! So even if the goroutine is blocked, the context will be canceled after the timeout expires.But the timeout is 1 hour, which is way longer than the issue describes.
OK, I think I need to actually reproduce the issue to understand what's happening. Let me search for any related issues or PRs in the repository.
<tool_call> <function=bash_terminal> <parameter=command> git log --oneline --all | head -30 </tool_call>
Assistant
Let me try the git command again properly.
Assistant
Let me search for issues or changes related to service cleanup or pipeline completion.
<tool_call> <function=bash_terminal> <parameter=command> git log --oneline --all --grep="service" | head -20 </tool_call>
Assistant
Let me try running the git log command again properly.
Assistant
I found a relevant commit. Let me look at the changes in "Fix workflow hang on services (#6507)":
Assistant
This looks like exactly the issue I'm investigating. Let me look at the actual changes in this commit:
Assistant
Let me try a different approach to see the changes.
Assistant
Now I understand the issue better. The commit
60df1c618was a fix for a similar issue where the workflow would hang because services were not torn down properly. The fix was to:- Move
DestroyWorkflowto be called BEFOREuploadWait.Wait() - Add a stop channel to the dummy backend that services can wait on
However, looking at the commit, I see that the fix was applied to the dummy backend to allow services to be stopped, but the actual kubernetes backend might not have this stop channel mechanism.
Let me check if the kubernetes backend has the same issue - does it have a way to signal services to stop?
<tool_call> <function=bash_terminal> <parameter=command> grep -n "stop" pipeline/backend/kubernetes/kubernetes.go </tool_call>
- Move
Assistant
Let me continue exploring to find if the kubernetes backend has a stop mechanism for services.
Assistant
I see that the kubernetes backend has
StopTimeoutconfiguration for graceful shutdown. TheDestroyStepandDestroyWorkflowfunctions callstopPodwith the delete options that include theStopTimeout.But the issue is that the service pods are being destroyed (via
DestroyWorkflow), but the step state is not being updated to "success" or "failure" because the agent's goroutine is blocked inWaitStepwaiting for the pod to reach a terminal state.The problem is that the kubernetes
WaitStepfunction uses an informer that only listens for "update" events, not "delete" events. So when the pod is deleted byDestroyWorkflow, the informer doesn't trigger, and thefinishedchannel is never closed. TheWaitStepfunction waits until the context is canceled.But the context is only canceled after
uploadWait.Wait()completes, anduploadWait.Wait()waits for the goroutine to finish. This is a deadlock!The fix in commit
60df1c618added a stop channel to the dummy backend that services can wait on. The kubernetes backend needs a similar mechanism.Actually, wait. Let me re-read the issue more carefully. The issue says:
"The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
If the pods ARE destroyed, that means
DestroyWorkflowIS being called. But the step state is still "running", which means the server never received a state update withExited: true.This suggests that the agent's goroutine is blocked in
WaitStepand never finishes, so it never callstraceStepwith the final state.But wait - if the pods are destroyed, how does the user know? The issue says "The pods from kubernetes backend are destroyed properly". This might be observed from the Kubernetes API, not from the Woodpecker API.
So the flow is:
- All main steps complete
DestroyWorkflowis called, which destroys all pods (including services)- The service step's goroutine is blocked in
WaitStepwaiting for the pod to reach a terminal state - The informer doesn't see the delete event (no
DeleteFunc), so thefinishedchannel is never closed - The goroutine waits until the context is canceled
- But the context is only canceled after
uploadWait.Wait()completes - And
uploadWait.Wait()waits for the goroutine to finish - Deadlock!
But the issue says the pods ARE destroyed, which means step 2 happened. But the goroutine is still blocked in step 3. The user observes this from the Woodpecker API, which shows the service as "running".
Actually, I think the issue might be a timeout. The
workflowCtxhas a timeout (default 1 hour). So eventually, the context will be canceled, andWaitStepwill return. But by that time, the server might have already marked the pipeline as done.Actually, wait. Let me re-read the issue again. The issue says:
"These are only cleaned after the pipeline times out."
So the services are cleaned up only after the pipeline times out! This confirms that the goroutine is blocked in
WaitStepuntil the context times out.The fix should be to add a mechanism to signal services to stop, similar to what was done in the dummy backend in commit
60df1c618.Let me check if the kubernetes backend has a stop channel mechanism:
<tool_call> <function=bash_terminal> <parameter=command> grep -n "chan|stop|close" pipeline/backend/kubernetes/pod.go | head -30 </tool_call>
Assistant
Let me try running the grep command again properly.
Assistant
Let me look at the kubernetes backend's WaitStep function more carefully to see how it handles pod deletion:
Assistant
Now I understand the issue. The
WaitStepfunction uses an informer that only listens for "update" events. When the pod is deleted byDestroyWorkflow, the informer doesn't see the delete event (noDeleteFunc), so thefinishedchannel is never closed.The fix should be to add a
DeleteFuncto the informer that closes thefinishedchannel when the pod is deleted. This way, whenDestroyWorkflowdeletes the service pods, theWaitStepfunction will be notified and will return.Let me also check if there's a
DeleteFuncthat should be added:<tool_call> <function=bash_terminal> <parameter=command> sed -n '280,335p' pipeline/backend/kubernetes/kubernetes.go </tool_call>
Assistant
The command failed to produce output. Let me try again with a proper format.
Assistant
Now I understand the issue. The
WaitStepfunction in the kubernetes backend only listens for "update" events, not "delete" events. WhenDestroyWorkflowdeletes the service pods, the informer doesn't notifyWaitStepbecause there's noDeleteFunchandler. This causesWaitStepto block until the context times out, which is why the service steps remain "running" until the pipeline times out.The fix is to add a
DeleteFuncto the informer that closes thefinishedchannel when the pod is deleted. Let me implement this fix:Assistant
Now I understand the issue. The
WaitStepfunction in the kubernetes backend only listens for "update" events, not "delete" events. WhenDestroyWorkflowdeletes the service pods, the informer doesn't notifyWaitStepbecause there's noDeleteFunchandler. This causesWaitStepto block until the context times out, which is why the service steps remain "running" until the pipeline times out.The fix is to add a
DeleteFuncto the informer that closes thefinishedchannel when the pod is deleted. Let me implement this fix:Assistant
Let me apply the fix directly using the edit command:
<tool_call> <function=bash_terminal> <parameter=command> sed -n '306,322p' pipeline/backend/kubernetes/kubernetes.go </tool_call>
Assistant
The command failed to produce output. Let me try again with proper formatting.
Assistant
Now I can see the exact code. I need to:
- Add a
podDeletedfunction before the informer setup (after line 310) - Add
DeleteFunc: podDeletedto theResourceEventHandlerFuncs(line 316)
Let me apply the fix:
- Add a
Assistant
I need to open the file first:
Assistant
Now I can see the file is open. Let me scroll to the area where I need to make the changes (around line 310-320):
<tool_call> <function=bash_terminal> <parameter=command> goto 280 </tool_call>
Assistant
Let me scroll down to find the WaitStep function where I need to add the DeleteFunc handler.
Assistant
Now I can see the exact code. I need to:
- Add a
podDeletedfunction after line 310 (after thepodUpdatedfunction) - Add
DeleteFunc: podDeletedto theResourceEventHandlerFuncs(line 315-316)
Let me apply the fix:
<tool_call> <function=bash_terminal> <parameter=command> edit 310:311 << 'EOF' }
podDeleted := func(_ any) { // Pod was deleted (e.g., by DestroyWorkflow) - signal completion finishedOnce.Do(func() { close(finished) }) }EOF </tool_call>
- Add a
Assistant
Let me try using the edit command with the proper format:
Assistant
I see the
podDeletedfunction was added, but I also need to add it to theResourceEventHandlerFuncs. Let me do that now:Assistant
I see there's a duplicate error handling block. Let me fix that:
<tool_call> <function=bash_terminal> <parameter=command> edit 321:327 << 'EOF' ); err != nil { return nil, err }
stop := make(chan struct{}) si.Start(stop) defer close(stop)EOF </tool_call>
Assistant
Let me fix the duplicate error handling block that was created.
Assistant
I see there's a duplicate error handling block at lines 325-326. Let me remove it:
Assistant
Now there are duplicate lines at 328-329. Let me fix that:
<tool_call> <function=bash_terminal> <parameter=command> edit 326:330 << 'EOF' stop := make(chan struct{}) si.Start(stop) defer close(stop)
EOF </tool_call>
Assistant
Let me check the current state of the file and fix the duplicate lines:
Assistant
I see the duplicate lines. Let me fix them:
Assistant
Now let me verify the fix is correct:
Assistant
Now let me also fix the
TailStepfunction which has the same issue:Assistant
Let me look at more context for the TailStep function:
Assistant
I see the
TailStepfunction has the same issue. Let me add theDeleteFunchandler there too:Assistant
Let me add the
podDeletedhandler forTailStep:Assistant
Now let me add the
DeleteFuncto the event handler inTailStep:Assistant
Let me add the
DeleteFunc: podDeletedto the event handler:Assistant
Now let me verify the changes are correct:
Assistant
Now let me verify the fix compiles correctly:
Assistant
Let me set the GOPATH and try again:
Assistant
Let me set up the environment properly:
Assistant
There's a duplicate
DeleteFuncfield. This happened because I added it to bothWaitStepandTailStepusing the same sed command. Let me check both locations and fix the issue:Assistant
I see there's a duplicate at line 321. Let me fix this:
Assistant
Let me remove the duplicate
DeleteFuncline:Assistant
Now let me verify the fix compiles:
Assistant
The code compiles. Now let me also run the existing tests to make sure the fix doesn't break anything:
Assistant
The tests are passing so far. Let me run the tests for the runtime package as well since that's where the main issue might be:
<tool_call> <function=bash_terminal> <parameter=command> export GOPATH=/root/go && export GOMODCACHE=GOPATH/pkg/mod && export GOCACHE=GOPATH/.cache/go-build && go test ./pipeline/runtime/... -v -count=1 2>&1 | tail -50 </tool_call>
Assistant
Let me run the runtime tests properly.
Assistant
There are no test files in the runtime package. Let me check if the tests are in a different location.
<tool_call> <function=bash_terminal> <parameter=command> ls -la pipeline/runtime/ </tool_call>
Assistant
The command failed again. Let me try running the test command properly.
Assistant
There are test files in the runtime package. Let me run the tests:
<tool_call> <function=bash_terminal> <parameter=command> export GOPATH=/root/go && export GOMODCACHE=GOPATH/pkg/mod && export GOCACHE=GOPATH/.cache/go-build && go test ./pipeline/runtime/... -run TestWorkflowUnbounded -v -count=1 </tool_call>
Assistant
The command is failing with an error message. Let me try running a simpler test command to see if the Go testing environment is working.
Assistant
It says "no test files" but I see
runtime_test.goin the directory. Let me check the test file:<tool_call> <function=bash_terminal> <parameter=command> head -30 pipeline/runtime/runtime_test.go </tool_call>
Assistant
The command failed again. Let me try running the test command with proper syntax.
Assistant
The command is returning "no test files" even though there are test files. This might be a module path issue. Let me check the go.mod file and try running tests from the root:
<tool_call> <function=bash_terminal> <parameter=command> go test go.woodpecker-ci.org/woodpecker/v3/pipeline/runtime -run TestWorkflowUnbounded -v -count=1 2>&1 | head -50 </tool_call>
Assistant
Let me check the go.mod file and run tests from the root:
Assistant
The test files exist but Go isn't finding them. Let me check if they have a build tag issue or if they're not recognized as test files for some reason:
<tool_call> <function=bash_terminal> <parameter=command> head -20 pipeline/runtime/runtime_test.go </tool_call>
Assistant
Let me check the test files in a different way:
Assistant
I see! The test files have a
//go:build testbuild tag, which means they only run when thetesttag is specified. Let me run the tests with the test tag:<tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/runtime/... -tags test -run TestWorkflowUnbounded -v -count=1 2>&1 | head -50 </tool_call>
Assistant
The command is failing. Let me try running the test without the -v flag first to see if it works:
Assistant
The test passed! But I want to see the actual output to confirm the test ran. Let me run it with verbose output:
<tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/runtime/... -tags test -run TestWorkflowUnbounded -v -count=1 </tool_call>
Assistant
Let me try a different approach to run the tests with verbose output:
Assistant
The tests pass! This confirms that the runtime correctly handles unbounded services and detached steps without hanging. The key observation from the test output is:
delete workflow environmentis calledstop step dbis called (the service step)stop step background-workeris called (the detached step)logs and traces uploadedcompletes
This shows that the runtime is properly tearing down services after all main steps complete.
However, I realized my fix in the kubernetes backend might not be the root cause of the issue. Looking at the test output, it seems the runtime is already handling this correctly by calling
DestroyWorkflowbeforeuploadWait.Wait(). The fix in commit60df1c618already addresses this.But the issue in the bug report is specifically about the kubernetes backend. The problem is that when
DestroyWorkflowdeletes the pods, theWaitStepfunction in the kubernetes backend doesn't get notified because it only listens for "update" events, not "delete" events.My fix adds a
DeleteFunchandler that closes thefinishedchannel when the pod is deleted. This should allowWaitStepto return immediately when the pod is deleted byDestroyWorkflow, rather than waiting until the context times out.Let me verify my changes are correct by looking at the modified file:
<tool_call> <function=bash_terminal> <parameter=command> cat pipeline/backend/kubernetes/kubernetes.go | grep -A 10 "podDeleted" </tool_call>
Assistant
Let me check the changes I made to the kubernetes backend file.
Assistant
The changes look correct. I've added
podDeletedhandlers in bothWaitStep(line 312-315) andTailStep(line 396-399) that close the respective channels when a pod is deleted. TheDeleteFuncis added to the event handler in both places (line 320 and line 405).Now let me run all the kubernetes backend tests to make sure my changes don't break anything:
Assistant
All kubernetes backend tests pass. Let me also run the server-side tests related to pipeline status to make sure nothing is broken:
Assistant
The tests pass. Let me also run the step_status tests to make sure the status update logic is working correctly:
<tool_call> <function=bash_terminal> <parameter=command> go test -v ./server/pipeline/... -run TestUpdateStepStatus 2>&1 | tail -80 </tool_call>
Assistant
Let me run the step_status tests to verify the status update logic is working correctly.
Assistant
All the tests pass! The
TestUpdateStepStatustests verify that the step status update logic is working correctly.Now let me verify the fix by looking at the final state of the modified file and ensuring the changes are correct:
<tool_call> <function=bash_terminal> <parameter=command> cat pipeline/backend/kubernetes/kubernetes.go | head -350 | tail -70 </tool_call>
Assistant
The command failed again. Let me try using the edit command to view the relevant section of the file.
Assistant
Now let me check the TailStep function changes as well:
<tool_call> <function=bash_terminal> <parameter=command> sed -n '390,425p' pipeline/backend/kubernetes/kubernetes.go </tool_call>
Assistant
The command is failing to produce output. Let me try a different approach.
Assistant
The changes look correct. I have added the
podDeletedhandler in bothWaitStepandTailStepfunctions, and added theDeleteFuncto the event handlers. This ensures that when a pod is deleted (e.g., byDestroyWorkflow), thefinishedorupchannel is closed, allowing the waiting goroutines to proceed.Now let me verify that the entire fix is complete by running a broader set of tests to ensure nothing is broken:
<tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/... 2>&1 | tail -20 </tool_call>
Assistant
Let me run the tests in a different way:
Assistant
It shows FAIL at the end. Let me check which test failed:
<tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/... 2>&1 | grep -A 5 "FAIL" </tool_call>
Assistant
Let me check which test failed by searching for "FAIL" in the test output.
Assistant
The failure is in the dummy backend build. Let me check what's wrong:
Assistant
The build succeeds. The test failure might be a cached test issue. Let me clear the cache and run the tests again:
Assistant
Let me check which specific test failed:
<tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/... 2>&1 | grep "FAIL" </tool_call>
Assistant
The command seems to be failing to produce output. Let me try running the test differently to find the failure.
Assistant
The dummy backend is failing to build. Let me check what the actual error is:
<tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/backend/dummy/... 2>&1 </tool_call>
Assistant
The command is failing to produce output. Let me try running the test command with proper error handling.
Assistant
The dummy backend requires the
testbuild tag to compile. Let me run the tests with the test tag:<tool_call> <function=bash_terminal> <parameter=command> go test -tags test ./pipeline/... 2>&1 | tail -20 </tool_call>
Assistant
Let me run the tests with the test tag:
Assistant
All tests pass with the test tag! The earlier failures were due to the test tag being required for the dummy backend.
Now let me also run the server tests to make sure the fix doesn't break anything on the server side:
<tool_call> <function=bash_terminal> <parameter=command> go test -tags test ./server/... 2>&1 | tail -30 </tool_call>
Assistant
Let me run the server tests to ensure the fix doesn't break anything on the server side.
Assistant
There's a FAIL at the end. Let me check which test failed:
Assistant
The failures are in
server/routerandserver/webwhich are likely due to frontend assets or build configuration, not related to my changes. Let me verify this:<tool_call> <function=bash_terminal> <parameter=command> go test -tags test ./server/router/... 2>&1 | head -20 </tool_call>
Assistant
Let me check the router test failure details.
Assistant
The failures are due to missing frontend build assets (
web/web.go:26:12: pattern all:dist/*: no matching files found), not related to my changes. This is a pre-existing issue in the test environment where the web frontend hasn't been built.The important tests (kubernetes backend, runtime, server/pipeline) all pass. My fix is complete.
Let me now summarize the changes I made and verify they are correct:
Summary of Changes
I fixed the issue where service steps remain "running" in the Woodpecker API after the pods are destroyed by the Kubernetes backend.
Root Cause
The Kubernetes backend's
WaitStepfunction only listens for "update" events on pods, not "delete" events. WhenDestroyWorkflowdeletes service pods, the informer doesn't notifyWaitStep, causing it to block until the context times out (default 1 hour).Fix
I added
DeleteFunchandlers to the pod informers in bothWaitStepandTailStepfunctions. When a pod is deleted, these handlers close the respective channels (finishedforWaitStep,upforTailStep), allowing the waiting goroutines to proceed.Changes Made
In
pipeline/backend/kubernetes/kubernetes.go:- Added
podDeletedhandler inWaitStepfunction (lines 312-315):
podDeleted := func(_ any) { // Pod was deleted (e.g., by DestroyWorkflow) - signal completion finishedOnce.Do(func() { close(finished) }) }-
Added
DeleteFunc: podDeletedto the event handler inWaitStep(line 320) -
Added
podDeletedhandler inTailStepfunction (lines 396-399):
podDeleted := func(_ any) { // Pod was deleted - allow log streaming to proceed upOnce.Do(func() { close(up) }) }- Added
DeleteFunc: podDeletedto the event handler inTailStep(line 405)
These changes ensure that when
DestroyWorkflowdeletes service pods, the waiting goroutines are immediately notified and can return, allowing the step state to be updated to "success" or "failure" on the server.Let me submit the changes:
- Added
Assistant
The diff shows the correct changes I made to fix the issue. Let me submit the fix:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
claude-sonnet-5Compare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-35B-A3BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →
Qwen/Qwen3.5-35B-A3BCompare side by side →