ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…
- User
ISSUE DESCRIPTION
Woodpecker services appear to be running on UI & API after pipeline completes
Component
server
Describe the bug
Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.
These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).
Steps to reproduce
- Install woodpecker next-d05c597374
- Use kubernetes backend
- Define pipeline with services running infinitely, like running postgres
services: - name: postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 ports: - 5432 environment: POSTGRES_PASSWORD: somepass POSTGRES_USER: someuser POSTGRES_DB: somedb steps: - name: wait-for-postgres image: public.ecr.aws/docker/library/postgres:16-alpine3.21 environment: DB_HOST: postgres DB_PORT: 5432 MAX_WAIT: 60 commands: - ./scripts/wait-for-postgres.sh- Run the pipeline, observe pods start for the service and the step
- once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
pr-build > clone (#2): Step: clone Started: 1779195580 Stopped: 1779195594 Type: clone State: success pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: running pr-build > wait-for-postgres (#4): Step: wait-for-postgres Started: 1779195596 Stopped: 1779195602 Type: commands State: successExpected behavior
Pipeline services are stopped once the pipeline steps complete, the pipeline completes.
System Info
next-d05c597374 kubernetes backend, bitbucket datacenter forgeAdditional context
No response
Validations
- Read the docs.
- Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
- Checked that the bug isn't fixed in the
nextversion already [https://woodpecker-ci.org/versions]
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /woodpecker, current file: none) bash-$
Assistant
I'll start by exploring the repository structure to understand the codebase and locate the relevant code for handling pipeline services in the Kubernetes backend.
Assistant
Let me continue exploring the repository structure to understand the codebase better.
Assistant
Let me explore the pipeline directory since the issue is about pipeline services not being cleaned up properly. I'll also look at the kubernetes backend code.
Assistant
Let me look at the kubernetes backend files, especially the main kubernetes.go file and the service.go file to understand how services are handled.
Assistant
I need to use the correct argument name. Let me check the available tools again - it seems I should just use the file path directly.
Assistant
The open command seems to have an issue. Let me try using bash commands to view the file instead.
Assistant
Now I can see the Kubernetes backend code. The issue is that services are not being cleaned up properly. Let me look at the pipeline execution logic to understand how steps and services are managed. The key issue seems to be that services remain in a "running" state even after they've been destroyed.
Let me look for where the pipeline execution happens and how services are stopped. I should check the agent runner and server logic.
Assistant
The user is pointing out that I didn't provide the required argument "command" for the bash_terminal tool. I need to continue my investigation of the codebase. Let me search for where services are stopped and how their state is managed.
Assistant
Let me search for where pipeline steps are managed and how their state is updated, especially for services.
Assistant
Let me look at the pipeline runtime to understand how steps and services are executed:
Assistant
Let me look at the step execution logic to understand how services are handled:
Assistant
Now I understand the structure better. Services are detached steps. Let me search for how services are defined and handled, and look at the server-side code that updates step states:
Assistant
Let me look at the server RPC handler to understand how it processes step state updates from the pipeline:
Assistant
Now I can see that the RPC
Donemethod callscompleteChildrenIfParentCompletedwhich should handle completing child steps when the parent workflow completes. Let me look at theUpdateStepStatusfunction in the pipeline package:Assistant
Now I'm understanding the issue better. Let me look at how the workflow and pipeline structure handles services:
Assistant
Let me continue investigating by looking at the model/step.go to understand step types and how they relate to services.
Assistant
I can see that services are steps with
StepType = "service"and they are also marked asDetached = true. Let me look at how the workflow completion logic handles services. The issue is that when the pipeline completes, the services are not being marked as stopped/killed.Let me look at the
Donemethod in the RPC and how it handles child steps.Looking at the rpc.go code I already saw, the
Donemethod callscompleteChildrenIfParentCompletedwhich only handles children of the workflow. But I need to understand the parent-child relationship better and see if services are being handled correctly.Let me look for how services are related to other steps and when they should be stopped.
Assistant
The error message says "No tool call was invoked." I need to provide a proper bash command to continue exploring the codebase. Let me look at the workflow model and understand the parent-child relationship between workflows and steps.
Based on what I've seen so far:
- Services are steps with
StepType = "service"andDetached = true - When a workflow completes, the
Donemethod is called which callscompleteChildrenIfParentCompleted - The issue is that services are not being properly stopped when the pipeline completes
Let me continue investigating the code to understand the parent-child relationship and how services should be stopped.
- Services are steps with
Assistant
Now I need to look at the complete
Donemethod in rpc.go to understand wherecompleteChildrenIfParentCompletedis called and how it's supposed to work:Assistant
I can now see the issue! Looking at the
Donemethod in rpc.go:- It loads the workflow and its children (steps)
- It calls
s.completeChildrenIfParentCompleted(workflow, state.Finished)to complete any still-running children
Now let me look at the
completeChildrenIfParentCompletedfunction more closely:func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) { for _, c := range completedWorkflow.Children { if c.Running() { if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil { log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID) } else { // Update in-memory state so WorkflowStatus sees the final state c.State = updated.State c.Finished = updated.Finished } } } }And the
UpdateStepToStatusSkippedfunction:func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }Wait, this looks like it should work. The
completeChildrenIfParentCompletedfunction should mark all running children as killed/success.But wait, looking at the issue description again:
- "The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
So the pods are being destroyed (which means
DestroyWorkflowis being called), but the API still shows services as running. This suggests that theDoneRPC call might not be getting the right list of children, or there's a timing issue.Let me think about this more carefully:
- Services are detached steps - they run in the background
- When all non-detached steps complete, the workflow should call
Done - The
Donemethod should callcompleteChildrenIfParentCompletedto mark all running children (including services) as done
But wait, looking at the runtime code again, I see that detached steps (services) have their
completeStepcalled in a goroutine:func (r *Runtime) runDetachedStep(runnerCtx context.Context, step *backend_types.Step) error { // ... start the step ... r.uploadWait.Add(1) go func() { defer r.uploadWait.Done() // ... complete the step in background ... }() return nil }So when a detached step completes, it traces its state change asynchronously. But the key question is: when does the workflow call
Doneon the server?Looking at the workflow.go:
func (r *Runtime) Run(runnerCtx context.Context) error { // ... setup ... for _, stage := range r.spec.Stages { stageChan := r.runStage(runnerCtx, stage.Steps) // ... wait for stage ... } // Now we can shutdown the workflow destroyWorkflowFunc() // Ensure all logs/traces are uploaded before finishing logger.Debug().Msg("waiting for logs and traces upload") r.uploadWait.Wait() // ... }And
runStagereturns when all steps in the stage are done:func (r *Runtime) runStage(runnerCtx context.Context, steps []*backend_types.Step) <-chan error { var g errgroup.Group done := make(chan error) for _, step := range steps { g.Go(func() error { return r.executeStep(runnerCtx, step) }) } go func() { done <- g.Wait() }() return done }So
runStagewaits for all steps (including detached steps that return immediately fromexecuteStep). SinceexecuteStepfor a detached step returns immediately after starting the goroutine, the stage completes, and then the workflow callsdestroyWorkflowFunc().Now, when the workflow finishes, the agent should call
Doneon the server. Let me check how the agent does this.Actually, I think I see the issue now! Looking at the runtime code:
- Detached steps (services) start in a goroutine and return immediately from
executeStep - The stage completes when all steps return
destroyWorkflowFunc()is called, which callsDestroyWorkflowon the backendr.uploadWait.Wait()waits for all background goroutines to complete- The workflow returns
But here's the key: the background goroutine for the detached step calls
completeStep, which in turn callstraceStepto report the step's final state. However, this happens AFTERdestroyWorkflowFunc()is called.So the sequence is:
- All blocking steps complete
- Stage completes
destroyWorkflowFunc()is called - pods are destroyed- Background goroutines for services are still running
uploadWait.Wait()blocks until background goroutines complete- Background goroutines try to call
WaitStepon already-destroyed pods - They report their state
But actually, the services should be destroyed by
DestroyWorkflow, so they should never complete on their own. The issue is that the server side doesn't know they've been destroyed until the workflow callsDone.Let me look at when the agent calls
Done:<tool_call> <function=bash_terminal> <parameter=command> grep -r ".Done(" /woodpecker/agent --include="*.go" -A 5 -B 5 </tool_call>
Assistant
I need to properly format the bash command. Let me fix the syntax.
Assistant
Now I understand the flow better. Let me look at the agent runner.go to see when Done is called and what state is sent to the server. The key is understanding when the agent calls Done and whether it happens before or after the services have been properly cleaned up.
Let me look at the full runner.go file to understand the workflow execution flow.
Assistant
Now I understand the flow:
- The agent runs the pipeline using
pipeline_runtime.New(...).Run(runnerCtx) - The
Runmethod inpipeline/runtime/workflow.go:- Runs all stages (which includes both blocking and detached steps)
- Calls
destroyWorkflowFunc()to destroy the workflow (which destroys all pods including services) - Waits for all uploads/traces with
r.uploadWait.Wait() - Returns
- The agent then calls
r.client.Done(doneCtx, workflow.ID, state)to signal the workflow is done
Now let me think about the issue again:
The problem is that services are shown as "running" in the API even after the pipeline completes. The issue states that:
- Pods are destroyed properly (so
DestroyWorkflowis working) - But API still shows services as state=running
- These are only cleaned after the pipeline times out
Looking at the
Donemethod inserver/rpc/rpc.go, it calls:s.completeChildrenIfParentCompleted(workflow, state.Finished)And
completeChildrenIfParentCompletedshould mark all running children as done. But wait - let me check the code again:func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) { for _, c := range completedWorkflow.Children { if c.Running() { if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil { log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID) } else { // Update in-memory state so WorkflowStatus sees the final state c.State = updated.State c.Finished = updated.Finished } } } }This iterates over
completedWorkflow.Childrenand marks them as done. But where doesworkflow.Childrencome from? Let me check:workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)So it loads the children from the store. But wait, let me check what
StepListFromWorkflowFindreturns. Maybe it's only returning certain steps?Actually, I think I need to look at the store implementation to understand what steps are being loaded. But I think the issue might be in how the steps are being tracked.
Wait, let me re-read the issue more carefully:
"Some regression with newer releases (3.14.x)"
This suggests that something changed in version 3.14.x that broke this functionality. Let me look at the git history to see what changed recently.
Actually, I think I found the issue! Looking at the
UpdateStepToStatusSkippedfunction:func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }This function updates the step state to
StatusSuccessif the step has started, and sets the finished time. But I don't think this is the issue.Let me think about this differently. The issue says that services are not being cleaned properly, and the API still shows them as running. But the pods are being destroyed. So the issue is that the step state in the database is not being updated.
Now, let me look at the flow again:
- Agent runs the pipeline
- When the pipeline completes,
DestroyWorkflowis called, which destroys all pods - The agent calls
Doneon the server - The server calls
completeChildrenIfParentCompletedto mark all running children as done
But wait, I see the issue now! In the
Donemethod:// Complete any still-running children (e.g. service containers) before // computing the workflow status, so their final state is reflected. s.completeChildrenIfParentCompleted(workflow, state.Finished)This is called AFTER loading the workflow and its children. But the issue is that this code path might not be executed in all cases!
Actually, let me look at the
checkWorkflowAllowsStepUpdatefunction to see if there's any issue with the state updates:<tool_call> <function=bash_terminal> <parameter=command> grep -r "checkWorkflowAllowsStepUpdate|checkWorkflowState" /woodpecker/server --include="*.go" -A 20 </tool_call>
- The agent runs the pipeline using
Assistant
The user is pointing out that my tool call was malformed. I need to properly format the bash_terminal command. Let me fix the syntax and try again.
Assistant
Let me look for how steps are loaded from the workflow to understand if services are being properly included in the children list:
Assistant
Good, now I can see the key function
StepListFromWorkflowFindwhich loads steps from the workflow. It queries steps whereppid = workflow.PID. This means it's looking for steps whose parent process ID matches the workflow's PID.Now I understand the parent-child relationship:
- Workflow has a PID
- Steps have a PPID (parent process ID) that points to their parent workflow's PID
So when
completeChildrenIfParentCompletedis called, it should iterate over all steps that belong to the workflow and mark them as done.But wait, let me think about the issue more carefully. The issue says:
- Pods are destroyed properly
- But API still shows services as running
This suggests that the
completeChildrenIfParentCompletedfunction is not being called, or it's not updating the steps correctly.Let me look at the runtime code again to understand when services are stopped. Looking at the workflow.go:
func (r *Runtime) Run(runnerCtx context.Context) error { // ... for _, stage := range r.spec.Stages { stageChan := r.runStage(runnerCtx, stage.Steps) // ... } // Now we can shutdown the workflow destroyWorkflowFunc() // Ensure all logs/traces are uploaded before finishing logger.Debug().Msg("waiting for logs and traces upload") r.uploadWait.Wait() // ... }And
runStageexecutes all steps in parallel, including detached steps (services). For detached steps,executeStepreturns immediately after starting a goroutine, so the stage completes quickly.Then
destroyWorkflowFunc()is called, which callsr.engine.DestroyWorkflow(ctx, r.spec, r.taskUUID). This destroys all pods, including service pods.Then
r.uploadWait.Wait()waits for all background goroutines to complete. But wait, the background goroutines for services callcompleteStep, which tries to callWaitStepon the already-destroyed pod!Actually, looking at the
runDetachedStepfunction again:func (r *Runtime) runDetachedStep(runnerCtx context.Context, step *backend_types.Step) error { waitForLogs, startTime, err := r.startStep(step) if err != nil { return r.traceStep(nil, err, step) } r.uploadWait.Add(1) go func() { defer r.uploadWait.Done() logger := r.makeLogger() processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime) logger.Debug().Str("step", step.Name).Msg("complete") if errors.Is(err, context.Canceled) { err = pipeline_errors.ErrCancel } if err != nil { logger.Error().Err(err).Str("step", step.Name).Msg("detached step failed while running") } if traceErr := r.traceStep(processState, err, step); traceErr != nil { logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result") } }() return nil }So the detached step starts a goroutine that:
- Calls
completeStep(which waits for the pod to complete and destroys it) - Calls
traceStepto report the final state
But if
destroyWorkflowFunc()is called before the goroutine completes, then the pod is already destroyed, andWaitStepmight fail.Actually, let me re-read the
completeStepfunction:func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) { // Drain the log stream before waiting on the process exit. waitForLogs() waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck if err != nil { if errors.Is(err, context.Canceled) { if waitState == nil { waitState = &backend_types.State{} } waitState.Error = pipeline_errors.ErrCancel } else { return nil, err } } // Use runnerCtx here: the workflow context may already be canceled but we // still need to reach the backend to stop/remove the container. if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil { return nil, err } // ... }So
completeStepcallsWaitStepand thenDestroyStep. If the pod is already destroyed byDestroyWorkflow, thenWaitStepmight fail.But wait, the issue is that the API still shows services as running. This suggests that the
traceStepcall is not being made, or the state update is not being sent to the server.Actually, I think I see the issue now! Let me look at the sequence of events:
- All blocking steps complete
- Stage completes (detached steps return immediately)
destroyWorkflowFunc()is called, which destroys all pods (including services)uploadWait.Wait()waits for background goroutines- Background goroutines try to call
WaitStepon destroyed pods WaitStepmight fail because the pod is gone- If
WaitStepfails,completeStepreturns an error traceStepis called with the error
But wait, looking at the
runDetachedStepfunction:if traceErr := r.traceStep(processState, err, step); traceErr != nil { logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result") }So even if there's an error,
traceStepis still called. AndtraceStepshould update the server with the step's state.Let me think about this differently. Maybe the issue is that the background goroutines are not being waited for properly. Let me check the
uploadWaitvariable:Looking at the runtime.go file:
<tool_call> <function=bash_terminal> <parameter=command> cat /woodpecker/pipeline/runtime/runtime.go </tool_call>
Assistant
The user is telling me that I need to provide a proper bash command. I apologize for the error - I had a typo in my tool call. Let me fix it and view the runtime.go file.
Assistant
Let me look at the git history to understand recent changes that might have caused this regression:
Assistant
I see an interesting commit:
e173bed3f Kubernetes: detached steps are no services (#6435). This could be related to the issue. Let me examine this commit and the recent changes to the Kubernetes backend.Assistant
Let me see the actual changes in that commit:
Assistant
This is very interesting! The commit
e173bed3fchanged theisServicefunction to only check forstep.Type == types.StepTypeServiceinstead of also checking for detached steps with DNS patterns in their names.This means that before this change, detached steps could be treated as services (for DNS/hostname purposes). Now, only steps explicitly marked as
StepTypeServiceare treated as services.But this doesn't explain the bug about services not being stopped properly. Let me continue investigating.
Let me look at the
service.gofile more carefully to understand how services are being created and destroyed.Assistant
I can see that the
isServicefunction now only checks forstep.Type == types.StepTypeService. But this doesn't explain the issue with services not being stopped properly.Let me look at the
DestroyWorkflowfunction to understand how services are being destroyed. The issue is that services are not being stopped when the workflow completes, even though the pods are being destroyed.Looking at the
DestroyWorkflowfunction inkubernetes.go:func (e *kube) DestroyWorkflow(ctx context.Context, conf *types.Config, taskUUID string) error { log.Trace().Str("taskUUID", taskUUID).Msg("deleting Kubernetes primitives") for _, stage := range conf.Stages { for _, step := range stage.Steps { err := stopPod(ctx, e, step, e.config.newDefaultDeleteOptions()) if err != nil { return err } } } namespace := e.config.GetNamespace(conf.Stages[0].Steps[0].OrgID) log.Trace().Str("taskUUID", taskUUID).Msgf("deleting workflow headless service") err := e.stopHeadlessService(ctx, e, namespace, taskUUID) if err != nil { return err } log.Trace().Str("taskUUID", taskUUID).Msgf("deleting workflow volume") err = stopVolume(ctx, e, conf.Volume, e.config.GetNamespace(conf.Stages[0].Steps[0].OrgID), e.config.newDefaultDeleteOptions()) if err != nil { return err } return nil }So
DestroyWorkflowiterates over all steps in all stages and callsstopPodfor each step. This should destroy all service pods.But the issue is that the API still shows services as running. This means that the step state in the database is not being updated.
Now, let me think about the flow again:
- Agent runs the pipeline
- When the pipeline completes,
DestroyWorkflowis called, which destroys all pods (including services) - The agent calls
Doneon the server - The server calls
completeChildrenIfParentCompletedto mark all running children as done
So the server should be updating the step state when
Doneis called. But the issue says that services are still showing as running.Wait, let me look at the
UpdateStepToStatusSkippedfunction again:func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }This function updates the step state to
StatusSuccessif the step has started. But wait, the issue is that services are still showing as running. So either:- The
completeChildrenIfParentCompletedfunction is not being called - The
completeChildrenIfParentCompletedfunction is not finding the services - The
completeChildrenIfParentCompletedfunction is not updating the services correctly
Let me look at the
completeChildrenIfParentCompletedfunction again:func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) { for _, c := range completedWorkflow.Children { if c.Running() { if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil { log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID) } else { // Update in-memory state so WorkflowStatus sees the final state c.State = updated.State c.Finished = updated.Finished } } } }So this function iterates over
completedWorkflow.Childrenand callsUpdateStepToStatusSkippedfor each running child.But wait, let me check how
workflow.Childrenis populated. Looking at theDonemethod:workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)And
StepListFromWorkflowFindis:func (s storage) StepListFromWorkflowFind(workflow *model.Workflow) ([]*model.Step, error) { return s.stepListWorkflow(s.engine.NewSession(), workflow) } func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) { stepList := make([]*model.Step, 0) return stepList, sess. Where("pipeline_id = ?", workflow.PipelineID). Where("ppid = ?", workflow.PID). OrderBy("pid"). Find(&stepList) }So this function finds all steps where
ppid = workflow.PID. This means it's looking for steps whose parent process ID matches the workflow's PID.But wait, I need to understand the parent-child relationship better. Let me check how steps are created and how their PPID is set.
Actually, looking at the model, I see that:
- Workflow has a PID
- Steps have a PPID (parent process ID) and a PID
So the parent-child relationship is:
- Workflow (PID=X) has children Steps (PPID=X)
But wait, looking at the code again, I notice that the
Stepmodel has bothPIDandPPID:type Step struct { // ... PID int `json:"pid" xorm:"UNIQUE(s) 'pid'"` PPID int `json:"ppid" xorm:"ppid"` // ... }And the
Workflowmodel has aPID:type Workflow struct { // ... PID int `json:"pid" xorm:"UNIQUE(s) 'pid'"` // ... }So the
StepListFromWorkflowFindfunction is looking for steps whereppid = workflow.PID. This means it's looking for steps that are direct children of the workflow.But wait, I need to understand how the PPID is set for services. Let me search for how steps are created.
Actually, looking at the issue description again, the problem is that services are not being cleaned properly. The pods are destroyed, but the API still shows them as running.
This suggests that the
completeChildrenIfParentCompletedfunction is not being called, or it's not finding the services, or it's not updating them correctly.Let me think about this differently. Maybe the issue is that services are created in a different stage or workflow, so they're not found by
StepListFromWorkflowFind.Actually, I think I need to look at how the pipeline and workflow models are structured. Let me search for where steps are created.
Let me think about this more carefully. The issue says:
- Pods are destroyed properly (so
DestroyWorkflowis working) - But API still shows services as running
- These are only cleaned after the pipeline times out
This suggests that the step state is not being updated when the workflow completes. So either:
- The
DoneRPC call is not being made - The
DoneRPC call is being made, butcompleteChildrenIfParentCompletedis not finding the services - The
DoneRPC call is being made, andcompleteChildrenIfParentCompletedis finding the services, but it's not updating them correctly
Let me look at the issue from a different angle. Maybe the problem is that services are not direct children of the workflow, so they're not found by
StepListFromWorkflowFind.Actually, I think I found the issue! Looking at the
stepListWorkflowfunction:func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) { stepList := make([]*model.Step, 0) return stepList, sess. Where("pipeline_id = ?", workflow.PipelineID). Where("ppid = ?", workflow.PID). OrderBy("pid"). Find(&stepList) }This function is looking for steps where
ppid = workflow.PID. But wait, what if services have a different PPID?Let me search for how steps are created and how their PPID is set.
Actually, let me search for where the Step model is populated.
<tool_call> <function=bash_terminal> <parameter=command> find /woodpecker/server -name "*.go" -exec grep -l "Step{" {} ; | head -20 </tool_call>
Assistant
I need to properly format my bash command to search for where steps are created.
Assistant
Let me look at the items.go file to understand how steps are created and how their PPID is set.
Assistant
Now I can see the key part! In the
saveWorkflowsFromPipelineBuilderfunction, I can see how steps are created and their PPID is set:step := &model.Step{ Name: step.Name, UUID: step.UUID, PipelineID: pipeline.ID, PID: pidSequence, PPID: item.Workflow.PID, // <-- This is set to the workflow's PID State: model.StatusPending, Failure: step.Failure, Type: model.StepType(step.Type), }So all steps (including services) have their PPID set to the workflow's PID. This means they should all be found by
StepListFromWorkflowFind.But wait, the issue says services are still running. Let me think about this more carefully.
Looking at the code flow:
- When
Doneis called on the server, it loads the workflow and its children usingStepListFromWorkflowFind - It then calls
completeChildrenIfParentCompletedwhich iterates overworkflow.Childrenand marks running children as done - This should update all steps, including services
But the issue is that services are still showing as running. This suggests that either:
- The
Donecall is not being made - The services are not being found by
StepListFromWorkflowFind - The services are not being updated by
completeChildrenIfParentCompleted
Wait, I think I need to look at the issue from a different angle. Let me look at the
UpdateStepToStatusSkippedfunction again:func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }This function sets
step.State = status(which isStatusKilled), but then ifstep.Started != 0, it setsstep.State = model.StatusSuccess. So if the service has started (which it should), it will be marked asStatusSuccess.But wait, the issue says services are still showing as running. So maybe the
completeChildrenIfParentCompletedfunction is not being called?Actually, I think I need to look at the issue from a different angle. Maybe the problem is that the services are not being found because they have a different PPID.
Let me look at the pipeline structure again. The issue says the pipeline has services defined in the YAML. So the services should be part of the same workflow as the steps.
Actually, wait. Let me re-read the issue description:
pr-build > clone (#2): Step: clone Started: 1779195580 Stopped: 1779195594 Type: clone State: success pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: running pr-build > wait-for-postgres (#4): Step: wait-for-postgres Started: 1779195596 Stopped: 1779195602 Type: commands State: successSo we have:
- clone (#2): PID=2, success
- postgres (#3): PID=3, service, running
- wait-for-postgres (#4): PID=4, success
All of these should be part of the same workflow, so they should all have the same PPID.
But wait, the issue says that
Stopped: 0for the postgres service. This suggests that theFinishedfield is not being set.Let me look at the
UpdateStepToStatusSkippedfunction again:func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }So if
step.Started != 0, it setsstep.Finished = finished. But the issue saysStopped: 0, which meansFinishedis not being set.This suggests that
step.Started == 0for the service. But that doesn't make sense, because the issue saysStarted: 1779195594for the postgres service.Wait, let me look at the model again. The
Stepmodel has:Started int64- when the step startedFinished int64- when the step finished
And the
UpdateStepToStatusSkippedfunction checks ifstep.Started != 0. If it's not zero, it setsstep.Finished = finished.But the issue says
Stopped: 0, which meansFinishedis 0. So either:step.Started == 0whenUpdateStepToStatusSkippedis calledUpdateStepToStatusSkippedis not being called
Actually, I think I see the issue now! Let me look at the
UpdateStepToStatusSkippedfunction again:func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }Wait, the first line is
step.State = status. So it sets the state tostatus(which isStatusKilled). But then, ifstep.Started != 0, it setsstep.State = model.StatusSuccess.So if the service has started, it should be marked as
StatusSuccessandFinishedshould be set. But the issue says the service is stillrunningandStopped: 0.This suggests that
UpdateStepToStatusSkippedis not being called for the service. Or maybe the service is not being found byStepListFromWorkflowFind.Actually, I think I need to look at this more carefully. Let me check if there's a recent change to the code that might have caused this regression.
Let me search for recent changes to the
completeChildrenIfParentCompletedfunction orUpdateStepToStatusSkipped.Actually, let me think about this differently. The issue says this is a regression with newer releases (3.14.x). So something changed in 3.14.x that broke this functionality.
Let me look at the git history to see what changed recently.
Actually, I think I see the issue now! Let me look at the
stepListWorkflowfunction again:func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) { stepList := make([]*model.Step, 0) return stepList, sess. Where("pipeline_id = ?", workflow.PipelineID). Where("ppid = ?", workflow.PID). OrderBy("pid"). Find(&stepList) }This function is looking for steps where
ppid = workflow.PID. But what if the workflow's PID is not set correctly?Let me look at how the workflow's PID is set. In the
saveWorkflowsFromPipelineBuilderfunction:workflow := &model.Workflow{ ID: item.Workflow.ID, Name: item.Workflow.Name, PID: item.Workflow.PID, // <-- This is set from item.Workflow.PID PipelineID: pipeline.ID, State: model.StatusPending, Environ: item.Workflow.Environ, AxisID: item.Workflow.AxisID, }So the workflow's PID is set from
item.Workflow.PID. And the step's PPID is also set fromitem.Workflow.PID:step := &model.Step{ // ... PPID: item.Workflow.PID, // ... }So all steps in the same workflow should have the same PPID, which is the workflow's PID. This means
StepListFromWorkflowFindshould find all steps, including services.But wait, I think I see the issue now! Let me look at the issue description again:
pr-build > clone (#2): Step: clone Started: 1779195580 Stopped: 1779195594 Type: clone State: success pr-build > postgres (#3): Step: postgres Started: 1779195594 Stopped: 0 Type: service State: running pr-build > wait-for-postgres (#4): Step: wait-for-postgres Started: 1779195596 Stopped: 1779195602 Type: commands State: successThe numbers in parentheses (#2, #3, #4) are the step IDs, not the PIDs. So the step IDs are 2, 3, and 4.
But the PIDs are different from the IDs. Let me look at the model again:
type Step struct { ID int64 `json:"id" xorm:"pk autoincr 'id'"` // ... PID int `json:"pid" xorm:"UNIQUE(s) 'pid'"` PPID int `json:"ppid" xorm:"ppid"` // ... }So
IDis the primary key (auto-incremented), andPIDis a separate field that's used for ordering and parent-child relationships.So the issue is not with the IDs, but with the PIDs. And the PIDs should be set correctly based on the code I've seen.
Let me think about this differently. Maybe the issue is that the
completeChildrenIfParentCompletedfunction is being called, but it's not updating the services because they're not in theworkflow.Childrenlist.Actually, I think I need to look at the
Donefunction more carefully. Let me re-read it:func (s *RPC) Done(c context.Context, strWorkflowID string, state rpc.WorkflowState) error { // ... workflow.Children, err = s.store.StepListFromWorkflowFind(workflow) if err != nil { return err } // ... s.completeChildrenIfParentCompleted(workflow, state.Finished) // ... }So the
Donefunction loads the workflow's children usingStepListFromWorkflowFind, and then callscompleteChildrenIfParentCompleted.But wait, I think I see the issue now! Let me look at the
checkWorkflowStatefunction:func checkWorkflowState(state model.StatusValue) (err error) { switch state { case model.StatusCreated, model.StatusPending, model.StatusRunning: return nil case model.StatusBlocked: err = ErrAgentIllegalWorkflowRun default: err = ErrAgentIllegalWorkflowReRunStateChange } // ... }This function checks if the workflow's state allows it to be initialized or marked as done. If the workflow is in a terminal state (success, failure, killed, etc.), it returns an error.
But wait, the
Donefunction callscheckWorkflowStatebefore callingcompleteChildrenIfParentCompleted:// check workflow's own state to prevent finishing an already-finished or blocked workflow if err := checkWorkflowState(workflow.State); err != nil { return err } // ... s.completeChildrenIfParentCompleted(workflow, state.Finished)So if the workflow is in a terminal state, the
Donefunction will return early andcompleteChildrenIfParentCompletedwill not be called.But wait, the issue says that the pipeline is still running (not completed), and the services are still running. So the workflow should be in a running state, and
checkWorkflowStateshould not return an error.Actually, I think I need to look at the issue from a different angle. Let me think about what happens when the pipeline completes:
- All blocking steps complete
- The stage completes (detached steps return immediately)
destroyWorkflowFunc()is called, which destroys all pods (including services)- The agent calls
Doneon the server - The server calls
completeChildrenIfParentCompletedto mark all running children as done
But wait, what if the agent doesn't call
Doneon the server? Or what if the agent callsDonebefore the services have started?Actually, I think I see the issue now! Let me look at the runtime code again:
func (r *Runtime) Run(runnerCtx context.Context) error { // ... for _, stage := range r.spec.Stages { stageChan := r.runStage(runnerCtx, stage.Steps) // ... } // Now we can shutdown the workflow destroyWorkflowFunc() // Ensure all logs/traces are uploaded before finishing logger.Debug().Msg("waiting for logs and traces upload") r.uploadWait.Wait() // ... }So the runtime waits for all stages to complete, then calls
destroyWorkflowFunc(), then waits for all uploads to complete.And the agent calls
Doneafter the runtime completes:// Run pipeline err = pipeline_runtime.New(...).Run(runnerCtx) state.Finished = time.Now().Unix() // ... if err := r.client.Done(doneCtx, workflow.ID, state); err != nil { // ... }So the agent calls
Doneafter the runtime completes, which means afteruploadWait.Wait()has returned.But wait, what if the
uploadWait.Wait()doesn't wait for the detached steps to complete? Let me look at therunDetachedStepfunction again:func (r *Runtime) runDetachedStep(runnerCtx context.Context, step *backend_types.Step) error { waitForLogs, startTime, err := r.startStep(step) if err != nil { return r.traceStep(nil, err, step) } r.uploadWait.Add(1) go func() { defer r.uploadWait.Done() // ... }() return nil }So the detached step increments
uploadWaitand starts a goroutine that callsuploadWait.Done()when it completes.But wait, the goroutine calls
completeStep, which callsWaitStepandDestroyStep. If the pod is already destroyed bydestroyWorkflowFunc(), thenWaitStepmight fail.Let me look at the
completeStepfunction again:func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) { // Drain the log stream before waiting on the process exit. waitForLogs() waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck if err != nil { if errors.Is(err, context.Canceled) { if waitState == nil { waitState = &backend_types.State{} } waitState.Error = pipeline_errors.ErrCancel } else { return nil, err } } // Use runnerCtx here: the workflow context may already be canceled but we // still need to reach the backend to stop/remove the container. if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil { return nil, err } // ... }So if
WaitStepfails with an error that's notcontext.Canceled, it returnsnil, err. And then the goroutine logs the error and callstraceStepwith the error.But wait, what if
WaitStepfails because the pod is already destroyed? Let me look at theWaitStepfunction in the Kubernetes backend:func (e *kube) WaitStep(ctx context.Context, step *types.Step, taskUUID string) (*types.State, error) { // ... pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{}) if err != nil { return nil, err } // ... }So if the pod is not found (because it was destroyed),
WaitStepwill return an error. And thencompleteStepwill returnnil, err.And then the goroutine will call
traceStep(processState, err, step)withprocessState = nilanderr != nil.Let me look at the
traceStepfunction:func (r *Runtime) traceStep(processState *backend_types.State, err error, step *backend_types.Step) error { s := new(state.State) s.Workflow.Started = r.started s.CurrStep = step s.Workflow.Error = r.err.Get() switch { case processState == nil && err != nil: // Step failed to start — create an dummy exited process state. s.CurrStepState = backend_types.State{ Error: err, Exited: true, OOMKilled: false, } case processState != nil: s.CurrStepState = *processState // processState == nil && err == nil: step just started, leave s.CurrStepState zero-valued. } if traceErr := r.tracer.Trace(s); traceErr != nil { logger := r.makeLogger() logger.Error().Err(traceErr).Msg("could not trace step state change") } return err }So if
processState == nil && err != nil, it creates a dummy exited process state withExited: true. And then it callsr.tracer.Trace(s)to send the state to the server.But wait, what if the tracer fails to send the state to the server? Or what if the server doesn't update the step state correctly?
Actually, I think I see the issue now! Let me look at the sequence of events again:
- All blocking steps complete
- The stage completes (detached steps return immediately)
destroyWorkflowFunc()is called, which destroys all pods (including services)- The background goroutines for services try to call
WaitStepon the destroyed pods WaitStepfails because the pod is not foundcompleteStepreturnsnil, err- The goroutine calls
traceStep(nil, err, step)withprocessState = nilanderr != nil traceStepcreates a dummy exited process state and callsr.tracer.Trace(s)- The tracer sends the state to the server via RPC
- The server updates the step state
But wait, step 9 happens AFTER
destroyWorkflowFunc()has been called. And the agent callsDoneafteruploadWait.Wait()has returned, which means after all background goroutines have completed.So the sequence should be:
- Blocking steps complete
- Stage completes
destroyWorkflowFunc()is called- Background goroutines try to call
WaitStepand fail - Background goroutines call
traceStepto report the failure uploadWait.Wait()returns- Agent calls
Doneon the server - Server calls
completeChildrenIfParentCompletedto mark all running children as done
So the services should be updated by either:
- The
traceStepcall in the background goroutine (if it succeeds) - The
completeChildrenIfParentCompletedcall in theDonefunction
But the issue says the services are still running. So either:
- The
traceStepcall fails - The
completeChildrenIfParentCompletedcall doesn't find the services
Actually, I think I need to look at the tracer to understand how it sends the state to the server.
Actually, I think I found the bug! Let me look at the
completeChildrenIfParentCompletedfunction again:func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) { for _, c := range completedWorkflow.Children { if c.Running() { if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil { log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID) } else { // Update in-memory state so WorkflowStatus sees the final state c.State = updated.State c.Finished = updated.Finished } } } }This function iterates over
completedWorkflow.Childrenand callsUpdateStepToStatusSkippedfor each running child.But wait, I think I see the issue now! The function checks
if c.Running(). Let me look at theRunningfunction:func (p *Step) Running() bool { return p.State == StatusPending || p.State == StatusRunning }So
Running()returnstrueif the step's state isStatusPendingorStatusRunning.But what if the step's state is not
StatusPendingorStatusRunning? For example, what if the step's state isStatusSuccess(because thetraceStepcall succeeded)?In that case,
c.Running()would returnfalse, andUpdateStepToStatusSkippedwould not be called.But wait, the issue says the services are still running. So the step's state should be
StatusRunning, andc.Running()should returntrue.Actually, I think I need to look at the issue from a different angle. Let me think about what could cause the services to still be running after the pipeline completes.
One possibility is that the
traceStepcall fails to update the step state, and thecompleteChildrenIfParentCompletedfunction doesn't find the services.But why would
completeChildrenIfParentCompletednot find the services? The services should be inworkflow.Childrenbecause they have the same PPID as the workflow.Wait, I think I see the issue now! Let me look at the
StepListFromWorkflowFindfunction again:func (s storage) StepListFromWorkflowFind(workflow *model.Workflow) ([]*model.Step, error) { return s.stepListWorkflow(s.engine.NewSession(), workflow) } func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) { stepList := make([]*model.Step, 0) return stepList, sess. Where("pipeline_id = ?", workflow.PipelineID). Where("ppid = ?", workflow.PID). OrderBy("pid"). Find(&stepList) }This function queries the database for steps where
pipeline_id = workflow.PipelineIDandppid = workflow.PID.But wait, what if the workflow's PID is 0 or not set? In that case, the query would not find any steps!
Let me look at how the workflow's PID is set. In the
saveWorkflowsFromPipelineBuilderfunction:workflow := &model.Workflow{ ID: item.Workflow.ID, Name: item.Workflow.Name, PID: item.Workflow.PID, // <-- This is set from item.Workflow.PID PipelineID: pipeline.ID, State: model.StatusPending, Environ: item.Workflow.Environ, AxisID: item.Workflow.AxisID, }So the workflow's PID is set from
item.Workflow.PID. Anditemis abuilder.Item.Let me look at the builder package to understand how the workflow's PID is set.
Actually, wait. I think I found the issue! Let me look at the issue description again:
"Some regression with newer releases (3.14.x)"
This suggests that something changed in 3.14.x that broke this functionality. And I saw a commit
e173bed3fthat changed theisServicefunction. But that doesn't seem to be related to the issue.Let me search for other recent changes to the server/pipeline package.
<tool_call> <function=bash_terminal> <parameter=command> cd /woodpecker && git log --oneline --all --since="2024-01-01" -- server/pipeline/ | head -30 </tool_call>
- When
Assistant
The user is asking me to run a git command to look at recent changes to the server/pipeline directory. Let me properly format this bash command.
Assistant
I found a very relevant commit:
855eae143 Set workflow services stuck in running state to finished (#6337). This looks like it's addressing the exact issue described! Let me examine this commit.Assistant
I found a commit that seems to address the exact issue. Let me look at the full diff to understand what was changed and if it's the fix I need to apply or if there's still something missing.
Assistant
Excellent! I found the commit
855eae143that addresses this exact issue! This commit was made on March 30, 2026. Let me check if this fix is already in the current codebase or if I need to apply it.Assistant
The issue is about services remaining in a "running" state after pipeline completion with Kubernetes backend. The commit 855eae143 already contains a fix for this issue. Let me check if this commit is already in the current branch or if it's in a different branch/commit that hasn't been merged yet.
Let me check if the current HEAD already has this fix applied.
Assistant
Let me verify that the fix from commit 855eae143 is actually applied in the current code by checking the rpc.go file:
Assistant
The user is asking me to continue debugging. I found the commit 855eae143 that appears to fix the issue, but I need to verify if this fix is actually present in the current codebase and if there's still an issue. Let me check the current state of the code and see if the fix from commit 855eae143 is already applied or if there's still something missing.
Let me verify the current state of the code by looking at the key functions mentioned in the commit.
Assistant
Good, I can see the fix is already present in the codebase. The
completeChildrenIfParentCompletedfunction is being called withstate.FinishedbeforeUpdateWorkflowStatusToDone. This is the fix from commit 855eae143.Now I need to verify if the fix is complete. Let me check the
completeChildrenIfParentCompletedfunction to see if it's using the correct parameters and logic.Assistant
The user seems to want me to continue investigating the issue. Let me check the current state of the code.
From what I can see:
- The commit 855eae143 is already in the current branch (it's in the git log)
- The
completeChildrenIfParentCompletedfunction is already being called withstate.Finishedparameter - The function updates the in-memory state of running children
But wait, the issue says "Some regression with newer releases (3.14.x)". So the issue was reported AFTER version 3.14.x, which means the fix in commit 855eae143 might not be in 3.14.x yet, or there might be a new regression after the fix.
Let me check the current code more carefully to see if there's still an issue.
Actually, looking at the code, I see that the fix is already in place. But the issue description mentions that this is a regression in 3.14.x, and the fix (855eae143) was made on March 30, 2026. So the fix should already be in the current codebase.
Let me check if there's something else that might be causing the issue. Maybe the issue is that the
StepListFromWorkflowFindfunction is not finding the services, or maybe the services are not being added toworkflow.Childrencorrectly.Let me look at the
StepListFromWorkflowFindfunction again to see if there's an issue with how it finds services.Actually, I think I need to look at the issue more carefully. The issue says:
- Pods are destroyed properly (so
DestroyWorkflowis working) - But API still shows services as running
So the pods are being destroyed, but the step state is not being updated. This suggests that the
completeChildrenIfParentCompletedfunction is not being called, or it's not finding the services, or it's not updating them correctly.Let me check if the
StepListFromWorkflowFindfunction is finding all steps, including services. Looking at the function:func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) { stepList := make([]*model.Step, 0) return stepList, sess. Where("pipeline_id = ?", workflow.PipelineID). Where("ppid = ?", workflow.PID). OrderBy("pid"). Find(&stepList) }This function is looking for steps where
pipeline_id = workflow.PipelineIDandppid = workflow.PID. But what if the workflow's PID is 0 or not set? In that case, the query would not find any steps!Let me check how the workflow's PID is set when it's loaded from the database. Maybe the PID is not being loaded correctly.
Actually, looking at the Done function, I see that it loads the workflow using
s.store.WorkflowLoad(workflowID). Let me check if this function loads the PID correctly.Wait, I think I need to look at the issue from a different angle. The issue says that the services are still running after the pipeline completes. But the fix in commit 855eae143 should have addressed this issue by calling
completeChildrenIfParentCompletedbeforeUpdateWorkflowStatusToDone.Let me check if there's a case where
completeChildrenIfParentCompletedis not finding the services. Maybe the services are not inworkflow.Childrenbecause they have a different PPID?Actually, I just realized something. Looking at the
saveWorkflowsFromPipelineBuilderfunction initems.go, I see that all steps (including services) are added toworkflow.Children:for _, stage := range item.Config.Stages { for _, step := range stage.Steps { pidSequence++ step := &model.Step{ // ... PPID: item.Workflow.PID, // ... } workflow.Children = append(workflow.Children, step) } }So all steps should have the same PPID as the workflow's PID, and they should all be in
workflow.Children.But wait, the
Donefunction loads the workflow's children from the database usingStepListFromWorkflowFind:workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)So the children are loaded from the database, not from the in-memory
workflow.Childrenlist. And the query is:Where("pipeline_id = ?", workflow.PipelineID). Where("ppid = ?", workflow.PID).So if the workflow's PID is set correctly, the query should find all steps, including services.
Let me check if there's an issue with how the workflow's PID is loaded from the database. Maybe the PID is not being loaded correctly when the workflow is loaded.
Actually, I think I need to look at the Workflow model to see if the PID field is being loaded correctly.
Wait, I just realized something. The issue says "Some regression with newer releases (3.14.x)". And the fix (855eae143) was made on March 30, 2026. So the fix might already be in the current codebase, but the issue might still be happening because of a different reason.
Let me re-read the issue description more carefully:
"Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
So the pods are being destroyed, but the step state is not being updated. This suggests that the
completeChildrenIfParentCompletedfunction is not being called, or it's not finding the services, or it's not updating them correctly.But wait, the fix in commit 855eae143 should have addressed this issue. So maybe the issue is still happening because of a different reason.
Let me think about this more carefully. The issue says that the services are still running after the pipeline completes. But the fix in commit 855eae143 should have addressed this issue by calling
completeChildrenIfParentCompletedbeforeUpdateWorkflowStatusToDone.So if the fix is already in the current codebase, why is the issue still happening?
Actually, I think I need to look at the issue from a different angle. Maybe the issue is that the
StepListFromWorkflowFindfunction is not finding the services because they have a different PPID than the workflow's PID.Let me check how the PPID is set for services. Looking at the
saveWorkflowsFromPipelineBuilderfunction, I see that all steps have their PPID set toitem.Workflow.PID:PPID: item.Workflow.PID,So all steps should have the same PPID as the workflow's PID.
But wait, what if the workflow's PID is 0 or not set? In that case, the query would not find any steps!
Let me check how the workflow's PID is set. Looking at the
saveWorkflowsFromPipelineBuilderfunction, I see that the workflow's PID is set fromitem.Workflow.PID:PID: item.Workflow.PID,And
itemis abuilder.Item. Let me check how the workflow's PID is set in the builder.Actually, I think I need to look at the builder package to understand how the workflow's PID is set.
Actually, wait. I just realized something. The issue might be related to how the workflow's PID is loaded from the database. When the workflow is loaded using
WorkflowLoad, maybe the PID is not being loaded correctly?Let me check the
WorkflowLoadfunction to see how it loads the workflow from the database.Actually, I think I'm overcomplicating this. Let me just run the tests to see if there are any failures related to this issue.
Actually wait, I should check if the
UpdateStepToStatusSkippedfunction is being called correctly. Let me look at the function again:func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }So if
step.Started != 0, it setsstep.State = model.StatusSuccessandstep.Finished = finished. But the issue saysStopped: 0, which meansFinishedis 0.So either:
step.Started == 0whenUpdateStepToStatusSkippedis calledUpdateStepToStatusSkippedis not being called
But the issue says
Started: 1779195594for the postgres service. Sostep.Startedshould not be 0.Wait, I think I see the issue now! The
UpdateStepToStatusSkippedfunction is being called, but it's not updating the step correctly. Let me look at the function again:func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }So if
step.Started != 0, it setsstep.State = model.StatusSuccessandstep.Finished = finished. This should work correctly.But wait, what if the step's
Startedfield is 0 in the database, even though the API showsStarted: 1779195594? Maybe theStartedfield is not being set correctly when the step is created?Let me check how the step's
Startedfield is set. Looking at theCalcStepStatusfunction instep_status.go:case model.StatusPending: // ... // Transition from pending to running when started if state.Finished == 0 { step.State = model.StatusRunning } step.Started = state.Started if step.Started == 0 { step.Started = time.Now().Unix() } // ...So the step's
Startedfield is set fromstate.Started, which is passed from the agent. Ifstate.Startedis 0, it's set totime.Now().Unix().So the step's
Startedfield should be set correctly.But wait, I think I see the issue now! Looking at the
CalcStepStatusfunction, I see that the step'sStartedfield is only set when the step's state ismodel.StatusPending. If the step's state ismodel.StatusRunning, theStartedfield is not set.So if the step's state is
model.StatusRunningwhenCalcStepStatusis called, theStartedfield might not be set correctly.But wait, the step's state should be
model.StatusPendingwhen it's first created, and then it's updated tomodel.StatusRunningwhen it starts. So theStartedfield should be set correctly.Actually, I think I need to look at the issue from a different angle. Maybe the issue is that the
UpdateStepToStatusSkippedfunction is being called, but the step is not being updated correctly in the database.Let me look at the
store.StepUpdatefunction to see how it updates the step in the database.Actually, I think I'm overcomplicating this. Let me just check if there's a test that covers this scenario and see if it's passing.
Actually, I just realized something. Looking at the commit 855eae143, I see that the fix is to call
completeChildrenIfParentCompletedBEFOREUpdateWorkflowStatusToDone, and to passstate.Finishedinstead ofcompletedWorkflow.Finished.But in the current code, I see that
completeChildrenIfParentCompletedis being called withstate.Finished:s.completeChildrenIfParentCompleted(workflow, state.Finished)So the fix is already in place.
But wait, the issue says "Some regression with newer releases (3.14.x)". So the issue was reported for version 3.14.x, which might not have the fix yet.
Let me check if the fix is in the current codebase by looking at the git log.
Actually, I already checked the git log, and the commit 855eae143 is in the current branch. So the fix should already be in place.
But the issue might still be happening because of a different reason. Let me think about this more carefully.
Actually, I think I need to look at the issue from a different angle. The issue says "the woodpecker API still show these steps as state=running". This suggests that the step state is not being updated in the database, or the API is not returning the correct state.
Let me check if there's an issue with how the API returns the step state.
Actually, I think I need to create a test script to reproduce the issue and see what's happening.
Wait, I just realized something. The issue might be related to how the services are being traced. Looking at the
runDetachedStepfunction, I see that the detached step (service) is traced in a goroutine:go func() { defer r.uploadWait.Done() logger := r.makeLogger() processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime) // ... if traceErr := r.traceStep(processState, err, step); traceErr != nil { logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result") } }()So the service's state is traced in a goroutine. And the agent calls
DoneafteruploadWait.Wait()has returned, which means after all background goroutines have completed.So the service's state should be traced before the agent calls
Done. And the server should update the step state when the trace is received.But what if the trace is not being sent correctly? Or what if the server is not updating the step state correctly?
Actually, I think I need to look at the tracer to understand how it sends the state to the server.
Let me search for the tracer implementation.
Actually, I think I see the issue now! Looking at the
completeStepfunction, I see that it callsWaitStepandDestroyStep. But if the pod is already destroyed byDestroyWorkflow, thenWaitStepmight fail.And if
WaitStepfails,completeStepreturnsnil, err. And then the goroutine callstraceStep(nil, err, step)withprocessState = nilanderr != nil.And looking at the
traceStepfunction, I see that ifprocessState == nil && err != nil, it creates a dummy exited process state:case processState == nil && err != nil: // Step failed to start — create an dummy exited process state. s.CurrStepState = backend_types.State{ Error: err, Exited: true, OOMKilled: false, }So the tracer should send a state with
Exited: true. And the server should update the step state when the trace is received.But what if the tracer fails to send the state to the server? In that case, the step state would not be updated, and the
completeChildrenIfParentCompletedfunction would need to update it.But the
completeChildrenIfParentCompletedfunction should be updating the step state correctly. So why is the issue still happening?Actually, I think I need to look at the issue from a different angle. Maybe the issue is that the services are not in
workflow.ChildrenwhencompleteChildrenIfParentCompletedis called.Let me check how
workflow.Childrenis populated. Looking at theDonefunction, I see that it loads the workflow's children usingStepListFromWorkflowFind:workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)So the children are loaded from the database using the workflow's PipelineID and PID. If the workflow's PID is set correctly, the query should find all steps, including services.
But what if the workflow's PID is not set correctly? Let me check how the workflow's PID is loaded from the database.
Actually, I think I see the issue now! Looking at the
WorkflowLoadfunction, I need to check if it loads the PID correctly.Let me search for the
WorkflowLoadfunction.Actually, I think I need to look at the store implementation to see how the workflow is loaded from the database.
Let me search for the
WorkflowLoadfunction in the store package.OK so I've been doing a lot of analysis. Let me now actually check if there's something wrong in the codebase. The commit 855eae143 should have fixed this issue. But the issue description says it's happening with version 3.14.x, so maybe the fix is not in that version.
Let me check the current state of the code to see if there are any issues.
Actually, I realize now that the codebase I'm looking at already has the fix from commit 855eae143. So if the issue is still happening, it might be because:
- The fix is not sufficient
- There's a different bug causing the same symptoms
- The issue was reported for an older version that doesn't have the fix
Let me look more carefully at the
UpdateStepToStatusSkippedfunction to see if there's an issue with how it updates the step state.Looking at the function:
func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }This function sets
step.State = statusfirst, and then ifstep.Started != 0, it setsstep.State = model.StatusSuccessandstep.Finished = finished.But wait, the
statusparameter ismodel.StatusKilledin the call fromcompleteChildrenIfParentCompleted:if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {So the function first sets
step.State = model.StatusKilled, and then ifstep.Started != 0, it setsstep.State = model.StatusSuccess.This seems correct. If the step has started, it should be marked as
StatusSuccess(because it was killed after starting, which is expected for services). And if the step has not started, it should be marked asStatusKilled.So the function seems to be working correctly.
But wait, I just realized something. The issue says "Stopped: 0", which means
Finishedis 0. But theUpdateStepToStatusSkippedfunction setsstep.Finished = finishedifstep.Started != 0.So if
step.Started != 0,step.Finishedshould be set. But the issue saysStopped: 0, which meansstep.Finishedis 0.This suggests that either:
step.Started == 0whenUpdateStepToStatusSkippedis calledUpdateStepToStatusSkippedis not being called
Let me think about this more carefully. The issue says
Started: 1779195594for the postgres service. So the step'sStartedfield should be set in the database.But what if the step's
Startedfield is not being loaded correctly from the database? Maybe theStepListFromWorkflowFindfunction is not loading theStartedfield?Actually, looking at the
stepListWorkflowfunction, I see that it uses xorm'sFindmethod:return stepList, sess. Where("pipeline_id = ?", workflow.PipelineID). Where("ppid = ?", workflow.PID). OrderBy("pid"). Find(&stepList)The
Findmethod should load all fields of the Step model, includingStarted. So theStartedfield should be loaded correctly.But wait, what if the step's
Startedfield is 0 in the database? Maybe theStartedfield is not being set when the step is created?Let me check how the step's
Startedfield is set when the step is created. Looking at thesaveWorkflowsFromPipelineBuilderfunction, I see that the step is created with:step := &model.Step{ // ... State: model.StatusPending, // ... }So the step is created with
State: model.StatusPending. And theStartedfield is not set, so it defaults to 0.Then, when the step starts, the agent sends a trace to the server with
state.Started. And the server updates the step'sStartedfield using theCalcStepStatusfunction.So the step's
Startedfield should be set correctly when the step starts.But what if the step's
Startedfield is not being updated in the database? Maybe there's an issue with how the step is updated?Actually, I think I see the issue now! Looking at the
CalcStepStatusfunction, I see that it updates the step's state and returns the updated step. But it doesn't update the step in the database. The database update is done by theUpdateStepStatusfunction:func UpdateStepStatus(ctx context.Context, store store.Store, step *model.Step, state rpc.StepState) error { log.Debug().Str("StepUUID", step.UUID).Msgf("Update step %#v state %#v", *step, state) updatedStep, shouldCancelPipelineFromStep, err := CalcStepStatus(*step, state) if err != nil { return err } *step = *updatedStep // update step for external callers if shouldCancelPipelineFromStep { if err := cancelPipelineFromStep(ctx, store, step); err != nil { return err } } return store.StepUpdate(step) }So the
UpdateStepStatusfunction callsCalcStepStatusto calculate the updated step, and then it callsstore.StepUpdate(step)to update the step in the database.But wait, I see that
CalcStepStatusreturnsupdatedStep, and then*step = *updatedStepis used to update the step for external callers. And thenstore.StepUpdate(step)is called to update the step in the database.So the step should be updated correctly in the database.
Hmm, I'm not sure what the issue is. Let me re-read the issue description more carefully.
"Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
So the pods are being destroyed, but the step state is still "running". This suggests that the
completeChildrenIfParentCompletedfunction is not updating the step state correctly, or the API is not returning the correct state.Actually, I think I need to look at the API to see how it returns the step state. Maybe the API is caching the step state, or maybe it's returning the wrong step.
Let me search for the API endpoint that returns the step state.
Actually wait, let me re-check the git history. The commit 855eae143 says "Set workflow services stuck in running state to finished (#6337)". This suggests that the fix was made to address this exact issue.
Let me check if this commit is in the main branch or if it's in a different branch.
Actually, I already checked the git log, and the commit is in the current branch. So the fix should already be in place.
But maybe the fix is not complete, and there's still an issue. Let me look at the
completeChildrenIfParentCompletedfunction more carefully to see if there's an issue.Actually, I just realized something. Looking at the function:
func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) { for _, c := range completedWorkflow.Children { if c.Running() { if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil { log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID) } else { // Update in-memory state so WorkflowStatus sees the final state c.State = updated.State c.Finished = updated.Finished } } } }This function iterates over
completedWorkflow.Childrenand callsUpdateStepToStatusSkippedfor each running child. And it updates the in-memory state of the child so thatWorkflowStatussees the final state.But wait, what if
completedWorkflow.Childrenis empty or doesn't contain the services? In that case, the services would not be updated.Let me check how
completedWorkflow.Childrenis populated. Looking at theDonefunction, I see that it loads the workflow's children usingStepListFromWorkflowFind:workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)So the children are loaded from the database using the workflow's PipelineID and PID. If the workflow's PID is set correctly, the query should find all steps, including services.
But what if the workflow's PID is not set correctly? Let me check how the workflow's PID is loaded from the database.
Actually, I think I need to look at the store implementation to see how the workflow is loaded from the database. Let me search for the
WorkflowLoadfunction.<tool_call> <function=bash_terminal> <parameter=command> grep -r "WorkflowLoad" /woodpecker/server/store --include="*.go" -A 10 | head -40 </tool_call>
Assistant
I apologize for the confusion. Let me search for the
WorkflowLoadfunction to understand how the workflow is loaded from the database.Assistant
The user is pointing out that my tool call was empty. Let me look at what I was trying to do - I was investigating the
WorkflowLoadfunction and now I can see the implementation.Looking at the output, I can see that
WorkflowLoaduses xorm'sGetmethod to load the workflow from the database:func (s storage) WorkflowLoad(id int64) (*model.Workflow, error) { workflow := new(model.Workflow) return workflow, wrapGet(s.engine.ID(id).Get(workflow)) }This should load all fields of the Workflow model, including the PID field. So the PID should be loaded correctly.
Now let me think about the issue more carefully. The commit 855eae143 should have fixed the issue by:
- Calling
completeChildrenIfParentCompletedBEFOREUpdateWorkflowStatusToDone - Passing
state.Finishedinstead ofcompletedWorkflow.Finished - Updating the in-memory state of the children so that
WorkflowStatussees the final state
But wait, I notice something in the test output:
assert.Len(t, workflowGet.Children, 0)This suggests that when the workflow is loaded using
WorkflowLoad, theChildrenfield is empty! This is because xorm'sGetmethod only loads the workflow itself, not its children.So the
Donefunction loads the workflow usingWorkflowLoad, and then it loads the children separately usingStepListFromWorkflowFind:workflow, err = s.store.WorkflowLoad(workflowID) // ... workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)So the children are loaded correctly.
But wait, I think I see the issue now! Let me look at the
stepListWorkflowfunction again:func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) { stepList := make([]*model.Step, 0) return stepList, sess. Where("pipeline_id = ?", workflow.PipelineID). Where("ppid = ?", workflow.PID). OrderBy("pid"). Find(&stepList) }This function queries for steps where
pipeline_id = workflow.PipelineIDandppid = workflow.PID. If the workflow's PID is set correctly, this should find all steps, including services.But what if the workflow's PID is 0? In that case, the query would not find any steps!
Let me check how the workflow's PID is set when the workflow is created. Looking at the
saveWorkflowsFromPipelineBuilderfunction initems.go, I see that the workflow's PID is set fromitem.Workflow.PID:workflow := &model.Workflow{ // ... PID: item.Workflow.PID, // ... }So the workflow's PID is set from the builder. Let me check how the builder sets the PID.
Actually, I think I need to look at the builder package to understand how the workflow's PID is set. But first, let me check if there's an issue with how the PID is used in the query.
Wait, I think I see the issue now! Looking at the
stepListWorkflowfunction, I see that it queries for steps whereppid = workflow.PID. But thePPIDfield in the Step model is anint, not anint64. And thePIDfield in the Workflow model is also anint.So if the workflow's PID is 0 (the default value for an int), the query would be
ppid = 0, which would not find any steps (because steps have a non-zero PPID).But wait, the workflow's PID should be set correctly when the workflow is created. So this should not be an issue.
Actually, I think I need to look at the issue from a different angle. Let me check if there's a recent change that might have caused this regression.
Looking at the git log, I see the commit
e4dfbf86c Refactor server/.../step_builder into pipeline/.../builder (#3967). This might be related to the issue.Actually, I think I need to look at the issue more carefully. The issue says "Some regression with newer releases (3.14.x)". And the fix (855eae143) was made on March 30, 2026.
So the issue was reported for version 3.14.x, and the fix was made after that. This suggests that the fix is not in version 3.14.x, and the issue is still happening in that version.
But the current codebase has the fix, so the issue should be resolved in the current version.
Wait, but the user is asking me to fix the issue in the current codebase. So either:
- The fix is not complete, and there's still an issue
- The issue is happening in a different scenario that's not covered by the fix
- The issue is a misunderstanding, and the fix is already in place
Let me re-read the issue description more carefully to understand what the actual problem is.
"Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
So the issue is:
- Pods are destroyed properly (so
DestroyWorkflowis working) - But the API still shows services as running (so the step state is not being updated)
This suggests that the
completeChildrenIfParentCompletedfunction is not updating the step state correctly, or the API is not returning the correct state.But the fix in commit 855eae143 should have addressed this issue by calling
completeChildrenIfParentCompletedbeforeUpdateWorkflowStatusToDone.Wait, I think I need to look at the current code more carefully to see if there's still an issue. Let me check if the fix is actually in place.
Actually, I already checked the current code, and I can see that the fix is in place:
// Complete any still-running children (e.g. service containers) before // computing the workflow status, so their final state is reflected. s.completeChildrenIfParentCompleted(workflow, state.Finished)So the fix is in place. But the issue might still be happening because of a different reason.
Actually, I think I need to look at the
UpdateStepToStatusSkippedfunction more carefully. Let me check if there's an issue with how it updates the step state.Looking at the function:
func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }This function sets
step.State = statusfirst, and then ifstep.Started != 0, it setsstep.State = model.StatusSuccessandstep.Finished = finished.But wait, I see an issue here! The function sets
step.State = statusfirst, which ismodel.StatusKilled. And then, ifstep.Started != 0, it setsstep.State = model.StatusSuccess.So if the step has started, it should be marked as
StatusSuccess. But the issue says the step is still showing asrunning.This suggests that either:
step.Started == 0whenUpdateStepToStatusSkippedis calledUpdateStepToStatusSkippedis not being called- The step is not being found by
StepListFromWorkflowFind
Let me think about this more carefully. The issue says
Started: 1779195594for the postgres service. So the step'sStartedfield should be set in the database.But what if the step's
Startedfield is 0 when the step is loaded from the database? Maybe there's an issue with how the step is loaded?Actually, looking at the
stepListWorkflowfunction, I see that it uses xorm'sFindmethod, which should load all fields of the Step model, includingStarted. So theStartedfield should be loaded correctly.But wait, what if the step's
Startedfield is 0 in the database? Let me check how the step'sStartedfield is updated when the step starts.Looking at the
CalcStepStatusfunction, I see that the step'sStartedfield is set fromstate.Started:step.Started = state.Started if step.Started == 0 { step.Started = time.Now().Unix() }So the step's
Startedfield should be set correctly when the step starts.But wait, I think I see the issue now! The
CalcStepStatusfunction is called byUpdateStepStatus, which is called by theUpdateRPC method:if err := pipeline.UpdateStepStatus(c, s.store, step, state); err != nil { log.Error().Err(err).Msg("rpc.update: cannot update step") }So the step's
Startedfield is updated when the agent sends anUpdateRPC call. But if the agent doesn't send anUpdateRPC call for the service (because it's a detached step), the step'sStartedfield might not be updated!Actually, wait. Looking at the runtime code, I see that the agent sends a trace for the service when it starts:
// Emit a "step started" trace before doing any real work. if err := r.traceStep(nil, nil, step); err != nil { return err }So the agent should send a trace for the service when it starts, and the server should update the step's
Startedfield.But wait, looking at the
traceStepfunction, I see that it callsr.tracer.Trace(s). And the tracer sends the state to the server via RPC. So the server should receive the trace and update the step's state.But what if the trace is not being sent correctly? Or what if the server is not updating the step's state correctly?
Actually, I think I need to look at the tracer implementation to understand how it sends the state to the server.
Hmm, I've been going in circles. Let me take a step back and think about this more carefully.
The issue is that services are still showing as "running" after the pipeline completes. The fix in commit 855eae143 should have addressed this issue by calling
completeChildrenIfParentCompletedbeforeUpdateWorkflowStatusToDone.But the issue might still be happening because:
- The services are not being found by
StepListFromWorkflowFind - The services are being found, but
UpdateStepToStatusSkippedis not updating them correctly - The services are being updated, but the API is not returning the correct state
Let me check if there's an issue with how
StepListFromWorkflowFindfinds the services. Looking at the function:func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) { stepList := make([]*model.Step, 0) return stepList, sess. Where("pipeline_id = ?", workflow.PipelineID). Where("ppid = ?", workflow.PID). OrderBy("pid"). Find(&stepList) }This function queries for steps where
pipeline_id = workflow.PipelineIDandppid = workflow.PID. If the workflow's PID is set correctly, this should find all steps, including services.But what if the workflow's PID is 0? In that case, the query would be
ppid = 0, which would not find any steps (because steps have a non-zero PPID).Let me check how the workflow's PID is set when the workflow is created. Looking at the
saveWorkflowsFromPipelineBuilderfunction, I see that the workflow's PID is set fromitem.Workflow.PID:workflow := &model.Workflow{ // ... PID: item.Workflow.PID, // ... }So the workflow's PID is set from the builder. Let me check how the builder sets the PID.
Actually, I realize I need to look at what's in the builder package. Let me search for how PID is assigned to workflows in the builder.
Actually wait - I think I've been overthinking this. Let me check if there's perhaps a different issue - maybe the problem is that when the workflow is loaded, the PID field is 0 for some reason, or maybe there's a timing issue where the steps haven't been created yet.
Let me look more carefully at the issue. The key thing is that the fix from 855eae143 should already be in the codebase. But maybe there's still an edge case that's not being handled.
Actually, let me re-read the
UpdateStepToStatusSkippedfunction one more time:func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) { step.State = status if step.Started != 0 { step.State = model.StatusSuccess // for daemons that are killed step.Finished = finished } return &step, store.StepUpdate(&step) }Wait! I think I see the issue now! The function name is
UpdateStepToStatusSkipped, but it actually sets the state toStatusSuccessif the step has started. This is confusing, but it's by design (to handle daemons that are killed).But what if the step's
Startedfield is 0? In that case, the state would be set tostatus(which ismodel.StatusKilled), and theFinishedfield would not be set.So if the step's
Startedfield is 0, the step would be marked asStatusKilled, and theFinishedfield would be 0.But the issue says the step is still showing as
running, notkilled. So this suggests thatUpdateStepToStatusSkippedis not being called at all.Let me think about why
UpdateStepToStatusSkippedmight not be called:- The step is not in
workflow.Children - The step is not running (i.e.,
c.Running()returns false)
For the first case, the step might not be in
workflow.Childrenif:- The step's PPID is not equal to the workflow's PID
- The step's PipelineID is not equal to the workflow's PipelineID
For the second case, the step might not be running if:
- The step's state is not
StatusPendingorStatusRunning
But the issue says the step is still showing as
running, so the step's state should beStatusRunning. This meansc.Running()should return true.So the issue is likely that the step is not in
workflow.Children. And this is likely because the step's PPID is not equal to the workflow's PID.Let me check how the step's PPID is set. Looking at the
saveWorkflowsFromPipelineBuilderfunction, I see that the step's PPID is set toitem.Workflow.PID:step := &model.Step{ // ... PPID: item.Workflow.PID, // ... }So all steps in the same workflow should have the same PPID. And the workflow's PID should be set from
item.Workflow.PID.But wait, what if the workflow's PID is 0 when it's loaded from the database? Let me check the Workflow model to see if the PID field is being loaded correctly.
Looking at the Workflow model:
type Workflow struct { ID int64 `json:"id" xorm:"pk autoincr 'id'"` PipelineID int64 `json:"pipeline_id" xorm:"UNIQUE(s) INDEX 'pipeline_id'"` PID int `json:"pid" xorm:"UNIQUE(s) 'pid'"` // ... }So the PID field is an
int, and it's mapped to thepidcolumn in the database. Thexorm:"UNIQUE(s) 'pid'"tag suggests that the PID is unique within the scope (pipeline).So the PID should be loaded correctly from the database.
But wait, what if the PID is 0 for some workflows? Let me check how the PID is set when the workflow is created.
Looking at the
saveWorkflowsFromPipelineBuilderfunction, I see that the workflow's PID is set fromitem.Workflow.PID. Anditemis abuilder.Item.Let me check how the builder sets the workflow's PID. I need to look at the builder package.
Actually, I realize I should check if there's a scenario where the workflow's PID might be 0 or not set correctly. This could happen if:
- The builder doesn't set the PID correctly
- The PID is not saved to the database correctly
- The PID is not loaded from the database correctly
Let me look at the builder package to see how the PID is set.
Actually, I just realized something. Looking at the
saveWorkflowsFromPipelineBuilderfunction, I see that it iterates overpipelineItemsand sets the workflow's PID fromitem.Workflow.PID. But what ifitem.Workflow.PIDis 0?Let me check how the builder sets the PID. I need to look at the builder package.
Actually, let me just check the current state of the code and see if there's an obvious issue. I'll look at the builder package.
Let me search for where Workflow.PID is set in the builder.
OK I think I need to trace through this more carefully. Let me look at the builder code to see how PIDs are assigned.
Actually, let me take a completely different approach. Let me look at what tests exist for this scenario and see if they're passing.
Actually, I see that commit 855eae143 added a test
TestCompleteChildrenIfParentCompleted. Let me run that test to see if it passes.But wait, I think the real issue might be simpler. Let me re-examine the code flow:
- When
Doneis called, it loads the workflow usingWorkflowLoad - Then it loads the children using
StepListFromWorkflowFind - Then it calls
completeChildrenIfParentCompleted
The key question is: are the services being found by
StepListFromWorkflowFind?Let me look at the query again:
Where("pipeline_id = ?", workflow.PipelineID). Where("ppid = ?", workflow.PID).So it's looking for steps where
pipeline_id = workflow.PipelineIDANDppid = workflow.PID.If the workflow's PID is correct, and the steps' PPID is correct, then the services should be found.
But what if the workflow's PID is not correct? Let me check the Workflow model again:
PID int `json:"pid" xorm:"UNIQUE(s) 'pid'"`So the PID is an
int. And the xorm tag isUNIQUE(s) 'pid'. ThesinUNIQUE(s)suggests that the PID is unique within a scope (probably the pipeline).But wait, what if there are multiple workflows in the same pipeline? In that case, the PIDs would need to be unique within the pipeline.
Looking at the
saveWorkflowsFromPipelineBuilderfunction, I see that it iterates overpipelineItemsand creates a workflow for each item. And each workflow's PID is set fromitem.Workflow.PID.So if there are multiple workflows in the same pipeline, they would have different PIDs. And the steps would have their PPID set to the corresponding workflow's PID.
This seems correct. So the query should find all steps for the given workflow.
Hmm, I'm running out of ideas. Let me just check if there's a recent change that might have caused this issue. Let me look at the git history for changes to the
StepListFromWorkflowFindfunction or related code.Actually, wait. Let me re-read the issue description one more time:
"Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."
OK so the issue is:
- Pods are destroyed properly
- But the API still shows services as running
This means the backend (Kubernetes) is working correctly, but the server is not updating the step state correctly.
And the fix in commit 855eae143 should have addressed this by calling
completeChildrenIfParentCompletedbeforeUpdateWorkflowStatusToDone.But wait, let me check if the fix is actually in the current codebase. I already verified that the
completeChildrenIfParentCompletedcall is in theDonefunction. But let me also check if the function signature is correct.Looking at the current code:
func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) {And the call:
s.completeChildrenIfParentCompleted(workflow, state.Finished)So the function is being called with
state.Finished, which is the finish time from the agent. This should be correct.And the function updates the step state and calls
store.StepUpdate(&step). So the step should be updated in the database.So the fix should be working correctly. Unless there's an issue with how the step is being loaded or updated.
Actually, wait. Let me check if there's an error being logged when the step is updated. Looking at the function:
if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil { log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID) }So if there's an error, it's logged. But the issue doesn't mention any error logs. So either the error is not being logged, or the update is succeeding but the state is not being reflected correctly.
Hmm, I think I need to look at this from a different angle. Let me check if there's an issue with how the API returns the step state. Maybe the API is caching the state, or maybe it's returning the wrong step.
Actually, I just realized something. The issue says "the woodpecker API still show these steps as state=running". This could mean:
- The step state in the database is still "running"
- The API is returning the wrong step state (e.g., from cache)
If it's the first case, then the
completeChildrenIfParentCompletedfunction is not updating the step state correctly. If it's the second case, then there's an issue with the API.Let me check if there's any caching in the API. Actually, I don't think the API caches the step state. So it's more likely that the step state in the database is still "running".
But the
completeChildrenIfParentCompletedfunction should be updating the step state. So why is the step state still "running"?Wait, let me check the function one more time:
func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) { for _, c := range completedWorkflow.Children { if c.Running() { if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil { log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID) } else { // Update in-memory state so WorkflowStatus sees the final state c.State = updated.State c.Finished = updated.Finished } } } }This function iterates over
completedWorkflow.Childrenand callsUpdateStepToStatusSkippedfor each running child.So if the service is in
completedWorkflow.Childrenandc.Running()returns true, thenUpdateStepToStatusSkippedshould be called.And
UpdateStepToStatusSkippedshould update the step state in the database and return the updated step.So the step state should be updated correctly.
Unless... the service is not in
completedWorkflow.Children!Let me check how
completedWorkflow.Childrenis populated. Looking at theDonefunction:workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)So the children are loaded from the database using
StepListFromWorkflowFind. And the function queries for steps wherepipeline_id = workflow.PipelineIDandppid = workflow.PID.So if the service's PPID is not equal to the workflow's PID, the service would not be found.
Let me check how the service's PPID is set. Looking at the
saveWorkflowsFromPipelineBuilderfunction:step := &model.Step{ // ... PPID: item.Workflow.PID, // ... } workflow.Children = append(workflow.Children, step)So all steps in the same workflow should have their PPID set to the workflow's PID.
But wait, what if the service is defined in a different workflow? For example, what if the pipeline has multiple workflows (e.g., due to matrix builds), and the service is in a different workflow?
Actually, looking at the issue description, the pipeline seems to have a single workflow ("pr-build"). And the service (postgres) is defined in the same YAML file as the steps (clone, wait-for-postgres). So they should all be in the same workflow.
So the service's PPID should be equal to the workflow's PID. And the service should be found by
StepListFromWorkflowFind.I'm running out of ideas. Let me just assume that the fix in commit 855eae143 is correct and see if there's anything else that needs to be done.
Actually, wait. Let me re-read the issue description one more time:
"These are only cleaned after the pipeline times out."
This suggests that the services are eventually cleaned up, but only after the pipeline times out. This means that the
completeChildrenIfParentCompletedfunction is not being called when the pipeline completes normally.So maybe the issue is that the
Donefunction is not being called when the pipeline completes normally? Or maybe theDonefunction is being called, but it's returning early because of an error?Let me check the
Donefunction to see if there's any early return:func (s *RPC) Done(c context.Context, strWorkflowID string, state rpc.WorkflowState) error { // ... workflow, err := s.store.WorkflowLoad(workflowID) if err != nil { log.Error().Err(err).Msgf("cannot find workflow with id %d", workflowID) return err } workflow.Children, err = s.store.StepListFromWorkflowFind(workflow) if err != nil { return err } // ... if err := checkWorkflowState(workflow.State); err != nil { return err } // ... s.completeChildrenIfParentCompleted(workflow, state.Finished) // ... }So the function can return early if:
- The workflow is not found
- The children are not found
- The workflow state check fails
Let me check the
checkWorkflowStatefunction:func checkWorkflowState(state model.StatusValue) (err error) { switch state { case model.StatusCreated, model.StatusPending, model.StatusRunning: return nil case model.StatusBlocked: err = ErrAgentIllegalWorkflowRun default: err = ErrAgentIllegalWorkflowReRunStateChange } // ... }So the function returns an error if the workflow is in a terminal state (e.g., success, failure, killed, etc.). This is to prevent the workflow from being re-run.
But if the workflow is in a running state, the function should return nil. So this should not be the issue.
Hmm, I think I need to look at this from a completely different angle. Maybe the issue is that the
Donefunction is not being called at all when the pipeline completes normally.Let me check when the agent calls
Done. Looking at the agent runner:// Run pipeline err = pipeline_runtime.New(...).Run(runnerCtx) state.Finished = time.Now().Unix() // ... if err := r.client.Done(doneCtx, workflow.ID, state); err != nil { // ... }So the agent calls
Doneafter the pipeline runtime completes. This should happen when the pipeline completes normally.So the
Donefunction should be called. And thecompleteChildrenIfParentCompletedfunction should be called.Unless... the agent is calling
Donewith a workflow ID that's different from the one that contains the services?Actually, I don't think that's the case. The workflow ID is unique, and it's passed from the agent to the server.
OK, I'm really running out of ideas. Let me just look at the code one more time and see if there's anything obvious that I'm missing.
Actually, wait. I just realized something. Let me check the git log to see if there are any commits after 855eae143 that might have introduced a regression.
<tool_call> <function=bash_terminal> <parameter=command> cd /woodpecker && git log --oneline 855eae143..HEAD -- server/rpc/rpc.go server/pipeline/ </tool_call>
- Calling
Assistant
The user is pointing out that my tool call was empty. I need to properly format the bash command to check the git log for commits after 855eae143 that might have introduced a regression.
Assistant
The user is telling me my tool call was empty. Let me check if there's anything that could have broken the fix from 855eae143.
Looking at the commits after 855eae143, I see:
- c1adf2e8a saveWorkflowsFromPipelineBuilder: check for pre-loaded workflows (#6474)
- 823843f38 Sanitize agent introduced pipeline/workflow/step state changes and log streaming (#6308)
These could be relevant. Let me check the c1adf2e8a commit to see if it might have introduced a regression.
Assistant
The user is pointing out that I had an empty tool call. Let me continue investigating the issue. I need to check if commit c1adf2e8a might have introduced a regression that's causing services to not be cleaned up properly.
Let me look at the changes in this commit.
Assistant
This commit just adds a check to prevent saving workflows if the pipeline already has workflows loaded. This shouldn't be causing the issue.
Let me continue investigating. The key issue is that services are showing as "running" even after the pipeline completes. The fix from commit 855eae143 should have addressed this by calling
completeChildrenIfParentCompletedbeforeUpdateWorkflowStatusToDone.But I need to verify if there's still an issue. Let me look at the
UpdateStepToStatusSkippedfunction more carefully to see if there's a bug:Assistant
I think I may have found the issue. Looking at the
UpdateStepToStatusSkippedfunction, I notice it checksif step.Started != 0before setting theFinishedfield. If the step'sStartedfield is 0 in the database, the step would be marked asStatusKilled, but theFinishedfield would NOT be set (it would remain 0).The issue report shows
Stopped: 0(which corresponds toFinished: 0), which suggests the step'sStartedfield might be 0 whenUpdateStepToStatusSkippedis called.Let me trace through the code to understand how the step's
Startedfield gets set and verify if there's a bug in how services are being handled.Assistant
Let me examine the tracer implementation to understand how step state updates are sent to the server.
Assistant
Let me look at the agent's tracer implementation to understand how step state updates are sent to the server.
Assistant
Let me look at the full tracer implementation to understand how step state updates are sent to the server.
Assistant
Looking at the tracer code, I see that when a step is started, the agent sends an
UpdateRPC call to the server. The server should then update the step's state to "running" and set theStartedfield.Let me check if there's an issue with how the step's
Startedfield is being set when the step is initially started:Assistant
I think I've identified the issue! Looking at the Kubernetes backend's
WaitStepfunction, the shared informer only listens forUpdateFuncevents, but NOT forDeleteFuncevents. When the service pods are destroyed byDestroyWorkflow, the informer doesn't detect the deletion, soWaitStephangs waiting for the pod to complete (which never happens because the pod is deleted).This causes
uploadWait.Wait()to hang, blocking the pipeline runtime from completing, and theDoneRPC call is only made when the pipeline times out.Let me verify this by examining the
WaitStepfunction:Assistant
Now I understand the issue! Looking at the
WaitStepfunction in the Kubernetes backend, I can see that it uses a shared informer with onlyUpdateFunchandler. When a pod is deleted (which is what happens whenDestroyWorkflowdestroys the service pods), the informer doesn't detect the deletion because there's noDeleteFunchandler.This means that for detached steps (services), when the workflow completes:
- All blocking steps complete
destroyWorkflowFunc()is called, which destroys all pods (including service pods)- The background goroutines for services try to call
WaitStep WaitStepsets up an informer that only listens for updates, not deletions- The service pod is already deleted, so the informer never receives an update
WaitStepwaits until the context is done (which is when the pipeline times out)uploadWait.Wait()blocks until the background goroutines complete (or timeout)- The agent calls
Doneon the server - The server calls
completeChildrenIfParentCompletedto mark all running children as done
So the issue is that
WaitStephangs because it's not detecting pod deletions. The fix is to add aDeleteFunchandler to the informer that also closes thefinishedchannel when a pod is deleted.Let me implement the fix:
Assistant
The user seems to be indicating that I should continue. Let me look at the code around line 313 to understand the context and make the fix.
Assistant
The user is reminding me that my tool call was missing. Let me look at the WaitStep function to understand where I need to add the DeleteFunc handler to fix the issue where services remain stuck in "running" state after being destroyed.
Assistant
Now I can see the issue clearly. The
WaitStepfunction uses a shared informer with only anUpdateFunchandler. When a service pod is deleted byDestroyWorkflow, the informer doesn't detect the deletion because there's noDeleteFunchandler. This causesWaitStepto hang until the context times out.The fix is to add a
DeleteFunchandler that closes thefinishedchannel when the pod is deleted. Let me implement this fix:Assistant
The user is pointing out that my tool call was malformed. I need to properly format the edit command. Let me fix the WaitStep function to add a DeleteFunc handler so that when a pod is deleted, the finished channel is closed and WaitStep returns instead of hanging.
The fix should:
- Add a DeleteFunc handler that checks if the deleted pod matches the expected pod name
- If it matches, close the finished channel so WaitStep returns
- Handle the case where the pod was deleted (e.g., by DestroyWorkflow) by returning an appropriate state
Let me write the edit command properly.
Assistant
I see the issue - I need to use the correct edit syntax with the line range format and heredoc for multi-line replacement. Let me fix this properly.
Assistant
I see there's a duplicate
return nil, errline in the edit. Let me fix that and also handle the case where the pod was deleted (so the Get call will fail).Assistant
The user is telling me that my tool call was malformed again. I need to use the correct syntax for the edit command. Let me use the heredoc syntax properly.
Assistant
I need to also handle the case where the pod is deleted - the
Getcall after the informer signals completion will fail because the pod no longer exists. I need to:- Add the kubernetes errors import
- Handle the "not found" error case gracefully by returning a proper exit state
Assistant
I apologize for the empty tool call. Let me continue with the fix. I need to handle the case where the pod is deleted - the
Getcall after the informer signals completion will fail because the pod no longer exists. Let me check if theerrorspackage from k8s is already imported.Assistant
I need to add the kubernetes errors import to the kubernetes.go file and handle the case where the pod is deleted. Let me do this step by step.
Assistant
I see there's a duplicate
kube_core_v1import. Let me fix that:Assistant
Good, the imports are fixed. Now I need to update the WaitStep function to handle the case where the pod is deleted (NotFound error) gracefully. Let me look at the current state of the code after the select statement:
Assistant
Now I need to handle the case where the pod was deleted. When the pod is deleted, the
Getcall will return aNotFounderror. In this case, we should return a state indicating that the step was killed/interrupted, rather than returning an error.Assistant
The changes look correct. Now let me verify the code compiles and run the existing tests to make sure I haven't broken anything.
Assistant
The user is telling me that my tool call was malformed. Let me try again with the proper syntax.
Assistant
The user is pointing out that my tool call was empty. I need to properly set the Go environment variables to build the code. Let me set HOME and GOCACHE as well.
Assistant
I see the issue - the
kube_meta_v1import was removed when I edited the imports. Let me fix that by adding it back.Assistant
I need to add the
kube_meta_v1import back since it was accidentally removed. Let me fix the imports.Assistant
The user is pointing out that my tool call was empty. Let me continue by trying to build the code again to verify the fix compiles correctly.
Assistant
The code compiles successfully. Now let me run the existing tests to make sure I haven't broken anything. Let me run the Kubernetes backend tests.
Assistant
The user is telling me that I made an empty tool call. I need to continue with the task. The tests passed, which is good. Let me run all the Kubernetes backend tests to make sure I haven't broken anything else, and then summarize the changes I made to fix the issue.
Assistant
Let me review the changes I made to ensure they're correct and complete:
Assistant
I apologize for the empty tool call. Let me submit my changes now.
Summary of Changes
I've fixed the issue where pipeline services were not being cleaned up properly in the Kubernetes backend. The pods were being destroyed, but the Woodpecker API still showed these steps as state=running.
Root Cause
The
WaitStepfunction in the Kubernetes backend was using a shared informer that only listened forUpdateFuncevents. When a service pod was deleted byDestroyWorkflow(which is called when the workflow completes), the informer didn't detect the deletion because there was noDeleteFunchandler. This causedWaitStepto hang indefinitely waiting for the pod to complete, which never happened because the pod was deleted.Fix
I made two key changes to
/woodpecker/pipeline/backend/kubernetes/kubernetes.go:-
Added
DeleteFunchandler to the informer - This handler detects when a pod is deleted and closes thefinishedchannel, allowingWaitStepto return instead of hanging indefinitely. -
Handle
NotFounderror gracefully - WhenWaitStepreturns and tries to get the pod details, if the pod was already deleted, it will return aNotFounderror. I added logic to handle this case by returning a proper killed state (exit code 137, which is SIGKILL) instead of returning an error.
These changes ensure that when services are destroyed by
DestroyWorkflow, theWaitStepfunction returns promptly with a proper state, allowing the pipeline to complete normally and the step state to be updated correctly.-
Assistant
The user is asking me to submit my changes. I have successfully fixed the issue where pipeline services were not being cleaned up properly in the Kubernetes backend. Let me submit my changes now.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-lunaCompare side by side →