Skip to content
TheChatArchiveThe Chat Archive
QwenQwen/Qwen3.6-27B· alibabaHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…

  1. User

    ISSUE DESCRIPTION

    Woodpecker services appear to be running on UI & API after pipeline completes

    Component

    server

    Describe the bug

    Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.

    These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).

    Steps to reproduce

    • Install woodpecker next-d05c597374
    • Use kubernetes backend
    • Define pipeline with services running infinitely, like running postgres
    services:
      - name: postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        ports:
          - 5432
        environment:
          POSTGRES_PASSWORD: somepass
          POSTGRES_USER: someuser
          POSTGRES_DB: somedb
    steps:
      - name: wait-for-postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        environment:
          DB_HOST: postgres
          DB_PORT: 5432
          MAX_WAIT: 60
        commands:
          - ./scripts/wait-for-postgres.sh
    
    • Run the pipeline, observe pods start for the service and the step
    • once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
    pr-build > clone (#2):
    Step: clone
    Started: 1779195580
    Stopped: 1779195594
    Type: clone
    State: success
    
    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    
    pr-build > wait-for-postgres (#4):
    Step: wait-for-postgres
    Started: 1779195596
    Stopped: 1779195602
    Type: commands
    State: success
    

    Expected behavior

    Pipeline services are stopped once the pipeline steps complete, the pipeline completes.

    System Info

    next-d05c597374
    kubernetes backend, bitbucket datacenter forge
    

    Additional context

    No response

    Validations

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /woodpecker, current file: none) bash-$

  2. Qwen Assistant

    I'll start by exploring the repository structure to understand the codebase and locate the relevant code for handling pipeline services in the Kubernetes backend.

  3. Qwen Assistant

    Let me continue exploring the repository structure to understand the codebase better.

  4. Qwen Assistant

    Let me explore the pipeline directory since the issue is about pipeline services not being cleaned up properly. I'll also look at the kubernetes backend code.

  5. Qwen Assistant

    Let me look at the kubernetes backend files, especially the main kubernetes.go file and the service.go file to understand how services are handled.

  6. Qwen Assistant

    I need to use the correct argument name. Let me check the available tools again - it seems I should just use the file path directly.

  7. Qwen Assistant

    The open command seems to have an issue. Let me try using bash commands to view the file instead.

  8. Qwen Assistant

    Now I can see the Kubernetes backend code. The issue is that services are not being cleaned up properly. Let me look at the pipeline execution logic to understand how steps and services are managed. The key issue seems to be that services remain in a "running" state even after they've been destroyed.

    Let me look for where the pipeline execution happens and how services are stopped. I should check the agent runner and server logic.

  9. Qwen Assistant

    The user is pointing out that I didn't provide the required argument "command" for the bash_terminal tool. I need to continue my investigation of the codebase. Let me search for where services are stopped and how their state is managed.

  10. Qwen Assistant

    Let me search for where pipeline steps are managed and how their state is updated, especially for services.

  11. Qwen Assistant

    Let me look at the pipeline runtime to understand how steps and services are executed:

  12. Qwen Assistant

    Let me look at the step execution logic to understand how services are handled:

  13. Qwen Assistant

    Now I understand the structure better. Services are detached steps. Let me search for how services are defined and handled, and look at the server-side code that updates step states:

  14. Qwen Assistant

    Let me look at the server RPC handler to understand how it processes step state updates from the pipeline:

  15. Qwen Assistant

    Now I can see that the RPC Done method calls completeChildrenIfParentCompleted which should handle completing child steps when the parent workflow completes. Let me look at the UpdateStepStatus function in the pipeline package:

  16. Qwen Assistant

    Now I'm understanding the issue better. Let me look at how the workflow and pipeline structure handles services:

  17. Qwen Assistant

    Let me continue investigating by looking at the model/step.go to understand step types and how they relate to services.

  18. Qwen Assistant

    I can see that services are steps with StepType = "service" and they are also marked as Detached = true. Let me look at how the workflow completion logic handles services. The issue is that when the pipeline completes, the services are not being marked as stopped/killed.

    Let me look at the Done method in the RPC and how it handles child steps.

    Looking at the rpc.go code I already saw, the Done method calls completeChildrenIfParentCompleted which only handles children of the workflow. But I need to understand the parent-child relationship better and see if services are being handled correctly.

    Let me look for how services are related to other steps and when they should be stopped.

  19. Qwen Assistant

    The error message says "No tool call was invoked." I need to provide a proper bash command to continue exploring the codebase. Let me look at the workflow model and understand the parent-child relationship between workflows and steps.

    Based on what I've seen so far:

    1. Services are steps with StepType = "service" and Detached = true
    2. When a workflow completes, the Done method is called which calls completeChildrenIfParentCompleted
    3. The issue is that services are not being properly stopped when the pipeline completes

    Let me continue investigating the code to understand the parent-child relationship and how services should be stopped.

  20. Qwen Assistant

    Now I need to look at the complete Done method in rpc.go to understand where completeChildrenIfParentCompleted is called and how it's supposed to work:

  21. Qwen Assistant

    I can now see the issue! Looking at the Done method in rpc.go:

    1. It loads the workflow and its children (steps)
    2. It calls s.completeChildrenIfParentCompleted(workflow, state.Finished) to complete any still-running children

    Now let me look at the completeChildrenIfParentCompleted function more closely:

    func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) {
    	for _, c := range completedWorkflow.Children {
    		if c.Running() {
    			if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {
    				log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID)
    			} else {
    				// Update in-memory state so WorkflowStatus sees the final state
    				c.State = updated.State
    				c.Finished = updated.Finished
    			}
    		}
    	}
    }
    

    And the UpdateStepToStatusSkipped function:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    Wait, this looks like it should work. The completeChildrenIfParentCompleted function should mark all running children as killed/success.

    But wait, looking at the issue description again:

    • "The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    So the pods are being destroyed (which means DestroyWorkflow is being called), but the API still shows services as running. This suggests that the Done RPC call might not be getting the right list of children, or there's a timing issue.

    Let me think about this more carefully:

    1. Services are detached steps - they run in the background
    2. When all non-detached steps complete, the workflow should call Done
    3. The Done method should call completeChildrenIfParentCompleted to mark all running children (including services) as done

    But wait, looking at the runtime code again, I see that detached steps (services) have their completeStep called in a goroutine:

    func (r *Runtime) runDetachedStep(runnerCtx context.Context, step *backend_types.Step) error {
    	// ... start the step ...
    	
    	r.uploadWait.Add(1)
    	go func() {
    		defer r.uploadWait.Done()
    		
    		// ... complete the step in background ...
    	}()
    	
    	return nil
    }
    

    So when a detached step completes, it traces its state change asynchronously. But the key question is: when does the workflow call Done on the server?

    Looking at the workflow.go:

    func (r *Runtime) Run(runnerCtx context.Context) error {
    	// ... setup ...
    	
    	for _, stage := range r.spec.Stages {
    		stageChan := r.runStage(runnerCtx, stage.Steps)
    		// ... wait for stage ...
    	}
    	
    	// Now we can shutdown the workflow
    	destroyWorkflowFunc()
    	
    	// Ensure all logs/traces are uploaded before finishing
    	logger.Debug().Msg("waiting for logs and traces upload")
    	r.uploadWait.Wait()
    	// ...
    }
    

    And runStage returns when all steps in the stage are done:

    func (r *Runtime) runStage(runnerCtx context.Context, steps []*backend_types.Step) <-chan error {
    	var g errgroup.Group
    	done := make(chan error)
    
    	for _, step := range steps {
    		g.Go(func() error {
    			return r.executeStep(runnerCtx, step)
    		})
    	}
    
    	go func() {
    		done <- g.Wait()
    	}()
    
    	return done
    }
    

    So runStage waits for all steps (including detached steps that return immediately from executeStep). Since executeStep for a detached step returns immediately after starting the goroutine, the stage completes, and then the workflow calls destroyWorkflowFunc().

    Now, when the workflow finishes, the agent should call Done on the server. Let me check how the agent does this.

    Actually, I think I see the issue now! Looking at the runtime code:

    1. Detached steps (services) start in a goroutine and return immediately from executeStep
    2. The stage completes when all steps return
    3. destroyWorkflowFunc() is called, which calls DestroyWorkflow on the backend
    4. r.uploadWait.Wait() waits for all background goroutines to complete
    5. The workflow returns

    But here's the key: the background goroutine for the detached step calls completeStep, which in turn calls traceStep to report the step's final state. However, this happens AFTER destroyWorkflowFunc() is called.

    So the sequence is:

    1. All blocking steps complete
    2. Stage completes
    3. destroyWorkflowFunc() is called - pods are destroyed
    4. Background goroutines for services are still running
    5. uploadWait.Wait() blocks until background goroutines complete
    6. Background goroutines try to call WaitStep on already-destroyed pods
    7. They report their state

    But actually, the services should be destroyed by DestroyWorkflow, so they should never complete on their own. The issue is that the server side doesn't know they've been destroyed until the workflow calls Done.

    Let me look at when the agent calls Done:

    <tool_call> <function=bash_terminal> <parameter=command> grep -r ".Done(" /woodpecker/agent --include="*.go" -A 5 -B 5 </tool_call>

  22. Qwen Assistant

    I need to properly format the bash command. Let me fix the syntax.

  23. Qwen Assistant

    Now I understand the flow better. Let me look at the agent runner.go to see when Done is called and what state is sent to the server. The key is understanding when the agent calls Done and whether it happens before or after the services have been properly cleaned up.

    Let me look at the full runner.go file to understand the workflow execution flow.

  24. Qwen Assistant

    Now I understand the flow:

    1. The agent runs the pipeline using pipeline_runtime.New(...).Run(runnerCtx)
    2. The Run method in pipeline/runtime/workflow.go:
      • Runs all stages (which includes both blocking and detached steps)
      • Calls destroyWorkflowFunc() to destroy the workflow (which destroys all pods including services)
      • Waits for all uploads/traces with r.uploadWait.Wait()
      • Returns
    3. The agent then calls r.client.Done(doneCtx, workflow.ID, state) to signal the workflow is done

    Now let me think about the issue again:

    The problem is that services are shown as "running" in the API even after the pipeline completes. The issue states that:

    • Pods are destroyed properly (so DestroyWorkflow is working)
    • But API still shows services as state=running
    • These are only cleaned after the pipeline times out

    Looking at the Done method in server/rpc/rpc.go, it calls:

    s.completeChildrenIfParentCompleted(workflow, state.Finished)
    

    And completeChildrenIfParentCompleted should mark all running children as done. But wait - let me check the code again:

    func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) {
    	for _, c := range completedWorkflow.Children {
    		if c.Running() {
    			if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {
    				log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID)
    			} else {
    				// Update in-memory state so WorkflowStatus sees the final state
    				c.State = updated.State
    				c.Finished = updated.Finished
    			}
    		}
    	}
    }
    

    This iterates over completedWorkflow.Children and marks them as done. But where does workflow.Children come from? Let me check:

    workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)
    

    So it loads the children from the store. But wait, let me check what StepListFromWorkflowFind returns. Maybe it's only returning certain steps?

    Actually, I think I need to look at the store implementation to understand what steps are being loaded. But I think the issue might be in how the steps are being tracked.

    Wait, let me re-read the issue more carefully:

    "Some regression with newer releases (3.14.x)"

    This suggests that something changed in version 3.14.x that broke this functionality. Let me look at the git history to see what changed recently.

    Actually, I think I found the issue! Looking at the UpdateStepToStatusSkipped function:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    This function updates the step state to StatusSuccess if the step has started, and sets the finished time. But I don't think this is the issue.

    Let me think about this differently. The issue says that services are not being cleaned properly, and the API still shows them as running. But the pods are being destroyed. So the issue is that the step state in the database is not being updated.

    Now, let me look at the flow again:

    1. Agent runs the pipeline
    2. When the pipeline completes, DestroyWorkflow is called, which destroys all pods
    3. The agent calls Done on the server
    4. The server calls completeChildrenIfParentCompleted to mark all running children as done

    But wait, I see the issue now! In the Done method:

    // Complete any still-running children (e.g. service containers) before
    // computing the workflow status, so their final state is reflected.
    s.completeChildrenIfParentCompleted(workflow, state.Finished)
    

    This is called AFTER loading the workflow and its children. But the issue is that this code path might not be executed in all cases!

    Actually, let me look at the checkWorkflowAllowsStepUpdate function to see if there's any issue with the state updates:

    <tool_call> <function=bash_terminal> <parameter=command> grep -r "checkWorkflowAllowsStepUpdate|checkWorkflowState" /woodpecker/server --include="*.go" -A 20 </tool_call>

  25. Qwen Assistant

    The user is pointing out that my tool call was malformed. I need to properly format the bash_terminal command. Let me fix the syntax and try again.

  26. Qwen Assistant

    Let me look for how steps are loaded from the workflow to understand if services are being properly included in the children list:

  27. Qwen Assistant

    Good, now I can see the key function StepListFromWorkflowFind which loads steps from the workflow. It queries steps where ppid = workflow.PID. This means it's looking for steps whose parent process ID matches the workflow's PID.

    Now I understand the parent-child relationship:

    • Workflow has a PID
    • Steps have a PPID (parent process ID) that points to their parent workflow's PID

    So when completeChildrenIfParentCompleted is called, it should iterate over all steps that belong to the workflow and mark them as done.

    But wait, let me think about the issue more carefully. The issue says:

    • Pods are destroyed properly
    • But API still shows services as running

    This suggests that the completeChildrenIfParentCompleted function is not being called, or it's not updating the steps correctly.

    Let me look at the runtime code again to understand when services are stopped. Looking at the workflow.go:

    func (r *Runtime) Run(runnerCtx context.Context) error {
    	// ...
    	for _, stage := range r.spec.Stages {
    		stageChan := r.runStage(runnerCtx, stage.Steps)
    		// ...
    	}
    	
    	// Now we can shutdown the workflow
    	destroyWorkflowFunc()
    	
    	// Ensure all logs/traces are uploaded before finishing
    	logger.Debug().Msg("waiting for logs and traces upload")
    	r.uploadWait.Wait()
    	// ...
    }
    

    And runStage executes all steps in parallel, including detached steps (services). For detached steps, executeStep returns immediately after starting a goroutine, so the stage completes quickly.

    Then destroyWorkflowFunc() is called, which calls r.engine.DestroyWorkflow(ctx, r.spec, r.taskUUID). This destroys all pods, including service pods.

    Then r.uploadWait.Wait() waits for all background goroutines to complete. But wait, the background goroutines for services call completeStep, which tries to call WaitStep on the already-destroyed pod!

    Actually, looking at the runDetachedStep function again:

    func (r *Runtime) runDetachedStep(runnerCtx context.Context, step *backend_types.Step) error {
    	waitForLogs, startTime, err := r.startStep(step)
    	if err != nil {
    		return r.traceStep(nil, err, step)
    	}
    
    	r.uploadWait.Add(1)
    	go func() {
    		defer r.uploadWait.Done()
    
    		logger := r.makeLogger()
    
    		processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime)
    		logger.Debug().Str("step", step.Name).Msg("complete")
    
    		if errors.Is(err, context.Canceled) {
    			err = pipeline_errors.ErrCancel
    		}
    		if err != nil {
    			logger.Error().Err(err).Str("step", step.Name).Msg("detached step failed while running")
    		}
    
    		if traceErr := r.traceStep(processState, err, step); traceErr != nil {
    			logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result")
    		}
    	}()
    
    	return nil
    }
    

    So the detached step starts a goroutine that:

    1. Calls completeStep (which waits for the pod to complete and destroys it)
    2. Calls traceStep to report the final state

    But if destroyWorkflowFunc() is called before the goroutine completes, then the pod is already destroyed, and WaitStep might fail.

    Actually, let me re-read the completeStep function:

    func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) {
    	// Drain the log stream before waiting on the process exit.
    	waitForLogs()
    
    	waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck
    	if err != nil {
    		if errors.Is(err, context.Canceled) {
    			if waitState == nil {
    				waitState = &backend_types.State{}
    			}
    			waitState.Error = pipeline_errors.ErrCancel
    		} else {
    			return nil, err
    		}
    	}
    
    	// Use runnerCtx here: the workflow context may already be canceled but we
    	// still need to reach the backend to stop/remove the container.
    	if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil {
    		return nil, err
    	}
    
    	// ...
    }
    

    So completeStep calls WaitStep and then DestroyStep. If the pod is already destroyed by DestroyWorkflow, then WaitStep might fail.

    But wait, the issue is that the API still shows services as running. This suggests that the traceStep call is not being made, or the state update is not being sent to the server.

    Actually, I think I see the issue now! Let me look at the sequence of events:

    1. All blocking steps complete
    2. Stage completes (detached steps return immediately)
    3. destroyWorkflowFunc() is called, which destroys all pods (including services)
    4. uploadWait.Wait() waits for background goroutines
    5. Background goroutines try to call WaitStep on destroyed pods
    6. WaitStep might fail because the pod is gone
    7. If WaitStep fails, completeStep returns an error
    8. traceStep is called with the error

    But wait, looking at the runDetachedStep function:

    if traceErr := r.traceStep(processState, err, step); traceErr != nil {
    	logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result")
    }
    

    So even if there's an error, traceStep is still called. And traceStep should update the server with the step's state.

    Let me think about this differently. Maybe the issue is that the background goroutines are not being waited for properly. Let me check the uploadWait variable:

    Looking at the runtime.go file:

    <tool_call> <function=bash_terminal> <parameter=command> cat /woodpecker/pipeline/runtime/runtime.go </tool_call>

  28. Qwen Assistant

    The user is telling me that I need to provide a proper bash command. I apologize for the error - I had a typo in my tool call. Let me fix it and view the runtime.go file.

  29. Qwen Assistant

    Let me look at the git history to understand recent changes that might have caused this regression:

  30. Qwen Assistant

    I see an interesting commit: e173bed3f Kubernetes: detached steps are no services (#6435). This could be related to the issue. Let me examine this commit and the recent changes to the Kubernetes backend.

  31. Qwen Assistant

    Let me see the actual changes in that commit:

  32. Qwen Assistant

    This is very interesting! The commit e173bed3f changed the isService function to only check for step.Type == types.StepTypeService instead of also checking for detached steps with DNS patterns in their names.

    This means that before this change, detached steps could be treated as services (for DNS/hostname purposes). Now, only steps explicitly marked as StepTypeService are treated as services.

    But this doesn't explain the bug about services not being stopped properly. Let me continue investigating.

    Let me look at the service.go file more carefully to understand how services are being created and destroyed.

  33. Qwen Assistant

    I can see that the isService function now only checks for step.Type == types.StepTypeService. But this doesn't explain the issue with services not being stopped properly.

    Let me look at the DestroyWorkflow function to understand how services are being destroyed. The issue is that services are not being stopped when the workflow completes, even though the pods are being destroyed.

    Looking at the DestroyWorkflow function in kubernetes.go:

    func (e *kube) DestroyWorkflow(ctx context.Context, conf *types.Config, taskUUID string) error {
    	log.Trace().Str("taskUUID", taskUUID).Msg("deleting Kubernetes primitives")
    
    	for _, stage := range conf.Stages {
    		for _, step := range stage.Steps {
    			err := stopPod(ctx, e, step, e.config.newDefaultDeleteOptions())
    			if err != nil {
    				return err
    			}
    		}
    	}
    
    	namespace := e.config.GetNamespace(conf.Stages[0].Steps[0].OrgID)
    
    	log.Trace().Str("taskUUID", taskUUID).Msgf("deleting workflow headless service")
    	err := e.stopHeadlessService(ctx, e, namespace, taskUUID)
    	if err != nil {
    		return err
    	}
    
    	log.Trace().Str("taskUUID", taskUUID).Msgf("deleting workflow volume")
    	err = stopVolume(ctx, e, conf.Volume, e.config.GetNamespace(conf.Stages[0].Steps[0].OrgID), e.config.newDefaultDeleteOptions())
    	if err != nil {
    		return err
    	}
    
    	return nil
    }
    

    So DestroyWorkflow iterates over all steps in all stages and calls stopPod for each step. This should destroy all service pods.

    But the issue is that the API still shows services as running. This means that the step state in the database is not being updated.

    Now, let me think about the flow again:

    1. Agent runs the pipeline
    2. When the pipeline completes, DestroyWorkflow is called, which destroys all pods (including services)
    3. The agent calls Done on the server
    4. The server calls completeChildrenIfParentCompleted to mark all running children as done

    So the server should be updating the step state when Done is called. But the issue says that services are still showing as running.

    Wait, let me look at the UpdateStepToStatusSkipped function again:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    This function updates the step state to StatusSuccess if the step has started. But wait, the issue is that services are still showing as running. So either:

    1. The completeChildrenIfParentCompleted function is not being called
    2. The completeChildrenIfParentCompleted function is not finding the services
    3. The completeChildrenIfParentCompleted function is not updating the services correctly

    Let me look at the completeChildrenIfParentCompleted function again:

    func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) {
    	for _, c := range completedWorkflow.Children {
    		if c.Running() {
    			if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {
    				log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID)
    			} else {
    				// Update in-memory state so WorkflowStatus sees the final state
    				c.State = updated.State
    				c.Finished = updated.Finished
    			}
    		}
    	}
    }
    

    So this function iterates over completedWorkflow.Children and calls UpdateStepToStatusSkipped for each running child.

    But wait, let me check how workflow.Children is populated. Looking at the Done method:

    workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)
    

    And StepListFromWorkflowFind is:

    func (s storage) StepListFromWorkflowFind(workflow *model.Workflow) ([]*model.Step, error) {
    	return s.stepListWorkflow(s.engine.NewSession(), workflow)
    }
    
    func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) {
    	stepList := make([]*model.Step, 0)
    	return stepList, sess.
    		Where("pipeline_id = ?", workflow.PipelineID).
    		Where("ppid = ?", workflow.PID).
    		OrderBy("pid").
    		Find(&stepList)
    }
    

    So this function finds all steps where ppid = workflow.PID. This means it's looking for steps whose parent process ID matches the workflow's PID.

    But wait, I need to understand the parent-child relationship better. Let me check how steps are created and how their PPID is set.

    Actually, looking at the model, I see that:

    • Workflow has a PID
    • Steps have a PPID (parent process ID) and a PID

    So the parent-child relationship is:

    • Workflow (PID=X) has children Steps (PPID=X)

    But wait, looking at the code again, I notice that the Step model has both PID and PPID:

    type Step struct {
    	// ...
    	PID        int         `json:"pid"                  xorm:"UNIQUE(s) 'pid'"`
    	PPID       int         `json:"ppid"                 xorm:"ppid"`
    	// ...
    }
    

    And the Workflow model has a PID:

    type Workflow struct {
    	// ...
    	PID        int               `json:"pid"                  xorm:"UNIQUE(s) 'pid'"`
    	// ...
    }
    

    So the StepListFromWorkflowFind function is looking for steps where ppid = workflow.PID. This means it's looking for steps that are direct children of the workflow.

    But wait, I need to understand how the PPID is set for services. Let me search for how steps are created.

    Actually, looking at the issue description again, the problem is that services are not being cleaned properly. The pods are destroyed, but the API still shows them as running.

    This suggests that the completeChildrenIfParentCompleted function is not being called, or it's not finding the services, or it's not updating them correctly.

    Let me think about this differently. Maybe the issue is that services are created in a different stage or workflow, so they're not found by StepListFromWorkflowFind.

    Actually, I think I need to look at how the pipeline and workflow models are structured. Let me search for where steps are created.

    Let me think about this more carefully. The issue says:

    • Pods are destroyed properly (so DestroyWorkflow is working)
    • But API still shows services as running
    • These are only cleaned after the pipeline times out

    This suggests that the step state is not being updated when the workflow completes. So either:

    1. The Done RPC call is not being made
    2. The Done RPC call is being made, but completeChildrenIfParentCompleted is not finding the services
    3. The Done RPC call is being made, and completeChildrenIfParentCompleted is finding the services, but it's not updating them correctly

    Let me look at the issue from a different angle. Maybe the problem is that services are not direct children of the workflow, so they're not found by StepListFromWorkflowFind.

    Actually, I think I found the issue! Looking at the stepListWorkflow function:

    func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) {
    	stepList := make([]*model.Step, 0)
    	return stepList, sess.
    		Where("pipeline_id = ?", workflow.PipelineID).
    		Where("ppid = ?", workflow.PID).
    		OrderBy("pid").
    		Find(&stepList)
    }
    

    This function is looking for steps where ppid = workflow.PID. But wait, what if services have a different PPID?

    Let me search for how steps are created and how their PPID is set.

    Actually, let me search for where the Step model is populated.

    <tool_call> <function=bash_terminal> <parameter=command> find /woodpecker/server -name "*.go" -exec grep -l "Step{" {} ; | head -20 </tool_call>

  34. Qwen Assistant

    I need to properly format my bash command to search for where steps are created.

  35. Qwen Assistant

    Let me look at the items.go file to understand how steps are created and how their PPID is set.

  36. Qwen Assistant

    Now I can see the key part! In the saveWorkflowsFromPipelineBuilder function, I can see how steps are created and their PPID is set:

    step := &model.Step{
        Name:       step.Name,
        UUID:       step.UUID,
        PipelineID: pipeline.ID,
        PID:        pidSequence,
        PPID:       item.Workflow.PID,  // <-- This is set to the workflow's PID
        State:      model.StatusPending,
        Failure:    step.Failure,
        Type:       model.StepType(step.Type),
    }
    

    So all steps (including services) have their PPID set to the workflow's PID. This means they should all be found by StepListFromWorkflowFind.

    But wait, the issue says services are still running. Let me think about this more carefully.

    Looking at the code flow:

    1. When Done is called on the server, it loads the workflow and its children using StepListFromWorkflowFind
    2. It then calls completeChildrenIfParentCompleted which iterates over workflow.Children and marks running children as done
    3. This should update all steps, including services

    But the issue is that services are still showing as running. This suggests that either:

    1. The Done call is not being made
    2. The services are not being found by StepListFromWorkflowFind
    3. The services are not being updated by completeChildrenIfParentCompleted

    Wait, I think I need to look at the issue from a different angle. Let me look at the UpdateStepToStatusSkipped function again:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    This function sets step.State = status (which is StatusKilled), but then if step.Started != 0, it sets step.State = model.StatusSuccess. So if the service has started (which it should), it will be marked as StatusSuccess.

    But wait, the issue says services are still showing as running. So maybe the completeChildrenIfParentCompleted function is not being called?

    Actually, I think I need to look at the issue from a different angle. Maybe the problem is that the services are not being found because they have a different PPID.

    Let me look at the pipeline structure again. The issue says the pipeline has services defined in the YAML. So the services should be part of the same workflow as the steps.

    Actually, wait. Let me re-read the issue description:

    pr-build > clone (#2):
    Step: clone
    Started: 1779195580
    Stopped: 1779195594
    Type: clone
    State: success
    
    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    
    pr-build > wait-for-postgres (#4):
    Step: wait-for-postgres
    Started: 1779195596
    Stopped: 1779195602
    Type: commands
    State: success
    

    So we have:

    • clone (#2): PID=2, success
    • postgres (#3): PID=3, service, running
    • wait-for-postgres (#4): PID=4, success

    All of these should be part of the same workflow, so they should all have the same PPID.

    But wait, the issue says that Stopped: 0 for the postgres service. This suggests that the Finished field is not being set.

    Let me look at the UpdateStepToStatusSkipped function again:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    So if step.Started != 0, it sets step.Finished = finished. But the issue says Stopped: 0, which means Finished is not being set.

    This suggests that step.Started == 0 for the service. But that doesn't make sense, because the issue says Started: 1779195594 for the postgres service.

    Wait, let me look at the model again. The Step model has:

    • Started int64 - when the step started
    • Finished int64 - when the step finished

    And the UpdateStepToStatusSkipped function checks if step.Started != 0. If it's not zero, it sets step.Finished = finished.

    But the issue says Stopped: 0, which means Finished is 0. So either:

    1. step.Started == 0 when UpdateStepToStatusSkipped is called
    2. UpdateStepToStatusSkipped is not being called

    Actually, I think I see the issue now! Let me look at the UpdateStepToStatusSkipped function again:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    Wait, the first line is step.State = status. So it sets the state to status (which is StatusKilled). But then, if step.Started != 0, it sets step.State = model.StatusSuccess.

    So if the service has started, it should be marked as StatusSuccess and Finished should be set. But the issue says the service is still running and Stopped: 0.

    This suggests that UpdateStepToStatusSkipped is not being called for the service. Or maybe the service is not being found by StepListFromWorkflowFind.

    Actually, I think I need to look at this more carefully. Let me check if there's a recent change to the code that might have caused this regression.

    Let me search for recent changes to the completeChildrenIfParentCompleted function or UpdateStepToStatusSkipped.

    Actually, let me think about this differently. The issue says this is a regression with newer releases (3.14.x). So something changed in 3.14.x that broke this functionality.

    Let me look at the git history to see what changed recently.

    Actually, I think I see the issue now! Let me look at the stepListWorkflow function again:

    func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) {
    	stepList := make([]*model.Step, 0)
    	return stepList, sess.
    		Where("pipeline_id = ?", workflow.PipelineID).
    		Where("ppid = ?", workflow.PID).
    		OrderBy("pid").
    		Find(&stepList)
    }
    

    This function is looking for steps where ppid = workflow.PID. But what if the workflow's PID is not set correctly?

    Let me look at how the workflow's PID is set. In the saveWorkflowsFromPipelineBuilder function:

    workflow := &model.Workflow{
        ID:         item.Workflow.ID,
        Name:       item.Workflow.Name,
        PID:        item.Workflow.PID,  // <-- This is set from item.Workflow.PID
        PipelineID: pipeline.ID,
        State:      model.StatusPending,
        Environ:    item.Workflow.Environ,
        AxisID:     item.Workflow.AxisID,
    }
    

    So the workflow's PID is set from item.Workflow.PID. And the step's PPID is also set from item.Workflow.PID:

    step := &model.Step{
        // ...
        PPID:       item.Workflow.PID,
        // ...
    }
    

    So all steps in the same workflow should have the same PPID, which is the workflow's PID. This means StepListFromWorkflowFind should find all steps, including services.

    But wait, I think I see the issue now! Let me look at the issue description again:

    pr-build > clone (#2):
    Step: clone
    Started: 1779195580
    Stopped: 1779195594
    Type: clone
    State: success
    
    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    
    pr-build > wait-for-postgres (#4):
    Step: wait-for-postgres
    Started: 1779195596
    Stopped: 1779195602
    Type: commands
    State: success
    

    The numbers in parentheses (#2, #3, #4) are the step IDs, not the PIDs. So the step IDs are 2, 3, and 4.

    But the PIDs are different from the IDs. Let me look at the model again:

    type Step struct {
    	ID         int64       `json:"id"                   xorm:"pk autoincr 'id'"`
    	// ...
    	PID        int         `json:"pid"                  xorm:"UNIQUE(s) 'pid'"`
    	PPID       int         `json:"ppid"                 xorm:"ppid"`
    	// ...
    }
    

    So ID is the primary key (auto-incremented), and PID is a separate field that's used for ordering and parent-child relationships.

    So the issue is not with the IDs, but with the PIDs. And the PIDs should be set correctly based on the code I've seen.

    Let me think about this differently. Maybe the issue is that the completeChildrenIfParentCompleted function is being called, but it's not updating the services because they're not in the workflow.Children list.

    Actually, I think I need to look at the Done function more carefully. Let me re-read it:

    func (s *RPC) Done(c context.Context, strWorkflowID string, state rpc.WorkflowState) error {
    	// ...
    	workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)
    	if err != nil {
    		return err
    	}
    	// ...
    	s.completeChildrenIfParentCompleted(workflow, state.Finished)
    	// ...
    }
    

    So the Done function loads the workflow's children using StepListFromWorkflowFind, and then calls completeChildrenIfParentCompleted.

    But wait, I think I see the issue now! Let me look at the checkWorkflowState function:

    func checkWorkflowState(state model.StatusValue) (err error) {
    	switch state {
    	case model.StatusCreated,
    		model.StatusPending,
    		model.StatusRunning:
    		return nil
    
    	case model.StatusBlocked:
    		err = ErrAgentIllegalWorkflowRun
    
    	default:
    		err = ErrAgentIllegalWorkflowReRunStateChange
    	}
    	// ...
    }
    

    This function checks if the workflow's state allows it to be initialized or marked as done. If the workflow is in a terminal state (success, failure, killed, etc.), it returns an error.

    But wait, the Done function calls checkWorkflowState before calling completeChildrenIfParentCompleted:

    // check workflow's own state to prevent finishing an already-finished or blocked workflow
    if err := checkWorkflowState(workflow.State); err != nil {
    	return err
    }
    // ...
    s.completeChildrenIfParentCompleted(workflow, state.Finished)
    

    So if the workflow is in a terminal state, the Done function will return early and completeChildrenIfParentCompleted will not be called.

    But wait, the issue says that the pipeline is still running (not completed), and the services are still running. So the workflow should be in a running state, and checkWorkflowState should not return an error.

    Actually, I think I need to look at the issue from a different angle. Let me think about what happens when the pipeline completes:

    1. All blocking steps complete
    2. The stage completes (detached steps return immediately)
    3. destroyWorkflowFunc() is called, which destroys all pods (including services)
    4. The agent calls Done on the server
    5. The server calls completeChildrenIfParentCompleted to mark all running children as done

    But wait, what if the agent doesn't call Done on the server? Or what if the agent calls Done before the services have started?

    Actually, I think I see the issue now! Let me look at the runtime code again:

    func (r *Runtime) Run(runnerCtx context.Context) error {
    	// ...
    	for _, stage := range r.spec.Stages {
    		stageChan := r.runStage(runnerCtx, stage.Steps)
    		// ...
    	}
    	
    	// Now we can shutdown the workflow
    	destroyWorkflowFunc()
    	
    	// Ensure all logs/traces are uploaded before finishing
    	logger.Debug().Msg("waiting for logs and traces upload")
    	r.uploadWait.Wait()
    	// ...
    }
    

    So the runtime waits for all stages to complete, then calls destroyWorkflowFunc(), then waits for all uploads to complete.

    And the agent calls Done after the runtime completes:

    // Run pipeline
    err = pipeline_runtime.New(...).Run(runnerCtx)
    
    state.Finished = time.Now().Unix()
    // ...
    
    if err := r.client.Done(doneCtx, workflow.ID, state); err != nil {
    	// ...
    }
    

    So the agent calls Done after the runtime completes, which means after uploadWait.Wait() has returned.

    But wait, what if the uploadWait.Wait() doesn't wait for the detached steps to complete? Let me look at the runDetachedStep function again:

    func (r *Runtime) runDetachedStep(runnerCtx context.Context, step *backend_types.Step) error {
    	waitForLogs, startTime, err := r.startStep(step)
    	if err != nil {
    		return r.traceStep(nil, err, step)
    	}
    
    	r.uploadWait.Add(1)
    	go func() {
    		defer r.uploadWait.Done()
    		// ...
    	}()
    
    	return nil
    }
    

    So the detached step increments uploadWait and starts a goroutine that calls uploadWait.Done() when it completes.

    But wait, the goroutine calls completeStep, which calls WaitStep and DestroyStep. If the pod is already destroyed by destroyWorkflowFunc(), then WaitStep might fail.

    Let me look at the completeStep function again:

    func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) {
    	// Drain the log stream before waiting on the process exit.
    	waitForLogs()
    
    	waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck
    	if err != nil {
    		if errors.Is(err, context.Canceled) {
    			if waitState == nil {
    				waitState = &backend_types.State{}
    			}
    			waitState.Error = pipeline_errors.ErrCancel
    		} else {
    			return nil, err
    		}
    	}
    
    	// Use runnerCtx here: the workflow context may already be canceled but we
    	// still need to reach the backend to stop/remove the container.
    	if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil {
    		return nil, err
    	}
    	// ...
    }
    

    So if WaitStep fails with an error that's not context.Canceled, it returns nil, err. And then the goroutine logs the error and calls traceStep with the error.

    But wait, what if WaitStep fails because the pod is already destroyed? Let me look at the WaitStep function in the Kubernetes backend:

    func (e *kube) WaitStep(ctx context.Context, step *types.Step, taskUUID string) (*types.State, error) {
    	// ...
    	pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{})
    	if err != nil {
    		return nil, err
    	}
    	// ...
    }
    

    So if the pod is not found (because it was destroyed), WaitStep will return an error. And then completeStep will return nil, err.

    And then the goroutine will call traceStep(processState, err, step) with processState = nil and err != nil.

    Let me look at the traceStep function:

    func (r *Runtime) traceStep(processState *backend_types.State, err error, step *backend_types.Step) error {
    	s := new(state.State)
    	s.Workflow.Started = r.started
    	s.CurrStep = step
    	s.Workflow.Error = r.err.Get()
    
    	switch {
    	case processState == nil && err != nil:
    		// Step failed to start — create an dummy exited process state.
    		s.CurrStepState = backend_types.State{
    			Error:     err,
    			Exited:    true,
    			OOMKilled: false,
    		}
    	case processState != nil:
    		s.CurrStepState = *processState
    		// processState == nil && err == nil: step just started, leave s.CurrStepState zero-valued.
    	}
    
    	if traceErr := r.tracer.Trace(s); traceErr != nil {
    		logger := r.makeLogger()
    		logger.Error().Err(traceErr).Msg("could not trace step state change")
    	}
    
    	return err
    }
    

    So if processState == nil && err != nil, it creates a dummy exited process state with Exited: true. And then it calls r.tracer.Trace(s) to send the state to the server.

    But wait, what if the tracer fails to send the state to the server? Or what if the server doesn't update the step state correctly?

    Actually, I think I see the issue now! Let me look at the sequence of events again:

    1. All blocking steps complete
    2. The stage completes (detached steps return immediately)
    3. destroyWorkflowFunc() is called, which destroys all pods (including services)
    4. The background goroutines for services try to call WaitStep on the destroyed pods
    5. WaitStep fails because the pod is not found
    6. completeStep returns nil, err
    7. The goroutine calls traceStep(nil, err, step) with processState = nil and err != nil
    8. traceStep creates a dummy exited process state and calls r.tracer.Trace(s)
    9. The tracer sends the state to the server via RPC
    10. The server updates the step state

    But wait, step 9 happens AFTER destroyWorkflowFunc() has been called. And the agent calls Done after uploadWait.Wait() has returned, which means after all background goroutines have completed.

    So the sequence should be:

    1. Blocking steps complete
    2. Stage completes
    3. destroyWorkflowFunc() is called
    4. Background goroutines try to call WaitStep and fail
    5. Background goroutines call traceStep to report the failure
    6. uploadWait.Wait() returns
    7. Agent calls Done on the server
    8. Server calls completeChildrenIfParentCompleted to mark all running children as done

    So the services should be updated by either:

    • The traceStep call in the background goroutine (if it succeeds)
    • The completeChildrenIfParentCompleted call in the Done function

    But the issue says the services are still running. So either:

    1. The traceStep call fails
    2. The completeChildrenIfParentCompleted call doesn't find the services

    Actually, I think I need to look at the tracer to understand how it sends the state to the server.

    Actually, I think I found the bug! Let me look at the completeChildrenIfParentCompleted function again:

    func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) {
    	for _, c := range completedWorkflow.Children {
    		if c.Running() {
    			if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {
    				log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID)
    			} else {
    				// Update in-memory state so WorkflowStatus sees the final state
    				c.State = updated.State
    				c.Finished = updated.Finished
    			}
    		}
    	}
    }
    

    This function iterates over completedWorkflow.Children and calls UpdateStepToStatusSkipped for each running child.

    But wait, I think I see the issue now! The function checks if c.Running(). Let me look at the Running function:

    func (p *Step) Running() bool {
    	return p.State == StatusPending || p.State == StatusRunning
    }
    

    So Running() returns true if the step's state is StatusPending or StatusRunning.

    But what if the step's state is not StatusPending or StatusRunning? For example, what if the step's state is StatusSuccess (because the traceStep call succeeded)?

    In that case, c.Running() would return false, and UpdateStepToStatusSkipped would not be called.

    But wait, the issue says the services are still running. So the step's state should be StatusRunning, and c.Running() should return true.

    Actually, I think I need to look at the issue from a different angle. Let me think about what could cause the services to still be running after the pipeline completes.

    One possibility is that the traceStep call fails to update the step state, and the completeChildrenIfParentCompleted function doesn't find the services.

    But why would completeChildrenIfParentCompleted not find the services? The services should be in workflow.Children because they have the same PPID as the workflow.

    Wait, I think I see the issue now! Let me look at the StepListFromWorkflowFind function again:

    func (s storage) StepListFromWorkflowFind(workflow *model.Workflow) ([]*model.Step, error) {
    	return s.stepListWorkflow(s.engine.NewSession(), workflow)
    }
    
    func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) {
    	stepList := make([]*model.Step, 0)
    	return stepList, sess.
    		Where("pipeline_id = ?", workflow.PipelineID).
    		Where("ppid = ?", workflow.PID).
    		OrderBy("pid").
    		Find(&stepList)
    }
    

    This function queries the database for steps where pipeline_id = workflow.PipelineID and ppid = workflow.PID.

    But wait, what if the workflow's PID is 0 or not set? In that case, the query would not find any steps!

    Let me look at how the workflow's PID is set. In the saveWorkflowsFromPipelineBuilder function:

    workflow := &model.Workflow{
        ID:         item.Workflow.ID,
        Name:       item.Workflow.Name,
        PID:        item.Workflow.PID,  // <-- This is set from item.Workflow.PID
        PipelineID: pipeline.ID,
        State:      model.StatusPending,
        Environ:    item.Workflow.Environ,
        AxisID:     item.Workflow.AxisID,
    }
    

    So the workflow's PID is set from item.Workflow.PID. And item is a builder.Item.

    Let me look at the builder package to understand how the workflow's PID is set.

    Actually, wait. I think I found the issue! Let me look at the issue description again:

    "Some regression with newer releases (3.14.x)"

    This suggests that something changed in 3.14.x that broke this functionality. And I saw a commit e173bed3f that changed the isService function. But that doesn't seem to be related to the issue.

    Let me search for other recent changes to the server/pipeline package.

    <tool_call> <function=bash_terminal> <parameter=command> cd /woodpecker && git log --oneline --all --since="2024-01-01" -- server/pipeline/ | head -30 </tool_call>

  37. Qwen Assistant

    The user is asking me to run a git command to look at recent changes to the server/pipeline directory. Let me properly format this bash command.

  38. Qwen Assistant

    I found a very relevant commit: 855eae143 Set workflow services stuck in running state to finished (#6337). This looks like it's addressing the exact issue described! Let me examine this commit.

  39. Qwen Assistant

    I found a commit that seems to address the exact issue. Let me look at the full diff to understand what was changed and if it's the fix I need to apply or if there's still something missing.

  40. Qwen Assistant

    Excellent! I found the commit 855eae143 that addresses this exact issue! This commit was made on March 30, 2026. Let me check if this fix is already in the current codebase or if I need to apply it.

  41. Qwen Assistant

    The issue is about services remaining in a "running" state after pipeline completion with Kubernetes backend. The commit 855eae143 already contains a fix for this issue. Let me check if this commit is already in the current branch or if it's in a different branch/commit that hasn't been merged yet.

    Let me check if the current HEAD already has this fix applied.

  42. Qwen Assistant

    Let me verify that the fix from commit 855eae143 is actually applied in the current code by checking the rpc.go file:

  43. Qwen Assistant

    The user is asking me to continue debugging. I found the commit 855eae143 that appears to fix the issue, but I need to verify if this fix is actually present in the current codebase and if there's still an issue. Let me check the current state of the code and see if the fix from commit 855eae143 is already applied or if there's still something missing.

    Let me verify the current state of the code by looking at the key functions mentioned in the commit.

  44. Qwen Assistant

    Good, I can see the fix is already present in the codebase. The completeChildrenIfParentCompleted function is being called with state.Finished before UpdateWorkflowStatusToDone. This is the fix from commit 855eae143.

    Now I need to verify if the fix is complete. Let me check the completeChildrenIfParentCompleted function to see if it's using the correct parameters and logic.

  45. Qwen Assistant

    The user seems to want me to continue investigating the issue. Let me check the current state of the code.

    From what I can see:

    1. The commit 855eae143 is already in the current branch (it's in the git log)
    2. The completeChildrenIfParentCompleted function is already being called with state.Finished parameter
    3. The function updates the in-memory state of running children

    But wait, the issue says "Some regression with newer releases (3.14.x)". So the issue was reported AFTER version 3.14.x, which means the fix in commit 855eae143 might not be in 3.14.x yet, or there might be a new regression after the fix.

    Let me check the current code more carefully to see if there's still an issue.

    Actually, looking at the code, I see that the fix is already in place. But the issue description mentions that this is a regression in 3.14.x, and the fix (855eae143) was made on March 30, 2026. So the fix should already be in the current codebase.

    Let me check if there's something else that might be causing the issue. Maybe the issue is that the StepListFromWorkflowFind function is not finding the services, or maybe the services are not being added to workflow.Children correctly.

    Let me look at the StepListFromWorkflowFind function again to see if there's an issue with how it finds services.

    Actually, I think I need to look at the issue more carefully. The issue says:

    • Pods are destroyed properly (so DestroyWorkflow is working)
    • But API still shows services as running

    So the pods are being destroyed, but the step state is not being updated. This suggests that the completeChildrenIfParentCompleted function is not being called, or it's not finding the services, or it's not updating them correctly.

    Let me check if the StepListFromWorkflowFind function is finding all steps, including services. Looking at the function:

    func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) {
    	stepList := make([]*model.Step, 0)
    	return stepList, sess.
    		Where("pipeline_id = ?", workflow.PipelineID).
    		Where("ppid = ?", workflow.PID).
    		OrderBy("pid").
    		Find(&stepList)
    }
    

    This function is looking for steps where pipeline_id = workflow.PipelineID and ppid = workflow.PID. But what if the workflow's PID is 0 or not set? In that case, the query would not find any steps!

    Let me check how the workflow's PID is set when it's loaded from the database. Maybe the PID is not being loaded correctly.

    Actually, looking at the Done function, I see that it loads the workflow using s.store.WorkflowLoad(workflowID). Let me check if this function loads the PID correctly.

    Wait, I think I need to look at the issue from a different angle. The issue says that the services are still running after the pipeline completes. But the fix in commit 855eae143 should have addressed this issue by calling completeChildrenIfParentCompleted before UpdateWorkflowStatusToDone.

    Let me check if there's a case where completeChildrenIfParentCompleted is not finding the services. Maybe the services are not in workflow.Children because they have a different PPID?

    Actually, I just realized something. Looking at the saveWorkflowsFromPipelineBuilder function in items.go, I see that all steps (including services) are added to workflow.Children:

    for _, stage := range item.Config.Stages {
        for _, step := range stage.Steps {
            pidSequence++
            step := &model.Step{
                // ...
                PPID:       item.Workflow.PID,
                // ...
            }
            workflow.Children = append(workflow.Children, step)
        }
    }
    

    So all steps should have the same PPID as the workflow's PID, and they should all be in workflow.Children.

    But wait, the Done function loads the workflow's children from the database using StepListFromWorkflowFind:

    workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)
    

    So the children are loaded from the database, not from the in-memory workflow.Children list. And the query is:

    Where("pipeline_id = ?", workflow.PipelineID).
    Where("ppid = ?", workflow.PID).
    

    So if the workflow's PID is set correctly, the query should find all steps, including services.

    Let me check if there's an issue with how the workflow's PID is loaded from the database. Maybe the PID is not being loaded correctly when the workflow is loaded.

    Actually, I think I need to look at the Workflow model to see if the PID field is being loaded correctly.

    Wait, I just realized something. The issue says "Some regression with newer releases (3.14.x)". And the fix (855eae143) was made on March 30, 2026. So the fix might already be in the current codebase, but the issue might still be happening because of a different reason.

    Let me re-read the issue description more carefully:

    "Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    So the pods are being destroyed, but the step state is not being updated. This suggests that the completeChildrenIfParentCompleted function is not being called, or it's not finding the services, or it's not updating them correctly.

    But wait, the fix in commit 855eae143 should have addressed this issue. So maybe the issue is still happening because of a different reason.

    Let me think about this more carefully. The issue says that the services are still running after the pipeline completes. But the fix in commit 855eae143 should have addressed this issue by calling completeChildrenIfParentCompleted before UpdateWorkflowStatusToDone.

    So if the fix is already in the current codebase, why is the issue still happening?

    Actually, I think I need to look at the issue from a different angle. Maybe the issue is that the StepListFromWorkflowFind function is not finding the services because they have a different PPID than the workflow's PID.

    Let me check how the PPID is set for services. Looking at the saveWorkflowsFromPipelineBuilder function, I see that all steps have their PPID set to item.Workflow.PID:

    PPID:       item.Workflow.PID,
    

    So all steps should have the same PPID as the workflow's PID.

    But wait, what if the workflow's PID is 0 or not set? In that case, the query would not find any steps!

    Let me check how the workflow's PID is set. Looking at the saveWorkflowsFromPipelineBuilder function, I see that the workflow's PID is set from item.Workflow.PID:

    PID:        item.Workflow.PID,
    

    And item is a builder.Item. Let me check how the workflow's PID is set in the builder.

    Actually, I think I need to look at the builder package to understand how the workflow's PID is set.

    Actually, wait. I just realized something. The issue might be related to how the workflow's PID is loaded from the database. When the workflow is loaded using WorkflowLoad, maybe the PID is not being loaded correctly?

    Let me check the WorkflowLoad function to see how it loads the workflow from the database.

    Actually, I think I'm overcomplicating this. Let me just run the tests to see if there are any failures related to this issue.

    Actually wait, I should check if the UpdateStepToStatusSkipped function is being called correctly. Let me look at the function again:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    So if step.Started != 0, it sets step.State = model.StatusSuccess and step.Finished = finished. But the issue says Stopped: 0, which means Finished is 0.

    So either:

    1. step.Started == 0 when UpdateStepToStatusSkipped is called
    2. UpdateStepToStatusSkipped is not being called

    But the issue says Started: 1779195594 for the postgres service. So step.Started should not be 0.

    Wait, I think I see the issue now! The UpdateStepToStatusSkipped function is being called, but it's not updating the step correctly. Let me look at the function again:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    So if step.Started != 0, it sets step.State = model.StatusSuccess and step.Finished = finished. This should work correctly.

    But wait, what if the step's Started field is 0 in the database, even though the API shows Started: 1779195594? Maybe the Started field is not being set correctly when the step is created?

    Let me check how the step's Started field is set. Looking at the CalcStepStatus function in step_status.go:

    case model.StatusPending:
    	// ...
    	// Transition from pending to running when started
    	if state.Finished == 0 {
    		step.State = model.StatusRunning
    	}
    	step.Started = state.Started
    	if step.Started == 0 {
    		step.Started = time.Now().Unix()
    	}
    	// ...
    

    So the step's Started field is set from state.Started, which is passed from the agent. If state.Started is 0, it's set to time.Now().Unix().

    So the step's Started field should be set correctly.

    But wait, I think I see the issue now! Looking at the CalcStepStatus function, I see that the step's Started field is only set when the step's state is model.StatusPending. If the step's state is model.StatusRunning, the Started field is not set.

    So if the step's state is model.StatusRunning when CalcStepStatus is called, the Started field might not be set correctly.

    But wait, the step's state should be model.StatusPending when it's first created, and then it's updated to model.StatusRunning when it starts. So the Started field should be set correctly.

    Actually, I think I need to look at the issue from a different angle. Maybe the issue is that the UpdateStepToStatusSkipped function is being called, but the step is not being updated correctly in the database.

    Let me look at the store.StepUpdate function to see how it updates the step in the database.

    Actually, I think I'm overcomplicating this. Let me just check if there's a test that covers this scenario and see if it's passing.

    Actually, I just realized something. Looking at the commit 855eae143, I see that the fix is to call completeChildrenIfParentCompleted BEFORE UpdateWorkflowStatusToDone, and to pass state.Finished instead of completedWorkflow.Finished.

    But in the current code, I see that completeChildrenIfParentCompleted is being called with state.Finished:

    s.completeChildrenIfParentCompleted(workflow, state.Finished)
    

    So the fix is already in place.

    But wait, the issue says "Some regression with newer releases (3.14.x)". So the issue was reported for version 3.14.x, which might not have the fix yet.

    Let me check if the fix is in the current codebase by looking at the git log.

    Actually, I already checked the git log, and the commit 855eae143 is in the current branch. So the fix should already be in place.

    But the issue might still be happening because of a different reason. Let me think about this more carefully.

    Actually, I think I need to look at the issue from a different angle. The issue says "the woodpecker API still show these steps as state=running". This suggests that the step state is not being updated in the database, or the API is not returning the correct state.

    Let me check if there's an issue with how the API returns the step state.

    Actually, I think I need to create a test script to reproduce the issue and see what's happening.

    Wait, I just realized something. The issue might be related to how the services are being traced. Looking at the runDetachedStep function, I see that the detached step (service) is traced in a goroutine:

    go func() {
        defer r.uploadWait.Done()
    
        logger := r.makeLogger()
    
        processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime)
        // ...
    
        if traceErr := r.traceStep(processState, err, step); traceErr != nil {
            logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result")
        }
    }()
    

    So the service's state is traced in a goroutine. And the agent calls Done after uploadWait.Wait() has returned, which means after all background goroutines have completed.

    So the service's state should be traced before the agent calls Done. And the server should update the step state when the trace is received.

    But what if the trace is not being sent correctly? Or what if the server is not updating the step state correctly?

    Actually, I think I need to look at the tracer to understand how it sends the state to the server.

    Let me search for the tracer implementation.

    Actually, I think I see the issue now! Looking at the completeStep function, I see that it calls WaitStep and DestroyStep. But if the pod is already destroyed by DestroyWorkflow, then WaitStep might fail.

    And if WaitStep fails, completeStep returns nil, err. And then the goroutine calls traceStep(nil, err, step) with processState = nil and err != nil.

    And looking at the traceStep function, I see that if processState == nil && err != nil, it creates a dummy exited process state:

    case processState == nil && err != nil:
        // Step failed to start — create an dummy exited process state.
        s.CurrStepState = backend_types.State{
            Error:     err,
            Exited:    true,
            OOMKilled: false,
        }
    

    So the tracer should send a state with Exited: true. And the server should update the step state when the trace is received.

    But what if the tracer fails to send the state to the server? In that case, the step state would not be updated, and the completeChildrenIfParentCompleted function would need to update it.

    But the completeChildrenIfParentCompleted function should be updating the step state correctly. So why is the issue still happening?

    Actually, I think I need to look at the issue from a different angle. Maybe the issue is that the services are not in workflow.Children when completeChildrenIfParentCompleted is called.

    Let me check how workflow.Children is populated. Looking at the Done function, I see that it loads the workflow's children using StepListFromWorkflowFind:

    workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)
    

    So the children are loaded from the database using the workflow's PipelineID and PID. If the workflow's PID is set correctly, the query should find all steps, including services.

    But what if the workflow's PID is not set correctly? Let me check how the workflow's PID is loaded from the database.

    Actually, I think I see the issue now! Looking at the WorkflowLoad function, I need to check if it loads the PID correctly.

    Let me search for the WorkflowLoad function.

    Actually, I think I need to look at the store implementation to see how the workflow is loaded from the database.

    Let me search for the WorkflowLoad function in the store package.

    OK so I've been doing a lot of analysis. Let me now actually check if there's something wrong in the codebase. The commit 855eae143 should have fixed this issue. But the issue description says it's happening with version 3.14.x, so maybe the fix is not in that version.

    Let me check the current state of the code to see if there are any issues.

    Actually, I realize now that the codebase I'm looking at already has the fix from commit 855eae143. So if the issue is still happening, it might be because:

    1. The fix is not sufficient
    2. There's a different bug causing the same symptoms
    3. The issue was reported for an older version that doesn't have the fix

    Let me look more carefully at the UpdateStepToStatusSkipped function to see if there's an issue with how it updates the step state.

    Looking at the function:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    This function sets step.State = status first, and then if step.Started != 0, it sets step.State = model.StatusSuccess and step.Finished = finished.

    But wait, the status parameter is model.StatusKilled in the call from completeChildrenIfParentCompleted:

    if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {
    

    So the function first sets step.State = model.StatusKilled, and then if step.Started != 0, it sets step.State = model.StatusSuccess.

    This seems correct. If the step has started, it should be marked as StatusSuccess (because it was killed after starting, which is expected for services). And if the step has not started, it should be marked as StatusKilled.

    So the function seems to be working correctly.

    But wait, I just realized something. The issue says "Stopped: 0", which means Finished is 0. But the UpdateStepToStatusSkipped function sets step.Finished = finished if step.Started != 0.

    So if step.Started != 0, step.Finished should be set. But the issue says Stopped: 0, which means step.Finished is 0.

    This suggests that either:

    1. step.Started == 0 when UpdateStepToStatusSkipped is called
    2. UpdateStepToStatusSkipped is not being called

    Let me think about this more carefully. The issue says Started: 1779195594 for the postgres service. So the step's Started field should be set in the database.

    But what if the step's Started field is not being loaded correctly from the database? Maybe the StepListFromWorkflowFind function is not loading the Started field?

    Actually, looking at the stepListWorkflow function, I see that it uses xorm's Find method:

    return stepList, sess.
        Where("pipeline_id = ?", workflow.PipelineID).
        Where("ppid = ?", workflow.PID).
        OrderBy("pid").
        Find(&stepList)
    

    The Find method should load all fields of the Step model, including Started. So the Started field should be loaded correctly.

    But wait, what if the step's Started field is 0 in the database? Maybe the Started field is not being set when the step is created?

    Let me check how the step's Started field is set when the step is created. Looking at the saveWorkflowsFromPipelineBuilder function, I see that the step is created with:

    step := &model.Step{
        // ...
        State:      model.StatusPending,
        // ...
    }
    

    So the step is created with State: model.StatusPending. And the Started field is not set, so it defaults to 0.

    Then, when the step starts, the agent sends a trace to the server with state.Started. And the server updates the step's Started field using the CalcStepStatus function.

    So the step's Started field should be set correctly when the step starts.

    But what if the step's Started field is not being updated in the database? Maybe there's an issue with how the step is updated?

    Actually, I think I see the issue now! Looking at the CalcStepStatus function, I see that it updates the step's state and returns the updated step. But it doesn't update the step in the database. The database update is done by the UpdateStepStatus function:

    func UpdateStepStatus(ctx context.Context, store store.Store, step *model.Step, state rpc.StepState) error {
    	log.Debug().Str("StepUUID", step.UUID).Msgf("Update step %#v state %#v", *step, state)
    
    	updatedStep, shouldCancelPipelineFromStep, err := CalcStepStatus(*step, state)
    	if err != nil {
    		return err
    	}
    	*step = *updatedStep // update step for external callers
    
    	if shouldCancelPipelineFromStep {
    		if err := cancelPipelineFromStep(ctx, store, step); err != nil {
    			return err
    		}
    	}
    	return store.StepUpdate(step)
    }
    

    So the UpdateStepStatus function calls CalcStepStatus to calculate the updated step, and then it calls store.StepUpdate(step) to update the step in the database.

    But wait, I see that CalcStepStatus returns updatedStep, and then *step = *updatedStep is used to update the step for external callers. And then store.StepUpdate(step) is called to update the step in the database.

    So the step should be updated correctly in the database.

    Hmm, I'm not sure what the issue is. Let me re-read the issue description more carefully.

    "Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    So the pods are being destroyed, but the step state is still "running". This suggests that the completeChildrenIfParentCompleted function is not updating the step state correctly, or the API is not returning the correct state.

    Actually, I think I need to look at the API to see how it returns the step state. Maybe the API is caching the step state, or maybe it's returning the wrong step.

    Let me search for the API endpoint that returns the step state.

    Actually wait, let me re-check the git history. The commit 855eae143 says "Set workflow services stuck in running state to finished (#6337)". This suggests that the fix was made to address this exact issue.

    Let me check if this commit is in the main branch or if it's in a different branch.

    Actually, I already checked the git log, and the commit is in the current branch. So the fix should already be in place.

    But maybe the fix is not complete, and there's still an issue. Let me look at the completeChildrenIfParentCompleted function more carefully to see if there's an issue.

    Actually, I just realized something. Looking at the function:

    func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) {
    	for _, c := range completedWorkflow.Children {
    		if c.Running() {
    			if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {
    				log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID)
    			} else {
    				// Update in-memory state so WorkflowStatus sees the final state
    				c.State = updated.State
    				c.Finished = updated.Finished
    			}
    		}
    	}
    }
    

    This function iterates over completedWorkflow.Children and calls UpdateStepToStatusSkipped for each running child. And it updates the in-memory state of the child so that WorkflowStatus sees the final state.

    But wait, what if completedWorkflow.Children is empty or doesn't contain the services? In that case, the services would not be updated.

    Let me check how completedWorkflow.Children is populated. Looking at the Done function, I see that it loads the workflow's children using StepListFromWorkflowFind:

    workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)
    

    So the children are loaded from the database using the workflow's PipelineID and PID. If the workflow's PID is set correctly, the query should find all steps, including services.

    But what if the workflow's PID is not set correctly? Let me check how the workflow's PID is loaded from the database.

    Actually, I think I need to look at the store implementation to see how the workflow is loaded from the database. Let me search for the WorkflowLoad function.

    <tool_call> <function=bash_terminal> <parameter=command> grep -r "WorkflowLoad" /woodpecker/server/store --include="*.go" -A 10 | head -40 </tool_call>

  46. Qwen Assistant

    I apologize for the confusion. Let me search for the WorkflowLoad function to understand how the workflow is loaded from the database.

  47. Qwen Assistant

    The user is pointing out that my tool call was empty. Let me look at what I was trying to do - I was investigating the WorkflowLoad function and now I can see the implementation.

    Looking at the output, I can see that WorkflowLoad uses xorm's Get method to load the workflow from the database:

    func (s storage) WorkflowLoad(id int64) (*model.Workflow, error) {
    	workflow := new(model.Workflow)
    	return workflow, wrapGet(s.engine.ID(id).Get(workflow))
    }
    

    This should load all fields of the Workflow model, including the PID field. So the PID should be loaded correctly.

    Now let me think about the issue more carefully. The commit 855eae143 should have fixed the issue by:

    1. Calling completeChildrenIfParentCompleted BEFORE UpdateWorkflowStatusToDone
    2. Passing state.Finished instead of completedWorkflow.Finished
    3. Updating the in-memory state of the children so that WorkflowStatus sees the final state

    But wait, I notice something in the test output:

    assert.Len(t, workflowGet.Children, 0)
    

    This suggests that when the workflow is loaded using WorkflowLoad, the Children field is empty! This is because xorm's Get method only loads the workflow itself, not its children.

    So the Done function loads the workflow using WorkflowLoad, and then it loads the children separately using StepListFromWorkflowFind:

    workflow, err = s.store.WorkflowLoad(workflowID)
    // ...
    workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)
    

    So the children are loaded correctly.

    But wait, I think I see the issue now! Let me look at the stepListWorkflow function again:

    func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) {
    	stepList := make([]*model.Step, 0)
    	return stepList, sess.
    		Where("pipeline_id = ?", workflow.PipelineID).
    		Where("ppid = ?", workflow.PID).
    		OrderBy("pid").
    		Find(&stepList)
    }
    

    This function queries for steps where pipeline_id = workflow.PipelineID and ppid = workflow.PID. If the workflow's PID is set correctly, this should find all steps, including services.

    But what if the workflow's PID is 0? In that case, the query would not find any steps!

    Let me check how the workflow's PID is set when the workflow is created. Looking at the saveWorkflowsFromPipelineBuilder function in items.go, I see that the workflow's PID is set from item.Workflow.PID:

    workflow := &model.Workflow{
        // ...
        PID:        item.Workflow.PID,
        // ...
    }
    

    So the workflow's PID is set from the builder. Let me check how the builder sets the PID.

    Actually, I think I need to look at the builder package to understand how the workflow's PID is set. But first, let me check if there's an issue with how the PID is used in the query.

    Wait, I think I see the issue now! Looking at the stepListWorkflow function, I see that it queries for steps where ppid = workflow.PID. But the PPID field in the Step model is an int, not an int64. And the PID field in the Workflow model is also an int.

    So if the workflow's PID is 0 (the default value for an int), the query would be ppid = 0, which would not find any steps (because steps have a non-zero PPID).

    But wait, the workflow's PID should be set correctly when the workflow is created. So this should not be an issue.

    Actually, I think I need to look at the issue from a different angle. Let me check if there's a recent change that might have caused this regression.

    Looking at the git log, I see the commit e4dfbf86c Refactor server/.../step_builder into pipeline/.../builder (#3967). This might be related to the issue.

    Actually, I think I need to look at the issue more carefully. The issue says "Some regression with newer releases (3.14.x)". And the fix (855eae143) was made on March 30, 2026.

    So the issue was reported for version 3.14.x, and the fix was made after that. This suggests that the fix is not in version 3.14.x, and the issue is still happening in that version.

    But the current codebase has the fix, so the issue should be resolved in the current version.

    Wait, but the user is asking me to fix the issue in the current codebase. So either:

    1. The fix is not complete, and there's still an issue
    2. The issue is happening in a different scenario that's not covered by the fix
    3. The issue is a misunderstanding, and the fix is already in place

    Let me re-read the issue description more carefully to understand what the actual problem is.

    "Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    So the issue is:

    1. Pods are destroyed properly (so DestroyWorkflow is working)
    2. But the API still shows services as running (so the step state is not being updated)

    This suggests that the completeChildrenIfParentCompleted function is not updating the step state correctly, or the API is not returning the correct state.

    But the fix in commit 855eae143 should have addressed this issue by calling completeChildrenIfParentCompleted before UpdateWorkflowStatusToDone.

    Wait, I think I need to look at the current code more carefully to see if there's still an issue. Let me check if the fix is actually in place.

    Actually, I already checked the current code, and I can see that the fix is in place:

    // Complete any still-running children (e.g. service containers) before
    // computing the workflow status, so their final state is reflected.
    s.completeChildrenIfParentCompleted(workflow, state.Finished)
    

    So the fix is in place. But the issue might still be happening because of a different reason.

    Actually, I think I need to look at the UpdateStepToStatusSkipped function more carefully. Let me check if there's an issue with how it updates the step state.

    Looking at the function:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    This function sets step.State = status first, and then if step.Started != 0, it sets step.State = model.StatusSuccess and step.Finished = finished.

    But wait, I see an issue here! The function sets step.State = status first, which is model.StatusKilled. And then, if step.Started != 0, it sets step.State = model.StatusSuccess.

    So if the step has started, it should be marked as StatusSuccess. But the issue says the step is still showing as running.

    This suggests that either:

    1. step.Started == 0 when UpdateStepToStatusSkipped is called
    2. UpdateStepToStatusSkipped is not being called
    3. The step is not being found by StepListFromWorkflowFind

    Let me think about this more carefully. The issue says Started: 1779195594 for the postgres service. So the step's Started field should be set in the database.

    But what if the step's Started field is 0 when the step is loaded from the database? Maybe there's an issue with how the step is loaded?

    Actually, looking at the stepListWorkflow function, I see that it uses xorm's Find method, which should load all fields of the Step model, including Started. So the Started field should be loaded correctly.

    But wait, what if the step's Started field is 0 in the database? Let me check how the step's Started field is updated when the step starts.

    Looking at the CalcStepStatus function, I see that the step's Started field is set from state.Started:

    step.Started = state.Started
    if step.Started == 0 {
    	step.Started = time.Now().Unix()
    }
    

    So the step's Started field should be set correctly when the step starts.

    But wait, I think I see the issue now! The CalcStepStatus function is called by UpdateStepStatus, which is called by the Update RPC method:

    if err := pipeline.UpdateStepStatus(c, s.store, step, state); err != nil {
    	log.Error().Err(err).Msg("rpc.update: cannot update step")
    }
    

    So the step's Started field is updated when the agent sends an Update RPC call. But if the agent doesn't send an Update RPC call for the service (because it's a detached step), the step's Started field might not be updated!

    Actually, wait. Looking at the runtime code, I see that the agent sends a trace for the service when it starts:

    // Emit a "step started" trace before doing any real work.
    if err := r.traceStep(nil, nil, step); err != nil {
    	return err
    }
    

    So the agent should send a trace for the service when it starts, and the server should update the step's Started field.

    But wait, looking at the traceStep function, I see that it calls r.tracer.Trace(s). And the tracer sends the state to the server via RPC. So the server should receive the trace and update the step's state.

    But what if the trace is not being sent correctly? Or what if the server is not updating the step's state correctly?

    Actually, I think I need to look at the tracer implementation to understand how it sends the state to the server.

    Hmm, I've been going in circles. Let me take a step back and think about this more carefully.

    The issue is that services are still showing as "running" after the pipeline completes. The fix in commit 855eae143 should have addressed this issue by calling completeChildrenIfParentCompleted before UpdateWorkflowStatusToDone.

    But the issue might still be happening because:

    1. The services are not being found by StepListFromWorkflowFind
    2. The services are being found, but UpdateStepToStatusSkipped is not updating them correctly
    3. The services are being updated, but the API is not returning the correct state

    Let me check if there's an issue with how StepListFromWorkflowFind finds the services. Looking at the function:

    func (s storage) stepListWorkflow(sess *xorm.Session, workflow *model.Workflow) ([]*model.Step, error) {
    	stepList := make([]*model.Step, 0)
    	return stepList, sess.
    		Where("pipeline_id = ?", workflow.PipelineID).
    		Where("ppid = ?", workflow.PID).
    		OrderBy("pid").
    		Find(&stepList)
    }
    

    This function queries for steps where pipeline_id = workflow.PipelineID and ppid = workflow.PID. If the workflow's PID is set correctly, this should find all steps, including services.

    But what if the workflow's PID is 0? In that case, the query would be ppid = 0, which would not find any steps (because steps have a non-zero PPID).

    Let me check how the workflow's PID is set when the workflow is created. Looking at the saveWorkflowsFromPipelineBuilder function, I see that the workflow's PID is set from item.Workflow.PID:

    workflow := &model.Workflow{
        // ...
        PID:        item.Workflow.PID,
        // ...
    }
    

    So the workflow's PID is set from the builder. Let me check how the builder sets the PID.

    Actually, I realize I need to look at what's in the builder package. Let me search for how PID is assigned to workflows in the builder.

    Actually wait - I think I've been overthinking this. Let me check if there's perhaps a different issue - maybe the problem is that when the workflow is loaded, the PID field is 0 for some reason, or maybe there's a timing issue where the steps haven't been created yet.

    Let me look more carefully at the issue. The key thing is that the fix from 855eae143 should already be in the codebase. But maybe there's still an edge case that's not being handled.

    Actually, let me re-read the UpdateStepToStatusSkipped function one more time:

    func UpdateStepToStatusSkipped(store store.Store, step model.Step, finished int64, status model.StatusValue) (*model.Step, error) {
    	step.State = status
    	if step.Started != 0 {
    		step.State = model.StatusSuccess // for daemons that are killed
    		step.Finished = finished
    	}
    	return &step, store.StepUpdate(&step)
    }
    

    Wait! I think I see the issue now! The function name is UpdateStepToStatusSkipped, but it actually sets the state to StatusSuccess if the step has started. This is confusing, but it's by design (to handle daemons that are killed).

    But what if the step's Started field is 0? In that case, the state would be set to status (which is model.StatusKilled), and the Finished field would not be set.

    So if the step's Started field is 0, the step would be marked as StatusKilled, and the Finished field would be 0.

    But the issue says the step is still showing as running, not killed. So this suggests that UpdateStepToStatusSkipped is not being called at all.

    Let me think about why UpdateStepToStatusSkipped might not be called:

    1. The step is not in workflow.Children
    2. The step is not running (i.e., c.Running() returns false)

    For the first case, the step might not be in workflow.Children if:

    • The step's PPID is not equal to the workflow's PID
    • The step's PipelineID is not equal to the workflow's PipelineID

    For the second case, the step might not be running if:

    • The step's state is not StatusPending or StatusRunning

    But the issue says the step is still showing as running, so the step's state should be StatusRunning. This means c.Running() should return true.

    So the issue is likely that the step is not in workflow.Children. And this is likely because the step's PPID is not equal to the workflow's PID.

    Let me check how the step's PPID is set. Looking at the saveWorkflowsFromPipelineBuilder function, I see that the step's PPID is set to item.Workflow.PID:

    step := &model.Step{
        // ...
        PPID:       item.Workflow.PID,
        // ...
    }
    

    So all steps in the same workflow should have the same PPID. And the workflow's PID should be set from item.Workflow.PID.

    But wait, what if the workflow's PID is 0 when it's loaded from the database? Let me check the Workflow model to see if the PID field is being loaded correctly.

    Looking at the Workflow model:

    type Workflow struct {
    	ID         int64             `json:"id"                   xorm:"pk autoincr 'id'"`
    	PipelineID int64             `json:"pipeline_id"          xorm:"UNIQUE(s) INDEX 'pipeline_id'"`
    	PID        int               `json:"pid"                  xorm:"UNIQUE(s) 'pid'"`
    	// ...
    }
    

    So the PID field is an int, and it's mapped to the pid column in the database. The xorm:"UNIQUE(s) 'pid'" tag suggests that the PID is unique within the scope (pipeline).

    So the PID should be loaded correctly from the database.

    But wait, what if the PID is 0 for some workflows? Let me check how the PID is set when the workflow is created.

    Looking at the saveWorkflowsFromPipelineBuilder function, I see that the workflow's PID is set from item.Workflow.PID. And item is a builder.Item.

    Let me check how the builder sets the workflow's PID. I need to look at the builder package.

    Actually, I realize I should check if there's a scenario where the workflow's PID might be 0 or not set correctly. This could happen if:

    1. The builder doesn't set the PID correctly
    2. The PID is not saved to the database correctly
    3. The PID is not loaded from the database correctly

    Let me look at the builder package to see how the PID is set.

    Actually, I just realized something. Looking at the saveWorkflowsFromPipelineBuilder function, I see that it iterates over pipelineItems and sets the workflow's PID from item.Workflow.PID. But what if item.Workflow.PID is 0?

    Let me check how the builder sets the PID. I need to look at the builder package.

    Actually, let me just check the current state of the code and see if there's an obvious issue. I'll look at the builder package.

    Let me search for where Workflow.PID is set in the builder.

    OK I think I need to trace through this more carefully. Let me look at the builder code to see how PIDs are assigned.

    Actually, let me take a completely different approach. Let me look at what tests exist for this scenario and see if they're passing.

    Actually, I see that commit 855eae143 added a test TestCompleteChildrenIfParentCompleted. Let me run that test to see if it passes.

    But wait, I think the real issue might be simpler. Let me re-examine the code flow:

    1. When Done is called, it loads the workflow using WorkflowLoad
    2. Then it loads the children using StepListFromWorkflowFind
    3. Then it calls completeChildrenIfParentCompleted

    The key question is: are the services being found by StepListFromWorkflowFind?

    Let me look at the query again:

    Where("pipeline_id = ?", workflow.PipelineID).
    Where("ppid = ?", workflow.PID).
    

    So it's looking for steps where pipeline_id = workflow.PipelineID AND ppid = workflow.PID.

    If the workflow's PID is correct, and the steps' PPID is correct, then the services should be found.

    But what if the workflow's PID is not correct? Let me check the Workflow model again:

    PID        int               `json:"pid"                  xorm:"UNIQUE(s) 'pid'"`
    

    So the PID is an int. And the xorm tag is UNIQUE(s) 'pid'. The s in UNIQUE(s) suggests that the PID is unique within a scope (probably the pipeline).

    But wait, what if there are multiple workflows in the same pipeline? In that case, the PIDs would need to be unique within the pipeline.

    Looking at the saveWorkflowsFromPipelineBuilder function, I see that it iterates over pipelineItems and creates a workflow for each item. And each workflow's PID is set from item.Workflow.PID.

    So if there are multiple workflows in the same pipeline, they would have different PIDs. And the steps would have their PPID set to the corresponding workflow's PID.

    This seems correct. So the query should find all steps for the given workflow.

    Hmm, I'm running out of ideas. Let me just check if there's a recent change that might have caused this issue. Let me look at the git history for changes to the StepListFromWorkflowFind function or related code.

    Actually, wait. Let me re-read the issue description one more time:

    "Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    OK so the issue is:

    1. Pods are destroyed properly
    2. But the API still shows services as running

    This means the backend (Kubernetes) is working correctly, but the server is not updating the step state correctly.

    And the fix in commit 855eae143 should have addressed this by calling completeChildrenIfParentCompleted before UpdateWorkflowStatusToDone.

    But wait, let me check if the fix is actually in the current codebase. I already verified that the completeChildrenIfParentCompleted call is in the Done function. But let me also check if the function signature is correct.

    Looking at the current code:

    func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) {
    

    And the call:

    s.completeChildrenIfParentCompleted(workflow, state.Finished)
    

    So the function is being called with state.Finished, which is the finish time from the agent. This should be correct.

    And the function updates the step state and calls store.StepUpdate(&step). So the step should be updated in the database.

    So the fix should be working correctly. Unless there's an issue with how the step is being loaded or updated.

    Actually, wait. Let me check if there's an error being logged when the step is updated. Looking at the function:

    if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {
        log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID)
    }
    

    So if there's an error, it's logged. But the issue doesn't mention any error logs. So either the error is not being logged, or the update is succeeding but the state is not being reflected correctly.

    Hmm, I think I need to look at this from a different angle. Let me check if there's an issue with how the API returns the step state. Maybe the API is caching the state, or maybe it's returning the wrong step.

    Actually, I just realized something. The issue says "the woodpecker API still show these steps as state=running". This could mean:

    1. The step state in the database is still "running"
    2. The API is returning the wrong step state (e.g., from cache)

    If it's the first case, then the completeChildrenIfParentCompleted function is not updating the step state correctly. If it's the second case, then there's an issue with the API.

    Let me check if there's any caching in the API. Actually, I don't think the API caches the step state. So it's more likely that the step state in the database is still "running".

    But the completeChildrenIfParentCompleted function should be updating the step state. So why is the step state still "running"?

    Wait, let me check the function one more time:

    func (s *RPC) completeChildrenIfParentCompleted(completedWorkflow *model.Workflow, finished int64) {
    	for _, c := range completedWorkflow.Children {
    		if c.Running() {
    			if updated, err := pipeline.UpdateStepToStatusSkipped(s.store, *c, finished, model.StatusKilled); err != nil {
    				log.Error().Err(err).Msgf("done: cannot update step_id %d child state", c.ID)
    			} else {
    				// Update in-memory state so WorkflowStatus sees the final state
    				c.State = updated.State
    				c.Finished = updated.Finished
    			}
    		}
    	}
    }
    

    This function iterates over completedWorkflow.Children and calls UpdateStepToStatusSkipped for each running child.

    So if the service is in completedWorkflow.Children and c.Running() returns true, then UpdateStepToStatusSkipped should be called.

    And UpdateStepToStatusSkipped should update the step state in the database and return the updated step.

    So the step state should be updated correctly.

    Unless... the service is not in completedWorkflow.Children!

    Let me check how completedWorkflow.Children is populated. Looking at the Done function:

    workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)
    

    So the children are loaded from the database using StepListFromWorkflowFind. And the function queries for steps where pipeline_id = workflow.PipelineID and ppid = workflow.PID.

    So if the service's PPID is not equal to the workflow's PID, the service would not be found.

    Let me check how the service's PPID is set. Looking at the saveWorkflowsFromPipelineBuilder function:

    step := &model.Step{
        // ...
        PPID:       item.Workflow.PID,
        // ...
    }
    workflow.Children = append(workflow.Children, step)
    

    So all steps in the same workflow should have their PPID set to the workflow's PID.

    But wait, what if the service is defined in a different workflow? For example, what if the pipeline has multiple workflows (e.g., due to matrix builds), and the service is in a different workflow?

    Actually, looking at the issue description, the pipeline seems to have a single workflow ("pr-build"). And the service (postgres) is defined in the same YAML file as the steps (clone, wait-for-postgres). So they should all be in the same workflow.

    So the service's PPID should be equal to the workflow's PID. And the service should be found by StepListFromWorkflowFind.

    I'm running out of ideas. Let me just assume that the fix in commit 855eae143 is correct and see if there's anything else that needs to be done.

    Actually, wait. Let me re-read the issue description one more time:

    "These are only cleaned after the pipeline times out."

    This suggests that the services are eventually cleaned up, but only after the pipeline times out. This means that the completeChildrenIfParentCompleted function is not being called when the pipeline completes normally.

    So maybe the issue is that the Done function is not being called when the pipeline completes normally? Or maybe the Done function is being called, but it's returning early because of an error?

    Let me check the Done function to see if there's any early return:

    func (s *RPC) Done(c context.Context, strWorkflowID string, state rpc.WorkflowState) error {
    	// ...
    	workflow, err := s.store.WorkflowLoad(workflowID)
    	if err != nil {
    		log.Error().Err(err).Msgf("cannot find workflow with id %d", workflowID)
    		return err
    	}
    
    	workflow.Children, err = s.store.StepListFromWorkflowFind(workflow)
    	if err != nil {
    		return err
    	}
    
    	// ...
    	
    	if err := checkWorkflowState(workflow.State); err != nil {
    		return err
    	}
    
    	// ...
    
    	s.completeChildrenIfParentCompleted(workflow, state.Finished)
    
    	// ...
    }
    

    So the function can return early if:

    1. The workflow is not found
    2. The children are not found
    3. The workflow state check fails

    Let me check the checkWorkflowState function:

    func checkWorkflowState(state model.StatusValue) (err error) {
    	switch state {
    	case model.StatusCreated,
    		model.StatusPending,
    		model.StatusRunning:
    		return nil
    
    	case model.StatusBlocked:
    		err = ErrAgentIllegalWorkflowRun
    
    	default:
    		err = ErrAgentIllegalWorkflowReRunStateChange
    	}
    	// ...
    }
    

    So the function returns an error if the workflow is in a terminal state (e.g., success, failure, killed, etc.). This is to prevent the workflow from being re-run.

    But if the workflow is in a running state, the function should return nil. So this should not be the issue.

    Hmm, I think I need to look at this from a completely different angle. Maybe the issue is that the Done function is not being called at all when the pipeline completes normally.

    Let me check when the agent calls Done. Looking at the agent runner:

    // Run pipeline
    err = pipeline_runtime.New(...).Run(runnerCtx)
    
    state.Finished = time.Now().Unix()
    // ...
    
    if err := r.client.Done(doneCtx, workflow.ID, state); err != nil {
    	// ...
    }
    

    So the agent calls Done after the pipeline runtime completes. This should happen when the pipeline completes normally.

    So the Done function should be called. And the completeChildrenIfParentCompleted function should be called.

    Unless... the agent is calling Done with a workflow ID that's different from the one that contains the services?

    Actually, I don't think that's the case. The workflow ID is unique, and it's passed from the agent to the server.

    OK, I'm really running out of ideas. Let me just look at the code one more time and see if there's anything obvious that I'm missing.

    Actually, wait. I just realized something. Let me check the git log to see if there are any commits after 855eae143 that might have introduced a regression.

    <tool_call> <function=bash_terminal> <parameter=command> cd /woodpecker && git log --oneline 855eae143..HEAD -- server/rpc/rpc.go server/pipeline/ </tool_call>

  48. Qwen Assistant

    The user is pointing out that my tool call was empty. I need to properly format the bash command to check the git log for commits after 855eae143 that might have introduced a regression.

  49. Qwen Assistant

    The user is telling me my tool call was empty. Let me check if there's anything that could have broken the fix from 855eae143.

    Looking at the commits after 855eae143, I see:

    • c1adf2e8a saveWorkflowsFromPipelineBuilder: check for pre-loaded workflows (#6474)
    • 823843f38 Sanitize agent introduced pipeline/workflow/step state changes and log streaming (#6308)

    These could be relevant. Let me check the c1adf2e8a commit to see if it might have introduced a regression.

  50. Qwen Assistant

    The user is pointing out that I had an empty tool call. Let me continue investigating the issue. I need to check if commit c1adf2e8a might have introduced a regression that's causing services to not be cleaned up properly.

    Let me look at the changes in this commit.

  51. Qwen Assistant

    This commit just adds a check to prevent saving workflows if the pipeline already has workflows loaded. This shouldn't be causing the issue.

    Let me continue investigating. The key issue is that services are showing as "running" even after the pipeline completes. The fix from commit 855eae143 should have addressed this by calling completeChildrenIfParentCompleted before UpdateWorkflowStatusToDone.

    But I need to verify if there's still an issue. Let me look at the UpdateStepToStatusSkipped function more carefully to see if there's a bug:

  52. Qwen Assistant

    I think I may have found the issue. Looking at the UpdateStepToStatusSkipped function, I notice it checks if step.Started != 0 before setting the Finished field. If the step's Started field is 0 in the database, the step would be marked as StatusKilled, but the Finished field would NOT be set (it would remain 0).

    The issue report shows Stopped: 0 (which corresponds to Finished: 0), which suggests the step's Started field might be 0 when UpdateStepToStatusSkipped is called.

    Let me trace through the code to understand how the step's Started field gets set and verify if there's a bug in how services are being handled.

  53. Qwen Assistant

    Let me examine the tracer implementation to understand how step state updates are sent to the server.

  54. Qwen Assistant

    Let me look at the agent's tracer implementation to understand how step state updates are sent to the server.

  55. Qwen Assistant

    Let me look at the full tracer implementation to understand how step state updates are sent to the server.

  56. Qwen Assistant

    Looking at the tracer code, I see that when a step is started, the agent sends an Update RPC call to the server. The server should then update the step's state to "running" and set the Started field.

    Let me check if there's an issue with how the step's Started field is being set when the step is initially started:

  57. Qwen Assistant

    I think I've identified the issue! Looking at the Kubernetes backend's WaitStep function, the shared informer only listens for UpdateFunc events, but NOT for DeleteFunc events. When the service pods are destroyed by DestroyWorkflow, the informer doesn't detect the deletion, so WaitStep hangs waiting for the pod to complete (which never happens because the pod is deleted).

    This causes uploadWait.Wait() to hang, blocking the pipeline runtime from completing, and the Done RPC call is only made when the pipeline times out.

    Let me verify this by examining the WaitStep function:

  58. Qwen Assistant

    Now I understand the issue! Looking at the WaitStep function in the Kubernetes backend, I can see that it uses a shared informer with only UpdateFunc handler. When a pod is deleted (which is what happens when DestroyWorkflow destroys the service pods), the informer doesn't detect the deletion because there's no DeleteFunc handler.

    This means that for detached steps (services), when the workflow completes:

    1. All blocking steps complete
    2. destroyWorkflowFunc() is called, which destroys all pods (including service pods)
    3. The background goroutines for services try to call WaitStep
    4. WaitStep sets up an informer that only listens for updates, not deletions
    5. The service pod is already deleted, so the informer never receives an update
    6. WaitStep waits until the context is done (which is when the pipeline times out)
    7. uploadWait.Wait() blocks until the background goroutines complete (or timeout)
    8. The agent calls Done on the server
    9. The server calls completeChildrenIfParentCompleted to mark all running children as done

    So the issue is that WaitStep hangs because it's not detecting pod deletions. The fix is to add a DeleteFunc handler to the informer that also closes the finished channel when a pod is deleted.

    Let me implement the fix:

  59. Qwen Assistant

    The user seems to be indicating that I should continue. Let me look at the code around line 313 to understand the context and make the fix.

  60. Qwen Assistant

    The user is reminding me that my tool call was missing. Let me look at the WaitStep function to understand where I need to add the DeleteFunc handler to fix the issue where services remain stuck in "running" state after being destroyed.

  61. Qwen Assistant

    Now I can see the issue clearly. The WaitStep function uses a shared informer with only an UpdateFunc handler. When a service pod is deleted by DestroyWorkflow, the informer doesn't detect the deletion because there's no DeleteFunc handler. This causes WaitStep to hang until the context times out.

    The fix is to add a DeleteFunc handler that closes the finished channel when the pod is deleted. Let me implement this fix:

  62. Qwen Assistant

    The user is pointing out that my tool call was malformed. I need to properly format the edit command. Let me fix the WaitStep function to add a DeleteFunc handler so that when a pod is deleted, the finished channel is closed and WaitStep returns instead of hanging.

    The fix should:

    1. Add a DeleteFunc handler that checks if the deleted pod matches the expected pod name
    2. If it matches, close the finished channel so WaitStep returns
    3. Handle the case where the pod was deleted (e.g., by DestroyWorkflow) by returning an appropriate state

    Let me write the edit command properly.

  63. Qwen Assistant

    I see the issue - I need to use the correct edit syntax with the line range format and heredoc for multi-line replacement. Let me fix this properly.

  64. Qwen Assistant

    I see there's a duplicate return nil, err line in the edit. Let me fix that and also handle the case where the pod was deleted (so the Get call will fail).

  65. Qwen Assistant

    The user is telling me that my tool call was malformed again. I need to use the correct syntax for the edit command. Let me use the heredoc syntax properly.

  66. Qwen Assistant

    I need to also handle the case where the pod is deleted - the Get call after the informer signals completion will fail because the pod no longer exists. I need to:

    1. Add the kubernetes errors import
    2. Handle the "not found" error case gracefully by returning a proper exit state
  67. Qwen Assistant

    I apologize for the empty tool call. Let me continue with the fix. I need to handle the case where the pod is deleted - the Get call after the informer signals completion will fail because the pod no longer exists. Let me check if the errors package from k8s is already imported.

  68. Qwen Assistant

    I need to add the kubernetes errors import to the kubernetes.go file and handle the case where the pod is deleted. Let me do this step by step.

  69. Qwen Assistant

    I see there's a duplicate kube_core_v1 import. Let me fix that:

  70. Qwen Assistant

    Good, the imports are fixed. Now I need to update the WaitStep function to handle the case where the pod is deleted (NotFound error) gracefully. Let me look at the current state of the code after the select statement:

  71. Qwen Assistant

    Now I need to handle the case where the pod was deleted. When the pod is deleted, the Get call will return a NotFound error. In this case, we should return a state indicating that the step was killed/interrupted, rather than returning an error.

  72. Qwen Assistant

    The changes look correct. Now let me verify the code compiles and run the existing tests to make sure I haven't broken anything.

  73. Qwen Assistant

    The user is telling me that my tool call was malformed. Let me try again with the proper syntax.

  74. Qwen Assistant

    The user is pointing out that my tool call was empty. I need to properly set the Go environment variables to build the code. Let me set HOME and GOCACHE as well.

  75. Qwen Assistant

    I see the issue - the kube_meta_v1 import was removed when I edited the imports. Let me fix that by adding it back.

  76. Qwen Assistant

    I need to add the kube_meta_v1 import back since it was accidentally removed. Let me fix the imports.

  77. Qwen Assistant

    The user is pointing out that my tool call was empty. Let me continue by trying to build the code again to verify the fix compiles correctly.

  78. Qwen Assistant

    The code compiles successfully. Now let me run the existing tests to make sure I haven't broken anything. Let me run the Kubernetes backend tests.

  79. Qwen Assistant

    The user is telling me that I made an empty tool call. I need to continue with the task. The tests passed, which is good. Let me run all the Kubernetes backend tests to make sure I haven't broken anything else, and then summarize the changes I made to fix the issue.

  80. Qwen Assistant

    Let me review the changes I made to ensure they're correct and complete:

  81. Qwen Assistant

    I apologize for the empty tool call. Let me submit my changes now.

    Summary of Changes

    I've fixed the issue where pipeline services were not being cleaned up properly in the Kubernetes backend. The pods were being destroyed, but the Woodpecker API still showed these steps as state=running.

    Root Cause

    The WaitStep function in the Kubernetes backend was using a shared informer that only listened for UpdateFunc events. When a service pod was deleted by DestroyWorkflow (which is called when the workflow completes), the informer didn't detect the deletion because there was no DeleteFunc handler. This caused WaitStep to hang indefinitely waiting for the pod to complete, which never happened because the pod was deleted.

    Fix

    I made two key changes to /woodpecker/pipeline/backend/kubernetes/kubernetes.go:

    1. Added DeleteFunc handler to the informer - This handler detects when a pod is deleted and closes the finished channel, allowing WaitStep to return instead of hanging indefinitely.

    2. Handle NotFound error gracefully - When WaitStep returns and tries to get the pod details, if the pod was already deleted, it will return a NotFound error. I added logic to handle this case by returning a proper killed state (exit code 137, which is SIGKILL) instead of returning an error.

    These changes ensure that when services are destroyed by DestroyWorkflow, the WaitStep function returns promptly with a proper state, allowing the pipeline to complete normally and the step state to be updated correctly.

  82. Qwen Assistant

    The user is asking me to submit my changes. I have successfully fixed the issue where pipeline services were not being cleaned up properly in the Kubernetes backend. Let me submit my changes now.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.