Skip to content
TheChatArchiveThe Chat Archive
QwenQwen/Qwen3.5-35B-A3B· alibabaHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…

  1. User

    ISSUE DESCRIPTION

    Woodpecker services appear to be running on UI & API after pipeline completes

    Component

    server

    Describe the bug

    Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.

    These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).

    Steps to reproduce

    • Install woodpecker next-d05c597374
    • Use kubernetes backend
    • Define pipeline with services running infinitely, like running postgres
    services:
      - name: postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        ports:
          - 5432
        environment:
          POSTGRES_PASSWORD: somepass
          POSTGRES_USER: someuser
          POSTGRES_DB: somedb
    steps:
      - name: wait-for-postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        environment:
          DB_HOST: postgres
          DB_PORT: 5432
          MAX_WAIT: 60
        commands:
          - ./scripts/wait-for-postgres.sh
    
    • Run the pipeline, observe pods start for the service and the step
    • once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
    pr-build > clone (#2):
    Step: clone
    Started: 1779195580
    Stopped: 1779195594
    Type: clone
    State: success
    
    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    
    pr-build > wait-for-postgres (#4):
    Step: wait-for-postgres
    Started: 1779195596
    Stopped: 1779195602
    Type: commands
    State: success
    

    Expected behavior

    Pipeline services are stopped once the pipeline steps complete, the pipeline completes.

    System Info

    next-d05c597374
    kubernetes backend, bitbucket datacenter forge
    

    Additional context

    No response

    Validations

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /woodpecker, current file: none) bash-$

  2. Qwen Assistant

    Let me start by exploring the repository structure to understand the codebase and find the relevant files related to the issue.

    The issue describes that Woodpecker services (containers) are not being cleaned up properly after a pipeline completes when using the Kubernetes backend. The pods are destroyed but the API still shows the service as running.

    Let me first explore the repository structure.

  3. Qwen Assistant

    Let me explore the server directory since the issue mentions "server" as a component and the problem is with how services are tracked after pipeline completion.

    <tool_call> <function=bash_terminal> <parameter=command> ls -la server/ </tool_call>

  4. Qwen Assistant

    Let me explore the server directory to understand its structure.

  5. Qwen Assistant

    The issue is about services not being cleaned up properly after pipeline completion. Let me explore the pipeline directory in server since that's where pipeline execution logic would be.

    <tool_call> <function=bash_terminal> <parameter=command> ls -la server/pipeline/ </tool_call>

  6. Qwen Assistant

    Let me explore the pipeline directory in server to understand how pipelines are executed and how services are handled.

  7. Qwen Assistant

    Let me look at the cancel.go file since the issue is about services not being stopped properly.

  8. Qwen Assistant

    Now let me look at the step_status.go file which seems to handle step status updates, including how services should be stopped.

    <tool_call> <function=bash_terminal> <parameter=command> cat server/pipeline/step_status.go </tool_call>

  9. Qwen Assistant

    Let me look at the step_status.go file which should contain the logic for updating step status, including how services should be stopped.

  10. Qwen Assistant

    Now let me look at the agent's pipeline execution code to see how services are handled. The issue mentions that pods are destroyed properly, but the API still shows the service as running. Let me explore the pipeline directory in the main pipeline folder.

    <tool_call> <function=bash_terminal> <parameter=command> ls -la pipeline/ </tool_call>

  11. Qwen Assistant

    Let me explore the pipeline directory to find where services are handled.

  12. Qwen Assistant

    Let me explore the pipeline/backend directory, which likely contains the Kubernetes backend code that manages services.

    <tool_call> <function=bash_terminal> <parameter=command> ls -la pipeline/backend/ </tool_call>

  13. Qwen Assistant

    The output says "ERROR: No tool call was invoked" but it also shows the output. Let me try again with a simpler command.

  14. Qwen Assistant

    Let me explore the kubernetes backend directory to understand how services are cleaned up there.

  15. Qwen Assistant

    The kubernetes backend has files for managing pods, services, etc. Let me look at the main kubernetes.go file to understand how cleanup is handled when a pipeline completes.

  16. Qwen Assistant

    Now I see the key code. Looking at DestroyStep and DestroyWorkflow, they are the methods that destroy pods and other resources. Let me also check the agent code to see how steps report completion and whether service steps are properly handled.

    Let me look at the agent's pipeline runtime to understand how steps (including services) are tracked and reported.

    <tool_call> <function=bash_terminal> <parameter=command> ls -la agent/ </tool_call>

  17. Qwen Assistant

    The output was shown but the error message was confusing. Let me try running a command with proper structure.

  18. Qwen Assistant

    Let me look at the agent runner to understand how steps are run and completed:

  19. Qwen Assistant

    Now let me look at the pipeline runtime to understand how steps are executed, especially services:

    <tool_call> <function=bash_terminal> <parameter=command> ls -la pipeline/runtime/ </tool_call>

  20. Qwen Assistant

    Let me explore the pipeline/runtime directory to understand how the runtime executes steps including services.

  21. Qwen Assistant

    Let me look at the runtime.go file to understand how the runtime executes steps:

    <tool_call> <function=bash_terminal> <parameter=command> cat pipeline/runtime/runtime.go </tool_call>

  22. Qwen Assistant

    I see the issue - the bash_terminal tool is returning an error message saying "No tool call was invoked" but it's also showing the output. This is strange. Let me try running the command again.

  23. Qwen Assistant

    Let me look at the step.go file to see how steps are executed, especially services:

  24. Qwen Assistant

    Now I understand the flow better. Let me look at the workflow.go to see how stages and steps are executed, especially to understand when services are destroyed:

    <tool_call> <function=bash_terminal> <parameter=command> cat pipeline/runtime/workflow.go </tool_call>

  25. Qwen Assistant

    Let me look at the workflow.go to understand how stages and steps are executed:

  26. Qwen Assistant

    Now I understand the issue better. Looking at the flow:

    1. Run() in workflow.go iterates through all stages
    2. For each stage, runStage() is called which executes steps in parallel using errgroup
    3. After all stages complete, DestroyWorkflow() is called at the end of the defer
    4. DestroyStep() is called inside completeStep() which destroys individual steps (containers/pods)

    However, I notice that the issue is specifically about services (which are "detached" steps). Looking at runDetachedStep in step.go, it starts the step and returns immediately. The problem is that when a non-detached step (like wait-for-postgres) completes and returns, the workflow continues to the next stage, but service steps that are still running may not be properly reported as completed.

    Let me search for how service steps are handled differently, especially how they get marked as "running" vs "success" after being destroyed.

    Let me look at the server-side code that receives step status updates and updates the step state:

    <tool_call> <function=bash_terminal> <parameter=command> find server -type f -name "*.go" | xargs grep -l "service" | head -20 </tool_call>

  27. Qwen Assistant

    The command seems to have some issue. Let me try again with a simpler search command.

  28. Qwen Assistant

    Now I see that services can also be "detached" (container.Detached). Let me understand the flow better. Looking at the code:

    1. Services can be either detached or non-detached (blocking)
    2. For detached steps, runDetachedStep is called which returns immediately
    3. The problem is that after all main steps complete, the service pods are destroyed, but the step state might not be updated correctly.

    Let me look at the step type handling in step.go and understand how service steps are handled:

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "StepTypeService" pipeline/frontend/yaml/compiler/convert.go </tool_call>

  29. Qwen Assistant

    I'm having an issue with the tool calls. Let me check if the tool is actually working correctly by trying a simpler command.

  30. Qwen Assistant

    The tool is working. Let me see the context around line 95 in the convert.go file:

  31. Qwen Assistant

    This shows that services are marked as detached. Now let me understand the issue more. Looking at the code flow:

    1. In step.go, runDetachedStep is called for services
    2. The service step is started and logs are streamed in the background
    3. When the workflow context is done, DestroyWorkflow is called

    But the issue is that the service step's trace may not be updated with the final state after the pod is destroyed. Let me look at traceStep again and understand what happens:

    In traceStep, when processState != nil, it sets s.CurrStepState = *processState. This happens in completeStep after waiting for the step and destroying it.

    But wait - in runDetachedStep, the completeStep is called in a goroutine:

    go func() {
        defer r.uploadWait.Done()
        processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime)
        ...
        if traceErr := r.traceStep(processState, err, step); traceErr != nil {
            ...
        }
    }()
    

    The issue might be timing - when DestroyWorkflow is called, it might be called before the goroutine completes and traces the final state.

    Let me check if uploadWait.Wait() is called before DestroyWorkflow returns:

    Looking at workflow.go:

    for _, stage := range r.spec.Stages {
        ...
    }
    
    // Now we can shutdown the workflow
    destroyWorkflowFunc()
    
    // Ensure all logs/traces are uploaded before finishing
    logger.Debug().Msg("waiting for logs and traces upload")
    r.uploadWait.Wait()
    

    So DestroyWorkflow() is called first, then uploadWait.Wait() is called. This means the services are destroyed before the goroutines have finished tracing their final state!

    But wait, the issue says the pods ARE destroyed properly, but the API still shows the service as "running". Let me re-read the issue:

    "The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    So the issue is not about the state not being traced, but about the state not being updated to the server correctly. The server receives the state trace but doesn't update it properly.

    Let me look at how the server handles step status updates:

    <tool_call> <function=bash_terminal> <parameter=command> find server -type f -name "*.go" | xargs grep -l "UpdateStepStatus" </tool_call>

  32. Qwen Assistant

    Let me try again with the grep command to find files containing UpdateStepStatus.

  33. Qwen Assistant

    Let me look at the server's RPC handler to see how step status is received:

    <tool_call> <function=bash_terminal> <parameter=command> sed -n '200,230p' server/rpc/rpc.go </tool_call>

  34. Qwen Assistant

    The command is failing to produce output. Let me try a different approach.

  35. Qwen Assistant

    Now I found the issue! Looking at the Done method in server/rpc/rpc.go:

    if !model.IsThereRunningStage(currentPipeline.Workflows) {
        if currentPipeline, err = pipeline.UpdateStatusToDone(s.store, *currentPipeline, pipeline.PipelineStatus(currentPipeline.Workflows), workflow.Finished); err != nil {
            logger.Error().Err(err).Msgf("pipeline.UpdateStatusToDone: cannot update workflows final state")
        }
    }
    

    This only updates the final pipeline status when there's no running stage, but it doesn't ensure that all step statuses are properly updated before the workflow completes.

    The problem is:

    1. When the wait-for-postgres step completes, the step status is updated correctly
    2. The service step (postgres) is still running and hasn't reported completion yet
    3. The pipeline status is checked and updated
    4. Then DestroyWorkflow is called which destroys the service pods
    5. But the service step's final state is never reported because the goroutine hasn't finished tracing yet

    Looking at completeStep in pipeline/runtime/step.go:

    func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) {
        // Drain the log stream before waiting on the process exit.
        waitForLogs()
    
        waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck
        ...
        
        // Use runnerCtx here: the workflow context may already be canceled but we
        // still need to reach the backend to stop/remove the container.
        if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil {
            return nil, err
        }
        ...
        return waitState, nil
    }
    

    The issue is that DestroyStep is called BEFORE the final state is traced. After destroying the step, the code should still trace the final state, but there might be an issue with how the state is being reported.

    Wait, I see that after DestroyStep, the code continues to check errors and return the state. But then in runDetachedStep:

    go func() {
        defer r.uploadWait.Done()
        logger := r.makeLogger()
        processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime)
        logger.Debug().Str("step", step.Name).Msg("complete")
        
        if errors.Is(err, context.Canceled) {
            err = pipeline_errors.ErrCancel
        }
        if err != nil {
            logger.Error().Err(err).Str("step", step.Name).Msg("detached step failed while running")
        }
        
        if traceErr := r.traceStep(processState, err, step); traceErr != nil {
            logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result")
        }
    }()
    

    After completeStep returns, traceStep should be called. But the problem might be that when the pipeline workflow context is done, the traceStep might not be able to send the state to the server properly.

    Let me look more carefully at traceStep in step.go:

    func (r *Runtime) traceStep(processState *backend_types.State, err error, step *backend_types.Step) error {
        s := new(state.State)
        s.Workflow.Started = r.started
        s.CurrStep = step
        s.Workflow.Error = r.err.Get()
    
        switch {
        case processState == nil && err != nil:
            // Step failed to start — create an dummy exited process state.
            s.CurrStepState = backend_types.State{
                Error:     err,
                Exited:    true,
                OOMKilled: false,
            }
        case processState != nil:
            s.CurrStepState = *processState
            // processState == nil && err == nil: step just started, leave s.CurrStepState zero-valued.
        }
    
        if traceErr := r.tracer.Trace(s); traceErr != nil {
            logger := r.makeLogger()
            logger.Error().Err(traceErr).Msg("could not trace step state change")
        }
    
        return err
    }
    

    The tracer.Trace(s) is called. The tracer should be the agent's tracer which sends the state to the server via RPC. If this fails, the error is logged but not propagated.

    The issue might be that when the server's Update RPC handler receives the step update, something goes wrong with updating the service step state.

    Looking back at CalcStepStatus in server/pipeline/step_status.go:

    func CalcStepStatus(step model.Step, state rpc.StepState) (_ *model.Step, cancelPipelineFromStep bool, _ error) {
        switch step.State {
        case model.StatusPending:
            // Handle skip before anything else
            ...
        case model.StatusRunning:
            // Already running, check if it finished
            if state.Exited || state.Error != "" {
                step.Finished = state.Finished
                ...
                if state.ExitCode == 0 && state.Error == "" {
                    step.State = model.StatusSuccess
                } else {
                    step.State = model.StatusFailure
                    ...
                }
            }
        default:
            return nil, false, fmt.Errorf("step has state %s and does not expect rpc state updates", step.State)
        }
        ...
    }
    

    This looks correct - if the step is running and the state says it exited, it should update to success or failure. But wait, there's a check:

    if state.Exited || state.Error != "" {
    

    This means if state.Exited is false AND state.Error is empty, the step state won't be updated even if the container was destroyed!

    Now let me look at what state the agent sends. Looking at completeStep:

    bs := &backend_types.State{
        ExitCode:  int(cs.State.Terminated.ExitCode),
        Exited:    true,
        OOMKilled: false,
    }
    

    The state has Exited: true, so this should work...

    Wait, I need to re-read the issue more carefully. The issue says:

    "The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    So the pods ARE destroyed but the step state is still "running". This means the agent isn't reporting the final state, or the server isn't updating it correctly.

    Let me look at the WaitStep method in the kubernetes backend:

    func (e *kube) WaitStep(ctx context.Context, step *types.Step, taskUUID string) (*types.State, error) {
        podName, err := stepToPodName(step)
        ...
        
        select {
        case <-finished:
        case <-ctx.Done():
            return nil, ctx.Err()
        }
        
        pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{})
        if err != nil {
            return nil, err
        }
        
        // ... checks for image pull issues ...
        
        if len(pod.Status.ContainerStatuses) == 0 {
            return nil, fmt.Errorf("no container statuses found for pod %s", podName)
        }
        
        cs := pod.Status.ContainerStatuses[0]
        
        if cs.State.Terminated == nil {
            err := fmt.Errorf("no terminated state found for container %s/%s", podName, cs.Name)
            log.Error().Str("taskUUID", taskUUID).Str("pod", podName).Str("container", cs.Name).Interface("state", cs.State).Msg(err.Error())
            return nil, err
        }
        ...
    }
    

    This looks correct - it waits for the pod to terminate and returns the exit code.

    But wait, I think I see the issue now! Let me look at the DestroyStep method:

    func (e *kube) DestroyStep(ctx context.Context, step *types.Step, taskUUID string) error {
        var errs []error
        if needsRegistrySecret(step) {
            err := stopRegistrySecret(ctx, e, step, e.config.newDefaultDeleteOptions())
            if err != nil {
                errs = append(errs, err)
            }
        }
        if needsStepSecret(step) {
            err := stopStepSecret(ctx, e, step, e.config.newDefaultDeleteOptions())
            if err != nil {
                errs = append(errs, err)
            }
        }
        err := stopPod(ctx, e, step, e.config.newDefaultDeleteOptions())
        if err != nil {
            errs = append(errs, err)
        }
        return errors.Join(errs...)
    }
    

    This calls stopPod to destroy the pod. Let me check what stopPod does:

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "func stopPod" pipeline/backend/kubernetes/pod.go </tool_call>

  36. Qwen Assistant

    Let me search for the stopPod function:

  37. Qwen Assistant

    Let me look at the stopPod function and understand what's happening:

    <tool_call> <function=bash_terminal> <parameter=command> sed -n '736,780p' pipeline/backend/kubernetes/pod.go </tool_call>

  38. Qwen Assistant

    The tool is having issues. Let me try reading the file directly with cat.

  39. Qwen Assistant

    The stopPod function just deletes the pod. The issue might be a timing problem - if DestroyStep is called in completeStep, and DestroyStep calls stopPod, the pod is deleted. But what happens to the WaitStep that was called earlier?

    Looking at completeStep:

    waitForLogs()
    waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID)
    ...
    if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil {
        return nil, err
    }
    

    The order is:

    1. Wait for logs to drain
    2. Wait for the step to complete (WaitStep)
    3. Destroy the step (DestroyStep)

    So WaitStep should complete before DestroyStep is called. But wait - the issue says the pods ARE destroyed properly. This means DestroyStep is being called. But the state is not being updated.

    Let me check the kubernetes WaitStep function again more carefully. It waits for the pod to reach a terminal state:

    func (e *kube) WaitStep(ctx context.Context, step *types.Step, taskUUID string) (*types.State, error) {
        ...
        finished := make(chan struct{})
        var finishedOnce sync.Once
        
        podUpdated := func(_, newPod any) {
            pod, ok := newPod.(*kube_core_v1.Pod)
            if !ok {
                log.Error().Msgf("could not parse pod: %v", newPod)
                return
            }
            
            if pod.Name == podName {
                if isImagePullBackOffState(pod) || isInvalidImageName(pod) {
                    finishedOnce.Do(func() { close(finished) })
                }
                
                switch pod.Status.Phase {
                case kube_core_v1.PodSucceeded, kube_core_v1.PodFailed, kube_core_v1.PodUnknown:
                    finishedOnce.Do(func() { close(finished) })
                }
            }
        }
        ...
    }
    

    It waits for the pod to reach Succeeded, Failed, or Unknown phase. If the pod is deleted by DestroyStep before this wait completes, the informer might not see the final state.

    But wait - the issue says the wait-for-postgres step completes successfully, but the postgres service step shows as "running" with Stopped: 0. This means the service step's state was never updated to "success" or "failure".

    Let me re-read the issue output:

    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    

    The Stopped: 0 means the Finished time is 0, which means step.Finished was never set. Looking at CalcStepStatus:

    case model.StatusRunning:
        // Already running, check if it finished
        if state.Exited || state.Error != "" {
            step.Finished = state.Finished
            if step.Finished == 0 {
                step.Finished = time.Now().Unix()
            }
            ...
        }
    

    So if state.Exited is false and state.Error is empty, the step state is never updated. This means the agent is not sending the correct state.

    Looking at completeStep in step.go:

    waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID)
    

    If WaitStep fails (e.g., returns an error because the pod was deleted), the function returns early without calling DestroyStep. But the issue says the pods ARE destroyed, so DestroyStep is being called.

    Wait - let me look at the order again:

    func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) {
        // Drain the log stream before waiting on the process exit.
        waitForLogs()
    
        waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck
        if err != nil {
            if errors.Is(err, context.Canceled) {
                if waitState == nil {
                    waitState = &backend_types.State{}
                }
                waitState.Error = pipeline_errors.ErrCancel
            } else {
                return nil, err
            }
        }
    
        // Use runnerCtx here: the workflow context may already be canceled but we
        // still need to reach the backend to stop/remove the container.
        if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil {
            return nil, err
        }
        ...
    }
    

    If WaitStep returns context.Canceled, the code sets waitState.Error but doesn't set waitState.Exited. Then DestroyStep is called. After that:

    waitState.Started = startTime
    
    // Re-check context cancellation: the wait may have raced with cancellation.
    if ctxErr := r.ctx.Err(); ctxErr != nil && errors.Is(ctxErr, context.Canceled) {
        waitState.Error = pipeline_errors.ErrCancel
    }
    

    The waitState.Exited is not set in this case!

    But wait - if the pod was destroyed by DestroyStep, WaitStep should fail with some error, not necessarily context.Canceled.

    Let me check what WaitStep returns when the pod is already deleted:

    pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{})
    if err != nil {
        return nil, err
    }
    

    If the pod is deleted, Get will return a "not found" error. But the code checks for this case?

    No, it doesn't check for "not found" error. So WaitStep will return an error like "pod not found", and completeStep will return that error without setting waitState.Exited.

    Wait, but the code flow is:

    1. WaitStep is called - this waits for the pod to reach a terminal state
    2. If the context is canceled, WaitStep returns with context.Canceled
    3. DestroyStep is called - this deletes the pod

    But if the pod is deleted BEFORE WaitStep completes (e.g., by some other mechanism), then WaitStep will return an error.

    Actually, I think I finally understand the issue! Let me trace through the execution:

    1. The postgres service step is started
    2. The wait-for-postgres step completes
    3. The wait-for-postgres step's goroutine in runDetachedStep finishes
    4. The main loop continues to the next stage (there's no next stage in this case)
    5. After all stages complete, DestroyWorkflow is called
    6. DestroyWorkflow calls DestroyStep for ALL steps, including the service

    But the problem is: DestroyStep is called BEFORE the service step's goroutine has finished waiting for the pod and reporting the final state!

    Looking at workflow.go:

    for _, stage := range r.spec.Stages {
        ...
    }
    
    // Now we can shutdown the workflow
    destroyWorkflowFunc()
    
    // Ensure all logs/traces are uploaded before finishing
    logger.Debug().Msg("waiting for logs and traces upload")
    r.uploadWait.Wait()
    

    So DestroyWorkflow() is called BEFORE uploadWait.Wait(). This means the service pods are destroyed BEFORE the service step's goroutine has a chance to:

    1. Wait for the pod to complete
    2. Destroy the step (which is already done by DestroyWorkflow)
    3. Trace the final state

    But wait - DestroyWorkflow calls DestroyStep for all steps. So when the service step's goroutine eventually calls DestroyStep, the pod is already gone. And when WaitStep is called, it might get an error because the pod is already deleted.

    Let me check the completeStep function again:

    waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID)
    if err != nil {
        if errors.Is(err, context.Canceled) {
            if waitState == nil {
                waitState = &backend_types.State{}
            }
            waitState.Error = pipeline_errors.ErrCancel
        } else {
            return nil, err  // <- Returns without setting Exited!
        }
    }
    

    If WaitStep returns an error that's NOT context.Canceled (e.g., "pod not found"), the function returns immediately without setting waitState.Exited. Then the error is traced, but the state doesn't have Exited: true, so the server doesn't update the step state.

    Actually wait, let me re-read completeStep:

    func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) {
        // Drain the log stream before waiting on the process exit.
        waitForLogs()
    
        waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck
        if err != nil {
            if errors.Is(err, context.Canceled) {
                if waitState == nil {
                    waitState = &backend_types.State{}
                }
                waitState.Error = pipeline_errors.ErrCancel
            } else {
                return nil, err  // Returns error and nil state
            }
        }
    
        // Use runnerCtx here: the workflow context may already be canceled but we
        // still need to reach the backend to stop/remove the container.
        if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil {
            return nil, err
        }
        ...
    }
    

    If WaitStep returns an error (not context.Canceled), the function returns nil, err. This means processState in runDetachedStep is nil, and traceStep is called with processState == nil.

    Looking at traceStep:

    switch {
    case processState == nil && err != nil:
        // Step failed to start — create an dummy exited process state.
        s.CurrStepState = backend_types.State{
            Error:     err,
            Exited:    true,
            OOMKilled: false,
        }
    case processState != nil:
        s.CurrStepState = *processState
        ...
    }
    

    If processState == nil and err != nil, a dummy state with Exited: true is created. So the server should receive Exited: true even if WaitStep fails.

    But the issue says the pods ARE destroyed. This means DestroyStep is being called, which means WaitStep must have succeeded. So the flow is:

    1. WaitStep succeeds (pod reaches terminal state)
    2. DestroyStep is called (pod is deleted)
    3. The state is traced with Exited: true

    But the issue says the server still shows the service as "running". This means either:

    1. The server didn't receive the state update
    2. The server received it but didn't update the step state

    Let me check the server's Update method again. Looking at the code around line 212:

    if err := pipeline.UpdateStepStatus(c, s.store, step, state); err != nil {
        log.Error().Err(err).Msg("rpc.update: cannot update step")
    }
    

    If UpdateStepStatus fails, the error is logged. But maybe the error is not being logged or is being swallowed?

    Actually, I just realized something. Let me look at the DestroyWorkflow method in kubernetes.go:

    func (e *kube) DestroyWorkflow(ctx context.Context, conf *types.Config, taskUUID string) error {
        log.Trace().Str("taskUUID", taskUUID).Msg("deleting Kubernetes primitives")
    
        for _, stage := range conf.Stages {
            for _, step := range stage.Steps {
                err := stopPod(ctx, e, step, e.config.newDefaultDeleteOptions())
                if err != nil {
                    return err
                }
            }
        }
        ...
    }
    

    This iterates over ALL steps and destroys them, including services. This is called in DestroyWorkflow which is called BEFORE uploadWait.Wait() in workflow.go.

    So the problem is:

    1. The main pipeline steps (like wait-for-postgres) complete
    2. DestroyWorkflow is called, which destroys ALL pods, including the service pods
    3. The service step's goroutine (running runDetachedStep) is still waiting for the pod to complete
    4. When WaitStep is called, the pod is already deleted, so it returns an error
    5. The error is returned from completeStep, but it's not context.Canceled, so processState is nil
    6. traceStep is called with processState == nil and err != nil, which creates a dummy state with Exited: true
    7. But wait - this should still result in the state being updated!

    Actually, let me re-read the flow more carefully. In runDetachedStep:

    func (r *Runtime) runDetachedStep(runnerCtx context.Context, step *backend_types.Step) error {
        waitForLogs, startTime, err := r.startStep(step)
        if err != nil {
            return r.traceStep(nil, err, step)
        }
    
        // Container is up and logging is streaming — hand off to background.
        r.uploadWait.Add(1)
        go func() {
            defer r.uploadWait.Done()
    
            logger := r.makeLogger()
    
            processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime)
            logger.Debug().Str("step", step.Name).Msg("complete")
    
            if errors.Is(err, context.Canceled) {
                err = pipeline_errors.ErrCancel
            }
            if err != nil {
                logger.Error().Err(err).Str("step", step.Name).Msg("detached step failed while running")
            }
    
            if traceErr := r.traceStep(processState, err, step); traceErr != nil {
                logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result")
            }
        }()
    
        return nil
    }
    

    The goroutine runs completeStep, then calls traceStep. But if DestroyStep is called by DestroyWorkflow before the goroutine finishes, WaitStep will fail.

    But the goroutine should still call traceStep with the error state. Let me check what happens in traceStep when processState == nil and err != nil:

    case processState == nil && err != nil:
        // Step failed to start — create an dummy exited process state.
        s.CurrStepState = backend_types.State{
            Error:     err,
            Exited:    true,
            OOMKilled: false,
        }
    

    Wait, this says "Step failed to start" but it's also used when the step fails during execution. This might be the issue - the error message is misleading, but the code should still create a state with Exited: true.

    Actually, I think I found the issue! Let me look at completeStep again:

    func (r *Runtime) completeStep(runnerCtx context.Context, step *backend_types.Step, waitForLogs func(), startTime int64) (*backend_types.State, error) {
        waitForLogs()
    
        waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck
        if err != nil {
            if errors.Is(err, context.Canceled) {
                if waitState == nil {
                    waitState = &backend_types.State{}
                }
                waitState.Error = pipeline_errors.ErrCancel
            } else {
                return nil, err
            }
        }
    
        if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil {
            return nil, err
        }
    
        waitState.Started = startTime
    
        if ctxErr := r.ctx.Err(); ctxErr != nil && errors.Is(ctxErr, context.Canceled) {
            waitState.Error = pipeline_errors.ErrCancel
        }
        ...
    }
    

    If WaitStep succeeds, waitState is not nil and has Exited: true. Then DestroyStep is called. After DestroyStep, the code continues to set waitState.Started and check for context cancellation.

    But what if WaitStep succeeds with waitState.Exited = true, but then DestroyStep is called by DestroyWorkflow before the goroutine finishes? The DestroyStep call in completeStep might fail because the pod is already deleted, but the error is returned without updating the state.

    Actually, I just realized the order of operations in completeStep:

    1. WaitStep is called - waits for pod to complete
    2. DestroyStep is called - destroys the pod

    But if DestroyWorkflow is called before step 1 completes, the pod will be deleted by DestroyWorkflow. Then when WaitStep is called in step 1, it will get an error (pod not found), and completeStep will return an error.

    Wait, but the issue says the pods ARE destroyed properly. This implies that DestroyWorkflow is being called. But if the pods are destroyed by DestroyWorkflow, then WaitStep should fail.

    Let me re-read the issue more carefully:

    "The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    So the pods ARE destroyed, but the API shows the service as "running". This means:

    1. The pod was destroyed (either by DestroyStep or DestroyWorkflow)
    2. But the step state was not updated to "success" or "failure"

    This could happen if:

    1. WaitStep is called but returns an error (pod not found)
    2. The error is returned from completeStep
    3. traceStep is called with processState == nil and err != nil
    4. A dummy state with Exited: true is created
    5. The server receives the state update
    6. But the server doesn't update the step state correctly

    Actually, wait. Let me check the server's UpdateStepStatus function again:

    func UpdateStepStatus(ctx context.Context, store store.Store, step *model.Step, state rpc.StepState) error {
        updatedStep, shouldCancelPipelineFromStep, err := CalcStepStatus(*step, state)
        if err != nil {
            return err
        }
        *step = *updatedStep
    
        if shouldCancelPipelineFromStep {
            if err := cancelPipelineFromStep(ctx, store, step); err != nil {
                return err
            }
        }
        return store.StepUpdate(step)
    }
    

    And CalcStepStatus:

    func CalcStepStatus(step model.Step, state rpc.StepState) (_ *model.Step, cancelPipelineFromStep bool, _ error) {
        switch step.State {
        case model.StatusPending:
            ...
        case model.StatusRunning:
            if state.Exited || state.Error != "" {
                step.Finished = state.Finished
                if step.Finished == 0 {
                    step.Finished = time.Now().Unix()
                }
                step.ExitCode = state.ExitCode
                step.Error = state.Error
    
                if state.ExitCode == 0 && state.Error == "" {
                    step.State = model.StatusSuccess
                } else {
                    step.State = model.StatusFailure
                    ...
                }
            }
        default:
            return nil, false, fmt.Errorf("step has state %s and does not expect rpc state updates", step.State)
        }
        ...
    }
    

    The key check is: if state.Exited || state.Error != "". If both are false, the step state is not updated.

    So if the agent sends a state with Exited: false and Error: "", the server won't update the step state. This would happen if WaitStep returns without an error but also doesn't set Exited: true.

    Let me check what WaitStep returns when the pod is in a terminal state:

    cs := pod.Status.ContainerStatuses[0]
    
    if cs.State.Terminated == nil {
        err := fmt.Errorf("no terminated state found for container %s/%s", podName, cs.Name)
        log.Error().Str("taskUUID", taskUUID).Str("pod", podName).Str("container", cs.Name).Interface("state", cs.State).Msg(err.Error())
        return nil, err
    }
    
    bs := &types.State{
        ExitCode:  int(cs.State.Terminated.ExitCode),
        Exited:    true,
        OOMKilled: false,
    }
    
    return bs, nil
    

    This looks correct - if the container is terminated, Exited: true is set.

    So the issue must be that the agent is sending a state with Exited: false and Error: "". Let me check what state the agent sends in traceStep:

    case processState == nil && err != nil:
        s.CurrStepState = backend_types.State{
            Error:     err,
            Exited:    true,
            OOMKilled: false,
        }
    case processState != nil:
        s.CurrStepState = *processState
        ...
    

    If processState != nil and processState.Exited == true, the agent should send a state with Exited: true. But what if processState.Exited is not set?

    Let me check the flow again. In completeStep:

    waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID)
    if err != nil {
        if errors.Is(err, context.Canceled) {
            if waitState == nil {
                waitState = &backend_types.State{}
            }
            waitState.Error = pipeline_errors.ErrCancel
        } else {
            return nil, err
        }
    }
    
    if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil {
        return nil, err
    }
    
    waitState.Started = startTime
    
    if ctxErr := r.ctx.Err(); ctxErr != nil && errors.Is(ctxErr, context.Canceled) {
        waitState.Error = pipeline_errors.ErrCancel
    }
    

    If WaitStep returns context.Canceled with waitState == nil, a new state is created but Exited is not set! Then DestroyStep is called, and the function returns waitState (which has Exited: false).

    But wait, waitState is created as &backend_types.State{}, which means Exited is the zero value false. Then waitState.Error is set, but waitState.Exited is never set to true.

    Then in traceStep:

    case processState != nil:
        s.CurrStepState = *processState
    

    The state is copied as-is, with Exited: false. The server receives this and sees Exited: false and Error: pipeline_errors.ErrCancel, so the check if state.Exited || state.Error != "" is true, and the step state should be updated.

    Hmm, but the issue says the service is shown as "running" with Stopped: 0. This means step.Finished is 0, which means the state was never updated.

    Wait, let me re-read the issue output:

    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    

    The Started time is set, but Stopped is 0. This means the step started but never finished. And the state is "running".

    This could happen if:

    1. The agent sent a state with Started time set
    2. But never sent a state with Exited: true

    Or:

    1. The agent sent a state with Exited: true and Finished time
    2. But the server didn't update the step correctly

    Let me check what state the agent sends when the step starts:

    In step.go, executeStep:

    if err := r.traceStep(nil, nil, step); err != nil {
        return err
    }
    

    This calls traceStep with processState == nil and err == nil. Looking at traceStep:

    switch {
    case processState == nil && err != nil:
        ...
    case processState != nil:
        s.CurrStepState = *processState
        // processState == nil && err == nil: step just started, leave s.CurrStepState zero-valued.
    }
    

    When processState == nil and err == nil, s.CurrStepState is left as the zero value. But s.CurrStep is set to the step.

    So the tracer sends:

    • s.Workflow.Started = r.started
    • s.CurrStep = step (which has Started and Finished times)
    • s.CurrStepState is zero-valued

    The server receives this and updates the step's Started time and sets the state to "running".

    But the issue is that the step's Finished time is never updated. This means the agent never sends a state with Exited: true.

    Wait, I think I understand the issue now! The problem is that DestroyStep is called BEFORE WaitStep completes in the goroutine. So the flow is:

    1. Main loop finishes all stages
    2. DestroyWorkflow is called, which destroys ALL pods (including services)
    3. The service step's goroutine is still waiting for WaitStep to complete
    4. WaitStep returns an error because the pod was deleted
    5. completeStep returns with an error
    6. traceStep is called with processState == nil and err != nil
    7. A dummy state with Exited: true is created and sent to the server

    But wait, if the dummy state is sent with Exited: true, the server should update the step state. Let me check the server's UpdateStepStatus function again:

    func UpdateStepStatus(ctx context.Context, store store.Store, step *model.Step, state rpc.StepState) error {
        updatedStep, shouldCancelPipelineFromStep, err := CalcStepStatus(*step, state)
        if err != nil {
            return err
        }
        ...
    }
    

    And CalcStepStatus:

    case model.StatusRunning:
        if state.Exited || state.Error != "" {
            step.Finished = state.Finished
            ...
        }
    

    If state.Error != "", the step state should be updated. But the issue says the step is still "running".

    Hmm, maybe the issue is that the server doesn't receive the state update at all. Let me check the Done method in rpc.go:

    func (s *RPC) Done(c context.Context, strWorkflowID string, state rpc.WorkflowState) error {
        ...
        
        // Check if all steps are done
        if !model.IsThereRunningStage(currentPipeline.Workflows) {
            if currentPipeline, err = pipeline.UpdateStatusToDone(s.store, *currentPipeline, pipeline.PipelineStatus(currentPipeline.Workflows), workflow.Finished); err != nil {
                logger.Error().Err(err).Msgf("pipeline.UpdateStatusToDone: cannot update workflows final state")
            }
        }
        ...
    }
    

    The Done method is called when the workflow is complete. At this point, it checks if all stages are done. But if a service step is still "running", the condition !model.IsThereRunningStage(currentPipeline.Workflows) will be false, and the pipeline won't be marked as done.

    Let me check IsThereRunningStage:

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "IsThereRunningStage" server/model/ </tool_call>

  40. Qwen Assistant

    The command failed to produce output. Let me try again with grep.

  41. Qwen Assistant

    Let me look at this function:

    <tool_call> <function=bash_terminal> <parameter=command> sed -n '50,65p' server/model/workflow.go </tool_call>

  42. Qwen Assistant

    The command is failing to produce output. Let me try a different approach - using cat to read the file.

  43. Qwen Assistant

    Now I understand the issue better. IsThereRunningStage checks if any workflow (stage) is running or pending. But the issue is about steps, not workflows.

    Let me trace through the flow again:

    1. The wait-for-postgres step completes successfully
    2. The main loop in workflow.go continues
    3. After all stages complete, DestroyWorkflow is called
    4. DestroyWorkflow destroys ALL pods, including services
    5. The service step's goroutine is still waiting for WaitStep to complete
    6. WaitStep returns an error because the pod was deleted
    7. The error is returned from completeStep
    8. traceStep is called with processState == nil and err != nil
    9. A dummy state with Exited: true is created and sent to the server
    10. The server updates the step state to "failure" (because there's an error)

    But wait - the issue says the service is shown as "running". This means the server didn't receive the state update, or the update was ignored.

    Actually, I think I finally understand the issue! Let me check the order of operations in runDetachedStep more carefully:

    go func() {
        defer r.uploadWait.Done()
        processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime)
        ...
        if traceErr := r.traceStep(processState, err, step); traceErr != nil {
            logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result")
        }
    }()
    

    The goroutine calls traceStep, which calls r.tracer.Trace(s). The tracer sends the state to the server via RPC. But what if the server closes the RPC connection before the tracer finishes sending the state?

    Actually, I think the issue might be with how DestroyWorkflow is called. Looking at workflow.go:

    for _, stage := range r.spec.Stages {
        stageChan := r.runStage(runnerCtx, stage.Steps)
        select {
        case <-r.ctx.Done():
            <-stageChan
            return pipeline_errors.ErrCancel
        case err := <-stageChan:
            if err != nil {
                r.err.Set(err)
            }
        }
    }
    
    // Now we can shutdown the workflow
    destroyWorkflowFunc()
    

    The destroyWorkflowFunc() is called AFTER all stages complete. But at this point, the service step's goroutine might still be running. The goroutine will call traceStep eventually, but by the time it does, the server might have already marked the pipeline as done.

    Actually, I think I found the real issue! Let me look at the traceStep function again:

    func (r *Runtime) traceStep(processState *backend_types.State, err error, step *backend_types.Step) error {
        s := new(state.State)
        s.Workflow.Started = r.started
        s.CurrStep = step
        s.Workflow.Error = r.err.Get()
    
        switch {
        case processState == nil && err != nil:
            s.CurrStepState = backend_types.State{
                Error:     err,
                Exited:    true,
                OOMKilled: false,
            }
        case processState != nil:
            s.CurrStepState = *processState
            ...
        }
    
        if traceErr := r.tracer.Trace(s); traceErr != nil {
            logger := r.makeLogger()
            logger.Error().Err(traceErr).Msg("could not trace step state change")
        }
    
        return err
    }
    

    The tracer is r.tracer, which is set via WithTracer option. Let me check what tracer is used:

    Looking at agent/runner.go:

    }).Run(runnerCtx)
    

    And pipeline_runtime.WithTracer(r.createTracer(ctxMeta, logger, workflow)).

    The tracer sends the state to the server via RPC. But what if the RPC connection is closed before the tracer finishes sending the state?

    Actually, I think the issue might be simpler. Let me check what happens when WaitStep is called after the pod has been deleted. The WaitStep function uses an informer to watch for pod updates:

    func (e *kube) WaitStep(ctx context.Context, step *types.Step, taskUUID string) (*types.State, error) {
        podName, err := stepToPodName(step)
        ...
        
        finished := make(chan struct{})
        var finishedOnce sync.Once
        
        podUpdated := func(_, newPod any) {
            pod, ok := newPod.(*kube_core_v1.Pod)
            ...
            switch pod.Status.Phase {
            case kube_core_v1.PodSucceeded, kube_core_v1.PodFailed, kube_core_v1.PodUnknown:
                finishedOnce.Do(func() { close(finished) })
            }
        }
        ...
        
        select {
        case <-finished:
        case <-ctx.Done():
            return nil, ctx.Err()
        }
        ...
    }
    

    The informer is watching for pod updates. If the pod is deleted by DestroyWorkflow, the informer might receive a "delete" event. But the podUpdated function only handles "update" events. So if the pod is deleted, the finished channel will never be closed, and WaitStep will wait until the context is canceled.

    But the context for WaitStep is r.ctx, which is the workflow context. The workflow context is canceled when the workflow is done. So WaitStep will return with context.Canceled.

    Looking at completeStep:

    waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID)
    if err != nil {
        if errors.Is(err, context.Canceled) {
            if waitState == nil {
                waitState = &backend_types.State{}
            }
            waitState.Error = pipeline_errors.ErrCancel
        } else {
            return nil, err
        }
    }
    

    If WaitStep returns context.Canceled, a new state is created with waitState.Error = pipeline_errors.ErrCancel. But waitState.Exited is not set! It's the zero value false.

    Then DestroyStep is called. The pod might already be deleted by DestroyWorkflow, but the stopPod function handles "not found" errors:

    err = engine.client.CoreV1().Pods(engine.config.GetNamespace(step.OrgID)).Delete(ctx, podName, deleteOpts)
    if errors.IsNotFound(err) {
        return nil
    }
    return err
    

    So DestroyStep returns nil even if the pod is already deleted.

    Then waitState.Started = startTime is set.

    Then:

    if ctxErr := r.ctx.Err(); ctxErr != nil && errors.Is(ctxErr, context.Canceled) {
        waitState.Error = pipeline_errors.ErrCancel
    }
    

    The waitState.Error is set to pipeline_errors.ErrCancel.

    Then the function returns waitState, nil.

    So the state returned has:

    • waitState.Started = startTime
    • waitState.Error = pipeline_errors.ErrCancel
    • waitState.Exited = false (zero value)

    Then in traceStep:

    case processState != nil:
        s.CurrStepState = *processState
    

    The state is copied as-is, with Exited: false and Error: pipeline_errors.ErrCancel.

    The server receives this state and checks:

    if state.Exited || state.Error != "" {
        step.Finished = state.Finished
        ...
    }
    

    Since state.Error != "", the step state should be updated. But the issue says the step is still "running".

    Wait, I just realized something. The WaitStep function might not return context.Canceled. It might return an error from the informer or the Get call.

    Let me check the WaitStep function again. After the select statement:

    select {
    case <-finished:
    case <-ctx.Done():
        return nil, ctx.Err()
    }
    
    pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{})
    

    If the finished channel is closed (pod reached terminal state), the code continues to get the pod. If the pod is deleted after the informer sees the terminal state but before the Get call, the Get will return an error.

    But the informer only sees update events, not delete events. So if the pod is deleted, the informer won't see it, and the finished channel won't be closed. WaitStep will wait until the context is canceled.

    Hmm, but the issue says the pods ARE destroyed properly. This means DestroyWorkflow is being called. But the service step's goroutine might not have finished yet.

    Actually, I think the issue might be a race condition. Let me trace through the flow more carefully:

    1. The wait-for-postgres step completes successfully
    2. The main loop in workflow.go continues to the next iteration
    3. After all stages complete, destroyWorkflowFunc() is called
    4. destroyWorkflowFunc() calls e.engine.DestroyWorkflow(ctx, r.spec, r.taskUUID)
    5. DestroyWorkflow iterates over all steps and calls stopPod for each
    6. The service step's goroutine is still waiting in WaitStep
    7. WaitStep is waiting for the informer to see a terminal state
    8. The pod is deleted by DestroyWorkflow
    9. The informer might see the delete event, but the podUpdated function only handles update events
    10. The finished channel is never closed
    11. WaitStep waits until the context is canceled
    12. The context is canceled (by destroyWorkflowFunc() or by the workflow completion)
    13. WaitStep returns context.Canceled
    14. completeStep creates a new state with Error = pipeline_errors.ErrCancel
    15. traceStep is called with the state
    16. The server receives the state and updates the step

    But wait - if WaitStep returns context.Canceled, the context is canceled. But the goroutine is still running. The traceStep should be called after completeStep returns.

    Actually, I think I might be overcomplicating this. Let me re-read the issue:

    "The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    The pods ARE destroyed, but the API shows the service as "running". This means:

    1. The pod was destroyed (confirmed)
    2. The step state is "running" (the issue)

    This could happen if:

    1. The agent never sent a state update after the pod was destroyed
    2. The agent sent a state update, but the server didn't process it
    3. The agent sent a state update, but the server updated the wrong step

    Let me check if there's any issue with how the step UUID is matched. In the server's Update method:

    step, err := s.store.StepByUUID(state.StepUUID)
    if err != nil {
        log.Error().Err(err).Msgf("cannot find step with uuid %s", state.StepUUID)
        return err
    }
    

    If the step is not found, the error is logged and returned. But what if the step was already updated by a previous trace?

    Actually, I think the issue might be with the order of operations. Let me check the flow again:

    1. The wait-for-postgres step completes
    2. The main loop finishes
    3. destroyWorkflowFunc() is called
    4. DestroyWorkflow destroys all pods
    5. The service step's goroutine calls WaitStep
    6. WaitStep returns context.Canceled because the pod was deleted
    7. completeStep creates a state with Error = pipeline_errors.ErrCancel
    8. traceStep is called with the state
    9. The server receives the state and updates the step to "failure"

    But the issue says the step is "running", not "failure". This means the server didn't receive the state update, or the update was ignored.

    Wait, I just realized something. The Done method in rpc.go is called when the workflow is complete:

    func (s *RPC) Done(c context.Context, strWorkflowID string, state rpc.WorkflowState) error {
        ...
        if !model.IsThereRunningStage(currentPipeline.Workflows) {
            if currentPipeline, err = pipeline.UpdateStatusToDone(s.store, *currentPipeline, pipeline.PipelineStatus(currentPipeline.Workflows), workflow.Finished); err != nil {
                logger.Error().Err(err).Msgf("pipeline.UpdateStatusToDone: cannot update workflows final state")
            }
        }
        ...
    }
    

    The Done method is called by the agent after all steps complete. At this point, the service step's goroutine might not have finished yet. The Done method checks if all stages are done, but the service step is still "running".

    Actually, IsThereRunningStage checks workflows (stages), not steps. So even if a step is "running", the workflow might be "success" or "failure".

    Wait, but the issue says the service step is "running" with Stopped: 0. This means the step's Finished time is 0, which means the step was never updated to "success" or "failure".

    Let me think about this differently. The issue might be that the service step's goroutine is blocked in WaitStep when DestroyWorkflow is called. The DestroyWorkflow function destroys the pod, but the goroutine is still waiting. The goroutine will eventually return from WaitStep (when the context is canceled), but by that time, the pipeline might have already been marked as done.

    Actually, I think I need to look at the actual timing. Let me check when Done is called:

    In agent/runner.go:

    err = pipeline_runtime.New(...).Run(runnerCtx)
    
    state.Finished = time.Now().Unix()
    ...
    if err := r.client.Done(doneCtx, workflow.ID, state); err != nil {
        logger.Error().Err(err).Msg("failed to update workflow status")
    }
    

    The Done method is called after pipeline_runtime.New(...).Run(runnerCtx) returns. The Run method in workflow.go is:

    func (r *Runtime) Run(runnerCtx context.Context) error {
        ...
        defer destroyWorkflowFunc()
    
        if err := r.engine.SetupWorkflow(r.ctx, r.spec, r.taskUUID); err != nil {
            ...
            return err
        }
    
        for _, stage := range r.spec.Stages {
            ...
        }
    
        destroyWorkflowFunc()
        
        r.uploadWait.Wait()
        
        return r.err.Get()
    }
    

    So DestroyWorkflow is called inside Run, then uploadWait.Wait() is called, then Run returns. After Run returns, the agent calls Done.

    The uploadWait.Wait() waits for all goroutines to finish (including the service step's goroutine). So the service step's goroutine should finish before Run returns.

    But wait - what if the service step's goroutine is blocked indefinitely? The WaitStep function waits for the pod to reach a terminal state or the context to be canceled. If the context is not canceled, WaitStep will wait forever.

    But the context for WaitStep is r.ctx, which is set in New:

    r.ctx = context.Background()
    

    And WithContext option can override this:

    pipeline_runtime.WithContext(workflowCtx)
    

    The workflowCtx is created in agent/runner.go:

    workflowCtx, _ := context.WithTimeout(ctxMeta, timeout)
    workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx)
    defer cancelWorkflowCtx(nil)
    

    The workflowCtx is canceled when the workflow completes or times out. So WaitStep should return when the context is canceled.

    But what if the context is canceled before WaitStep starts? Then WaitStep will return immediately with context.Canceled.

    Actually, I think I understand the issue now. The problem is that DestroyWorkflow is called INSIDE Run, before uploadWait.Wait(). So the flow is:

    1. All stages complete
    2. destroyWorkflowFunc() is called (inside Run)
    3. DestroyWorkflow destroys all pods
    4. The service step's goroutine is waiting in WaitStep
    5. The goroutine continues after WaitStep returns
    6. DestroyStep is called (but the pod is already deleted)
    7. The error is handled
    8. traceStep is called
    9. uploadWait.Done() is called
    10. uploadWait.Wait() returns
    11. Run returns
    12. The agent calls Done

    But the issue is that step 6 might fail, and the error might be returned from completeStep. Then traceStep is called with the error state.

    Actually, looking at completeStep again:

    if err := r.engine.DestroyStep(runnerCtx, step, r.taskUUID); err != nil {
        return nil, err
    }
    

    If DestroyStep fails, the error is returned. But waitState is already set from WaitStep. So the state is returned with the error from DestroyStep.

    Wait, but if DestroyStep fails, the function returns nil, err, not waitState, err. So the state is not returned.

    Let me re-read the issue more carefully. The issue says:

    "The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    So the pods ARE destroyed. This means DestroyWorkflow is being called. But the step state is not updated.

    I think the issue might be that the service step's goroutine is not finishing before Done is called. The uploadWait.Wait() should wait for all goroutines to finish, but what if the goroutine is blocked somewhere?

    Actually, let me look at the startStep function:

    func (r *Runtime) startStep(step *backend_types.Step) (func(), int64, error) {
        if err := r.engine.StartStep(r.ctx, step, r.taskUUID); err != nil {
            return nil, 0, err
        }
        startTime := time.Now().Unix()
    
        rc, err := r.engine.TailStep(r.ctx, step, r.taskUUID)
        if err != nil {
            return nil, 0, err
        }
    
        var wg sync.WaitGroup
        wg.Go(func() {
            logger := r.makeLogger()
            if err := r.logger(step, rc); err != nil {
                logger.Error().Err(err).Str("step", step.Name).Msg("step log streaming failed")
            }
            _ = rc.Close()
        })
    
        return wg.Wait, startTime, nil
    }
    

    The waitForLogs function is wg.Wait, which waits for the log streaming goroutine to finish. In completeStep:

    waitForLogs()
    waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID)
    

    waitForLogs waits for the log streaming to finish. But the log streaming might be blocked if the pod is deleted before the logs are fully read.

    Actually, I think the issue might be with the log streaming. Let me check the TailStep function in the kubernetes backend:

    func (e *kube) TailStep(ctx context.Context, step *types.Step, taskUUID string) (io.ReadCloser, error) {
        ...
        select {
        case <-up:
        case <-ctx.Done():
            return nil, ctx.Err()
        }
        ...
    }
    

    The TailStep function waits for the pod to be in a terminal state before returning the log stream. If the pod is deleted before it reaches a terminal state, the informer might not see the update, and the up channel will never be closed.

    But the informer should see the "delete" event when the pod is deleted. Let me check the informer setup:

    podUpdated := func(_, newPod any) {
        pod, ok := newPod.(*kube_core_v1.Pod)
        ...
        switch pod.Status.Phase {
        case kube_core_v1.PodRunning, kube_core_v1.PodSucceeded, kube_core_v1.PodFailed:
            upOnce.Do(func() { close(up) })
        }
    }
    
    if _, err := si.Core().V1().Pods().Informer().AddEventHandler(
        cache.ResourceEventHandlerFuncs{
            UpdateFunc: podUpdated,
        },
    ); err != nil {
        return nil, err
    }
    

    The informer only has an UpdateFunc, not a DeleteFunc. So when the pod is deleted, the informer won't trigger the podUpdated function. The up channel will never be closed, and TailStep will wait until the context is canceled.

    This could be the issue! The TailStep is waiting for the pod to reach a terminal state, but if the pod is deleted before that, the up channel is never closed. The TailStep returns with context.Canceled, and the error is returned from startStep.

    Then in runDetachedStep:

    waitForLogs, startTime, err := r.startStep(step)
    if err != nil {
        return r.traceStep(nil, err, step)
    }
    

    If startStep fails, traceStep is called with processState == nil and err != nil. A dummy state with Exited: true is created and sent to the server.

    But wait - this means the step state should be updated to "failure" (because there's an error). But the issue says the step is "running".

    Actually, I think the issue might be different. Let me re-read the issue again:

    "With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    The key phrase is "pods from kubernetes backend are destroyed properly". This means the pods ARE destroyed. But the step state is still "running".

    This suggests that:

    1. The pod was destroyed (by DestroyWorkflow)
    2. The step state was not updated to "success" or "failure"

    If the step state is still "running", it means:

    1. The agent never sent a state update after the pod was destroyed
    2. Or the agent sent a state update, but the server didn't process it

    I think the issue might be that the agent is blocked in WaitStep or TailStep when DestroyWorkflow is called. The DestroyWorkflow destroys the pod, but the agent is still waiting. The agent will eventually unblock (when the context is canceled), but by that time, the state update might not be processed correctly.

    Actually, let me look at the informer setup more carefully. The informer is created with:

    si := informers.NewSharedInformerFactoryWithOptions(e.client, defaultResyncDuration, informers.WithNamespace(e.config.GetNamespace(step.OrgID)))
    

    And the event handler is:

    if _, err := si.Core().V1().Pods().Informer().AddEventHandler(
        cache.ResourceEventHandlerFuncs{
            UpdateFunc: podUpdated,
        },
    ); err != nil {
        return nil, err
    }
    

    But there's no DeleteFunc! So when the pod is deleted, the informer won't trigger any event. The WaitStep function will wait forever for the pod to reach a terminal state.

    But the context is not canceled... Wait, the context IS canceled when the workflow completes. So WaitStep should return with context.Canceled.

    Hmm, but what if the context is canceled before WaitStep starts? Then WaitStep will return immediately with context.Canceled.

    Actually, I think I need to trace through the flow more carefully with actual timing:

    1. wait-for-postgres step completes successfully (time T1)
    2. The main loop finishes all stages (time T2)
    3. destroyWorkflowFunc() is called (time T3)
    4. DestroyWorkflow iterates over all steps and destroys them (time T4)
    5. At time T4, the postgres service pod is deleted
    6. The postgres service step's goroutine is in WaitStep, waiting for the pod to reach a terminal state
    7. The informer sees the delete event but doesn't trigger any handler (no DeleteFunc)
    8. WaitStep waits for the context to be canceled
    9. The context is canceled when the workflow completes (time T5)
    10. WaitStep returns context.Canceled (time T6)
    11. completeStep creates a state with Error = pipeline_errors.ErrCancel
    12. DestroyStep is called (but the pod is already deleted, so it returns nil)
    13. traceStep is called with the state
    14. The server receives the state and updates the step to "failure" (because there's an error)

    But the issue says the step is "running", not "failure". This means the server didn't receive the state update, or the update was ignored.

    Actually, I think the issue might be with how the state is sent. Let me check the tracer implementation:

    <tool_call> <function=bash_terminal> <parameter=command> cat agent/tracer.go </tool_call>

  44. Qwen Assistant

    The command failed to produce output. Let me try using grep to find the tracer implementation.

  45. Qwen Assistant

    Let me look at the agent/tracer.go file to see how the tracer works.

  46. Qwen Assistant

    I found the issue! In agent/tracer.go, the createTracer function creates a StepState with:

    stepState := rpc.StepState{
        StepUUID: state.CurrStep.UUID,
        Exited:   state.CurrStepState.Exited,
        ExitCode: state.CurrStepState.ExitCode,
        Started:  state.CurrStepState.Started,
        Canceled: errors.Is(state.CurrStepState.Error, pipeline_errors.ErrCancel),
        Skipped:  state.CurrStepState.Skipped,
    }
    if state.CurrStepState.Error != nil {
        stepState.Error = state.CurrStepState.Error.Error()
    }
    if state.CurrStepState.Exited {
        stepState.Finished = time.Now().Unix()
    }
    

    The key line is:

    if state.CurrStepState.Exited {
        stepState.Finished = time.Now().Unix()
    }
    

    If Exited is false, Finished is not set! This means the server receives a state with Exited: false and Finished: 0.

    Looking at the server's CalcStepStatus:

    case model.StatusRunning:
        if state.Exited || state.Error != "" {
            step.Finished = state.Finished
            ...
        }
    

    If Exited: false and Error: "", the step state is not updated. But if Exited: false and Error != "", the step state SHOULD be updated.

    So the issue might be that the agent is sending Exited: false and Error: "". Let me check when this happens.

    Actually, I think I found the real issue! Looking at the flow:

    1. WaitStep is called and waits for the pod to reach a terminal state
    2. The pod is deleted by DestroyWorkflow before it reaches a terminal state
    3. WaitStep returns context.Canceled
    4. completeStep creates a state with Error = pipeline_errors.ErrCancel but Exited: false
    5. traceStep copies the state with Exited: false and Error != ""
    6. The server receives the state with Exited: false and Error != ""
    7. The server updates the step state because state.Error != ""

    So the step state SHOULD be updated. But the issue says the step is still "running".

    Wait, let me check what happens when WaitStep is called and the pod is already deleted. The Get call might fail with "not found":

    pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{})
    if err != nil {
        return nil, err
    }
    

    If the pod is not found, WaitStep returns an error. Then in completeStep:

    waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID)
    if err != nil {
        if errors.Is(err, context.Canceled) {
            if waitState == nil {
                waitState = &backend_types.State{}
            }
            waitState.Error = pipeline_errors.ErrCancel
        } else {
            return nil, err  // Returns without setting waitState.Exited!
        }
    }
    

    If the error is NOT context.Canceled (e.g., "pod not found"), the function returns nil, err. This means processState in runDetachedStep is nil, and traceStep is called with processState == nil and err != nil.

    Looking at traceStep:

    case processState == nil && err != nil:
        s.CurrStepState = backend_types.State{
            Error:     err,
            Exited:    true,
            OOMKilled: false,
        }
    

    A dummy state with Exited: true is created. So the agent sends a state with Exited: true and Error != "".

    But wait - the dummy state is created in traceStep, not in completeStep. So completeStep returns nil, err, and processState is nil in runDetachedStep. Then traceStep is called with processState == nil and err != nil.

    Looking at runDetachedStep:

    processState, err := r.completeStep(runnerCtx, step, waitForLogs, startTime)
    ...
    if traceErr := r.traceStep(processState, err, step); traceErr != nil {
        logger.Error().Err(traceErr).Str("step", step.Name).Msg("failed to trace detached step result")
    }
    

    If completeStep returns nil, err, then processState is nil. Then traceStep is called with processState == nil and err != nil, which creates a dummy state with Exited: true.

    So the agent SHOULD send a state with Exited: true. But the issue says the step is "running".

    Hmm, let me think about this differently. Maybe the issue is that the server's Update method is not being called at all. Let me check if there's any error handling that might prevent the state from being sent.

    Actually, I think I need to look at the actual flow more carefully. Let me check what happens when DestroyWorkflow is called before the service step's goroutine finishes.

    Looking at workflow.go:

    for _, stage := range r.spec.Stages {
        stageChan := r.runStage(runnerCtx, stage.Steps)
        select {
        case <-r.ctx.Done():
            <-stageChan
            return pipeline_errors.ErrCancel
        case err := <-stageChan:
            if err != nil {
                r.err.Set(err)
            }
        }
    }
    
    destroyWorkflowFunc()
    

    After all stages complete, destroyWorkflowFunc() is called. The DestroyWorkflow function iterates over all steps and destroys them:

    func (e *kube) DestroyWorkflow(ctx context.Context, conf *types.Config, taskUUID string) error {
        for _, stage := range conf.Stages {
            for _, step := range stage.Steps {
                err := stopPod(ctx, e, step, e.config.newDefaultDeleteOptions())
                if err != nil {
                    return err
                }
            }
        }
        ...
    }
    

    This destroys ALL steps, including services. So when DestroyWorkflow is called, the service pods are destroyed.

    But the service step's goroutine might be blocked in WaitStep. The WaitStep function waits for the pod to reach a terminal state. If the pod is deleted, the informer won't see it (no DeleteFunc), so the finished channel is never closed. The goroutine waits until the context is canceled.

    The context is not explicitly canceled here. The context is r.ctx, which is set in New:

    r.ctx = context.Background()
    

    And WithContext option can override this:

    pipeline_runtime.WithContext(workflowCtx)
    

    The workflowCtx is created in agent/runner.go:

    workflowCtx, _ := context.WithTimeout(ctxMeta, timeout)
    workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx)
    defer cancelWorkflowCtx(nil)
    

    The workflowCtx is canceled when the workflow completes or times out. But when is the workflow considered "complete"?

    Looking at agent/runner.go:

    err = pipeline_runtime.New(...).Run(runnerCtx)
    
    state.Finished = time.Now().Unix()
    ...
    if err := r.client.Done(doneCtx, workflow.ID, state); err != nil {
        logger.Error().Err(err).Msg("failed to update workflow status")
    }
    

    The Done method is called after Run returns. The Run method returns when all stages complete and uploadWait.Wait() completes.

    So the flow is:

    1. All stages complete
    2. destroyWorkflowFunc() is called, which destroys all pods
    3. uploadWait.Wait() is called, which waits for all goroutines to finish
    4. The service step's goroutine is blocked in WaitStep
    5. The goroutine waits for the context to be canceled
    6. But the context is not canceled! The context is only canceled when cancelWorkflowCtx(nil) is called in the defer of agent/runner.go
    7. So the goroutine waits forever!

    Wait, but the issue says the pods ARE destroyed. So the goroutine must finish eventually. Let me check when cancelWorkflowCtx(nil) is called.

    In agent/runner.go:

    workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx)
    defer cancelWorkflowCtx(nil)
    

    The defer cancelWorkflowCtx(nil) is called when Run returns. But Run is inside the agent/runner.go:Run function. So the context is canceled when the Run function returns.

    But Run waits for uploadWait.Wait(), which waits for the service step's goroutine to finish. So there's a deadlock!

    Actually, no. The cancelWorkflowCtx(nil) is deferred in agent/runner.go, not in pipeline/runtime/workflow.go. So the context is canceled when agent/runner.go:Run returns, which is after pipeline/runtime/workflow.go:Run returns.

    Let me trace through the flow more carefully:

    1. agent/runner.go:Run starts
    2. workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx) is called
    3. defer cancelWorkflowCtx(nil) is registered
    4. pipeline_runtime.New(...).Run(runnerCtx) is called
    5. pipeline/runtime/workflow.go:Run starts
    6. All stages complete
    7. destroyWorkflowFunc() is called, which destroys all pods
    8. uploadWait.Wait() is called
    9. The service step's goroutine is blocked in WaitStep
    10. The goroutine waits for the context to be canceled
    11. The context is NOT canceled yet! The defer in agent/runner.go is not executed yet
    12. So the goroutine waits forever!

    But wait, the issue says the pods ARE destroyed. So the goroutine must finish eventually. Let me check if there's a timeout somewhere.

    Actually, I think the issue is that the goroutine is blocked in WaitStep when DestroyWorkflow is called. The DestroyWorkflow destroys the pod, but the goroutine is still waiting. The goroutine will eventually finish when the context is canceled, but the context is only canceled after uploadWait.Wait() completes.

    This is a deadlock! The uploadWait.Wait() waits for the goroutine to finish, but the goroutine is waiting for the context to be canceled, and the context is only canceled after uploadWait.Wait() completes.

    But the issue says the pods ARE destroyed. So the goroutine must finish. Let me check if there's another way the context is canceled.

    Actually, I think the issue might be with how the context is handled. Let me check the Run method in workflow.go:

    func (r *Runtime) Run(runnerCtx context.Context) error {
        ...
        defer destroyWorkflowFunc()
    
        if err := r.engine.SetupWorkflow(r.ctx, r.spec, r.taskUUID); err != nil {
            ...
        }
    
        for _, stage := range r.spec.Stages {
            stageChan := r.runStage(runnerCtx, stage.Steps)
            select {
            case <-r.ctx.Done():
                <-stageChan
                return pipeline_errors.ErrCancel
            case err := <-stageChan:
                if err != nil {
                    r.err.Set(err)
                }
            }
        }
    
        destroyWorkflowFunc()
        r.uploadWait.Wait()
        return r.err.Get()
    }
    

    The r.ctx is used for the step operations (like WaitStep). If the context is canceled, WaitStep will return. But when is r.ctx canceled?

    Looking at pipeline/runtime/option.go:

    func WithContext(ctx context.Context) Option {
        return func(r *Runtime) {
            r.ctx = ctx
        }
    }
    

    And in agent/runner.go:

    pipeline_runtime.New(
        workflow.Config,
        r.backend,
        pipeline_runtime.WithContext(workflowCtx),
        ...
    ).Run(runnerCtx)
    

    The workflowCtx is the context passed to Run. The workflowCtx is canceled when cancelWorkflowCtx(nil) is called in the defer of agent/runner.go.

    But wait - looking at agent/runner.go:

    workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx)
    defer cancelWorkflowCtx(nil)
    

    The defer cancelWorkflowCtx(nil) is called when the agent/runner.go:Run function returns. But agent/runner.go:Run waits for pipeline/runtime/workflow.go:Run to return, and pipeline/runtime/workflow.go:Run waits for uploadWait.Wait() to complete.

    So if the service step's goroutine is blocked in WaitStep, the uploadWait.Wait() will wait forever, and the context will never be canceled.

    But the issue says the pods ARE destroyed. So the goroutine must finish eventually. Let me check if there's a timeout in WaitStep.

    Actually, I just realized something. The WaitStep function uses select to wait for either the finished channel or the context to be canceled:

    select {
    case <-finished:
    case <-ctx.Done():
        return nil, ctx.Err()
    }
    

    If the context is canceled, WaitStep returns ctx.Err(). The context is canceled when cancelWorkflowCtx(nil) is called. But cancelWorkflowCtx(nil) is only called after uploadWait.Wait() completes.

    So there's a deadlock! Unless... the finished channel is closed somehow.

    Wait, I think I understand now. The finished channel is closed when the pod reaches a terminal state (Succeeded, Failed, Unknown). But if the pod is deleted by DestroyWorkflow, the informer might not see the terminal state. The finished channel is never closed.

    But the context IS canceled eventually. When cancelWorkflowCtx(nil) is called, the context is canceled. But cancelWorkflowCtx(nil) is called after uploadWait.Wait() completes. And uploadWait.Wait() waits for the goroutine to finish. And the goroutine is waiting for the context to be canceled.

    This is a deadlock! Unless there's a timeout somewhere.

    Actually, let me re-read the agent/runner.go:Run function more carefully:

    workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx)
    defer cancelWorkflowCtx(nil)
    
    // Periodically extend the workflow lease while running
    go func() {
        for {
            select {
            case <-workflowCtx.Done():
                logger.Debug().Msg("workflow context done")
                return
            case <-time.After(constant.TaskTimeout / 3):
                ...
            }
        }
    }()
    

    There's a goroutine that extends the workflow lease. But this doesn't cancel the context.

    Actually, I think the issue might be that the workflowCtx has a timeout. Let me check:

    timeout := time.Hour
    if minutes := workflow.Timeout; minutes != 0 {
        timeout = time.Duration(minutes) * time.Minute
    }
    
    workflowCtx, _ := context.WithTimeout(ctxMeta, timeout)
    workflowCtx, cancelWorkflowCtx := context.WithCancelCause(workflowCtx)
    

    Yes! The workflowCtx has a timeout of time.Hour (or minutes * time.Minute if specified). So even if the goroutine is blocked in WaitStep, the context will be canceled when the timeout expires.

    But the timeout is 1 hour, which is way longer than the issue describes. The issue says the service is shown as "running" immediately after the pipeline completes.

    OK, I think I need to step back and think about this more carefully. Let me re-read the issue:

    "The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    The key phrase is "pods from kubernetes backend are destroyed properly". This means the pods ARE destroyed. But the step state is still "running".

    If the pods are destroyed, the service step's goroutine must have finished. If the goroutine finished, traceStep was called, and the server received a state update. If the server received a state update with Exited: true, the step state should be updated.

    So the question is: why is the step state still "running"?

    Possible reasons:

    1. The agent never sent a state update (goroutine never finished)
    2. The agent sent a state update, but the server didn't process it
    3. The agent sent a state update with Exited: false and Error: ""
    4. The server received the state update but didn't update the step correctly

    I think the issue might be #1 - the goroutine never finishes. But the issue says the pods ARE destroyed, which implies the goroutine finished.

    Actually, wait. Let me re-read the issue description more carefully:

    "once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running"

    So the wait-for-postgres step completes, and the pods are destroyed. But the service is still "running". This suggests that the service step's goroutine is still waiting when the pods are destroyed.

    Let me think about this flow:

    1. wait-for-postgres step completes
    2. The main loop finishes
    3. destroyWorkflowFunc() is called
    4. DestroyWorkflow destroys all pods, including the service
    5. The service step's goroutine is still waiting in WaitStep
    6. The goroutine waits for the context to be canceled
    7. The context is canceled when cancelWorkflowCtx(nil) is called
    8. But cancelWorkflowCtx(nil) is only called after uploadWait.Wait() completes
    9. uploadWait.Wait() waits for the goroutine to finish
    10. Deadlock!

    But the issue says the pods ARE destroyed. So the goroutine must finish. How?

    Actually, I think I might be missing something. Let me check if the finished channel is closed when the pod is deleted.

    Looking at WaitStep:

    podUpdated := func(_, newPod any) {
        pod, ok := newPod.(*kube_core_v1.Pod)
        ...
        switch pod.Status.Phase {
        case kube_core_v1.PodSucceeded, kube_core_v1.PodFailed, kube_core_v1.PodUnknown:
            finishedOnce.Do(func() { close(finished) })
        }
    }
    

    The podUpdated function only handles update events. If the pod is deleted, the informer won't trigger this function. But maybe the informer triggers a delete event that causes the finished channel to be closed?

    Actually, no. The informer only has an UpdateFunc, not a DeleteFunc. So when the pod is deleted, the informer won't trigger any event.

    But wait - the finished channel might be closed by the DeleteFunc if it's set. Let me check:

    if _, err := si.Core().V1().Pods().Informer().AddEventHandler(
        cache.ResourceEventHandlerFuncs{
            UpdateFunc: podUpdated,
        },
    ); err != nil {
        return nil, err
    }
    

    There's no DeleteFunc. So when the pod is deleted, the informer won't trigger any event.

    Hmm, but the issue says the pods ARE destroyed. So something must be happening.

    Actually, I think I need to check if the informer also handles delete events by default. Let me check the Kubernetes informer documentation.

    Actually, the Kubernetes informer's AddEventHandler takes a cache.ResourceEventHandler which has AddFunc, UpdateFunc, and DeleteFunc. If DeleteFunc is not set, the informer won't call any function on delete events.

    So when the pod is deleted, the finished channel is never closed. The WaitStep function waits forever for the context to be canceled.

    But the context is only canceled after uploadWait.Wait() completes. And uploadWait.Wait() waits for the goroutine to finish. This is a deadlock!

    Unless... the context is canceled somewhere else.

    Actually, I just realized something. The workflowCtx has a timeout! So even if the goroutine is blocked, the context will be canceled after the timeout expires.

    But the timeout is 1 hour, which is way longer than the issue describes.

    OK, I think I need to actually reproduce the issue to understand what's happening. Let me search for any related issues or PRs in the repository.

    <tool_call> <function=bash_terminal> <parameter=command> git log --oneline --all | head -30 </tool_call>

  47. Qwen Assistant

    Let me try the git command again properly.

  48. Qwen Assistant

    Let me search for issues or changes related to service cleanup or pipeline completion.

    <tool_call> <function=bash_terminal> <parameter=command> git log --oneline --all --grep="service" | head -20 </tool_call>

  49. Qwen Assistant

    Let me try running the git log command again properly.

  50. Qwen Assistant

    I found a relevant commit. Let me look at the changes in "Fix workflow hang on services (#6507)":

  51. Qwen Assistant

    This looks like exactly the issue I'm investigating. Let me look at the actual changes in this commit:

  52. Qwen Assistant

    Let me try a different approach to see the changes.

  53. Qwen Assistant

    Now I understand the issue better. The commit 60df1c618 was a fix for a similar issue where the workflow would hang because services were not torn down properly. The fix was to:

    1. Move DestroyWorkflow to be called BEFORE uploadWait.Wait()
    2. Add a stop channel to the dummy backend that services can wait on

    However, looking at the commit, I see that the fix was applied to the dummy backend to allow services to be stopped, but the actual kubernetes backend might not have this stop channel mechanism.

    Let me check if the kubernetes backend has the same issue - does it have a way to signal services to stop?

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "stop" pipeline/backend/kubernetes/kubernetes.go </tool_call>

  54. Qwen Assistant

    Let me continue exploring to find if the kubernetes backend has a stop mechanism for services.

  55. Qwen Assistant

    I see that the kubernetes backend has StopTimeout configuration for graceful shutdown. The DestroyStep and DestroyWorkflow functions call stopPod with the delete options that include the StopTimeout.

    But the issue is that the service pods are being destroyed (via DestroyWorkflow), but the step state is not being updated to "success" or "failure" because the agent's goroutine is blocked in WaitStep waiting for the pod to reach a terminal state.

    The problem is that the kubernetes WaitStep function uses an informer that only listens for "update" events, not "delete" events. So when the pod is deleted by DestroyWorkflow, the informer doesn't trigger, and the finished channel is never closed. The WaitStep function waits until the context is canceled.

    But the context is only canceled after uploadWait.Wait() completes, and uploadWait.Wait() waits for the goroutine to finish. This is a deadlock!

    The fix in commit 60df1c618 added a stop channel to the dummy backend that services can wait on. The kubernetes backend needs a similar mechanism.

    Actually, wait. Let me re-read the issue more carefully. The issue says:

    "The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running."

    If the pods ARE destroyed, that means DestroyWorkflow IS being called. But the step state is still "running", which means the server never received a state update with Exited: true.

    This suggests that the agent's goroutine is blocked in WaitStep and never finishes, so it never calls traceStep with the final state.

    But wait - if the pods are destroyed, how does the user know? The issue says "The pods from kubernetes backend are destroyed properly". This might be observed from the Kubernetes API, not from the Woodpecker API.

    So the flow is:

    1. All main steps complete
    2. DestroyWorkflow is called, which destroys all pods (including services)
    3. The service step's goroutine is blocked in WaitStep waiting for the pod to reach a terminal state
    4. The informer doesn't see the delete event (no DeleteFunc), so the finished channel is never closed
    5. The goroutine waits until the context is canceled
    6. But the context is only canceled after uploadWait.Wait() completes
    7. And uploadWait.Wait() waits for the goroutine to finish
    8. Deadlock!

    But the issue says the pods ARE destroyed, which means step 2 happened. But the goroutine is still blocked in step 3. The user observes this from the Woodpecker API, which shows the service as "running".

    Actually, I think the issue might be a timeout. The workflowCtx has a timeout (default 1 hour). So eventually, the context will be canceled, and WaitStep will return. But by that time, the server might have already marked the pipeline as done.

    Actually, wait. Let me re-read the issue again. The issue says:

    "These are only cleaned after the pipeline times out."

    So the services are cleaned up only after the pipeline times out! This confirms that the goroutine is blocked in WaitStep until the context times out.

    The fix should be to add a mechanism to signal services to stop, similar to what was done in the dummy backend in commit 60df1c618.

    Let me check if the kubernetes backend has a stop channel mechanism:

    <tool_call> <function=bash_terminal> <parameter=command> grep -n "chan|stop|close" pipeline/backend/kubernetes/pod.go | head -30 </tool_call>

  56. Qwen Assistant

    Let me try running the grep command again properly.

  57. Qwen Assistant

    Let me look at the kubernetes backend's WaitStep function more carefully to see how it handles pod deletion:

  58. Qwen Assistant

    Now I understand the issue. The WaitStep function uses an informer that only listens for "update" events. When the pod is deleted by DestroyWorkflow, the informer doesn't see the delete event (no DeleteFunc), so the finished channel is never closed.

    The fix should be to add a DeleteFunc to the informer that closes the finished channel when the pod is deleted. This way, when DestroyWorkflow deletes the service pods, the WaitStep function will be notified and will return.

    Let me also check if there's a DeleteFunc that should be added:

    <tool_call> <function=bash_terminal> <parameter=command> sed -n '280,335p' pipeline/backend/kubernetes/kubernetes.go </tool_call>

  59. Qwen Assistant

    The command failed to produce output. Let me try again with a proper format.

  60. Qwen Assistant

    Now I understand the issue. The WaitStep function in the kubernetes backend only listens for "update" events, not "delete" events. When DestroyWorkflow deletes the service pods, the informer doesn't notify WaitStep because there's no DeleteFunc handler. This causes WaitStep to block until the context times out, which is why the service steps remain "running" until the pipeline times out.

    The fix is to add a DeleteFunc to the informer that closes the finished channel when the pod is deleted. Let me implement this fix:

  61. Qwen Assistant

    Now I understand the issue. The WaitStep function in the kubernetes backend only listens for "update" events, not "delete" events. When DestroyWorkflow deletes the service pods, the informer doesn't notify WaitStep because there's no DeleteFunc handler. This causes WaitStep to block until the context times out, which is why the service steps remain "running" until the pipeline times out.

    The fix is to add a DeleteFunc to the informer that closes the finished channel when the pod is deleted. Let me implement this fix:

  62. Qwen Assistant

    Let me apply the fix directly using the edit command:

    <tool_call> <function=bash_terminal> <parameter=command> sed -n '306,322p' pipeline/backend/kubernetes/kubernetes.go </tool_call>

  63. Qwen Assistant

    The command failed to produce output. Let me try again with proper formatting.

  64. Qwen Assistant

    Now I can see the exact code. I need to:

    1. Add a podDeleted function before the informer setup (after line 310)
    2. Add DeleteFunc: podDeleted to the ResourceEventHandlerFuncs (line 316)

    Let me apply the fix:

  65. Qwen Assistant

    I need to open the file first:

  66. Qwen Assistant

    Now I can see the file is open. Let me scroll to the area where I need to make the changes (around line 310-320):

    <tool_call> <function=bash_terminal> <parameter=command> goto 280 </tool_call>

  67. Qwen Assistant

    Let me scroll down to find the WaitStep function where I need to add the DeleteFunc handler.

  68. Qwen Assistant

    Now I can see the exact code. I need to:

    1. Add a podDeleted function after line 310 (after the podUpdated function)
    2. Add DeleteFunc: podDeleted to the ResourceEventHandlerFuncs (line 315-316)

    Let me apply the fix:

    <tool_call> <function=bash_terminal> <parameter=command> edit 310:311 << 'EOF' }

    podDeleted := func(_ any) {
    	// Pod was deleted (e.g., by DestroyWorkflow) - signal completion
    	finishedOnce.Do(func() { close(finished) })
    }
    

    EOF </tool_call>

  69. Qwen Assistant

    Let me try using the edit command with the proper format:

  70. Qwen Assistant

    I see the podDeleted function was added, but I also need to add it to the ResourceEventHandlerFuncs. Let me do that now:

  71. Qwen Assistant

    I see there's a duplicate error handling block. Let me fix that:

    <tool_call> <function=bash_terminal> <parameter=command> edit 321:327 << 'EOF' ); err != nil { return nil, err }

    stop := make(chan struct{})
    si.Start(stop)
    defer close(stop)
    

    EOF </tool_call>

  72. Qwen Assistant

    Let me fix the duplicate error handling block that was created.

  73. Qwen Assistant

    I see there's a duplicate error handling block at lines 325-326. Let me remove it:

  74. Qwen Assistant

    Now there are duplicate lines at 328-329. Let me fix that:

    <tool_call> <function=bash_terminal> <parameter=command> edit 326:330 << 'EOF' stop := make(chan struct{}) si.Start(stop) defer close(stop)

    EOF </tool_call>

  75. Qwen Assistant

    Let me check the current state of the file and fix the duplicate lines:

  76. Qwen Assistant

    I see the duplicate lines. Let me fix them:

  77. Qwen Assistant

    Now let me verify the fix is correct:

  78. Qwen Assistant

    Now let me also fix the TailStep function which has the same issue:

  79. Qwen Assistant

    Let me look at more context for the TailStep function:

  80. Qwen Assistant

    I see the TailStep function has the same issue. Let me add the DeleteFunc handler there too:

  81. Qwen Assistant

    Let me add the podDeleted handler for TailStep:

  82. Qwen Assistant

    Now let me add the DeleteFunc to the event handler in TailStep:

  83. Qwen Assistant

    Let me add the DeleteFunc: podDeleted to the event handler:

  84. Qwen Assistant

    Now let me verify the changes are correct:

  85. Qwen Assistant

    Now let me verify the fix compiles correctly:

  86. Qwen Assistant

    Let me set the GOPATH and try again:

  87. Qwen Assistant

    Let me set up the environment properly:

  88. Qwen Assistant

    There's a duplicate DeleteFunc field. This happened because I added it to both WaitStep and TailStep using the same sed command. Let me check both locations and fix the issue:

  89. Qwen Assistant

    I see there's a duplicate at line 321. Let me fix this:

  90. Qwen Assistant

    Let me remove the duplicate DeleteFunc line:

  91. Qwen Assistant

    Now let me verify the fix compiles:

  92. Qwen Assistant

    The code compiles. Now let me also run the existing tests to make sure the fix doesn't break anything:

  93. Qwen Assistant

    The tests are passing so far. Let me run the tests for the runtime package as well since that's where the main issue might be:

    <tool_call> <function=bash_terminal> <parameter=command> export GOPATH=/root/go && export GOMODCACHE=GOPATH/pkg/mod && export GOCACHE=GOPATH/.cache/go-build && go test ./pipeline/runtime/... -v -count=1 2>&1 | tail -50 </tool_call>

  94. Qwen Assistant

    Let me run the runtime tests properly.

  95. Qwen Assistant

    There are no test files in the runtime package. Let me check if the tests are in a different location.

    <tool_call> <function=bash_terminal> <parameter=command> ls -la pipeline/runtime/ </tool_call>

  96. Qwen Assistant

    The command failed again. Let me try running the test command properly.

  97. Qwen Assistant

    There are test files in the runtime package. Let me run the tests:

    <tool_call> <function=bash_terminal> <parameter=command> export GOPATH=/root/go && export GOMODCACHE=GOPATH/pkg/mod && export GOCACHE=GOPATH/.cache/go-build && go test ./pipeline/runtime/... -run TestWorkflowUnbounded -v -count=1 </tool_call>

  98. Qwen Assistant

    The command is failing with an error message. Let me try running a simpler test command to see if the Go testing environment is working.

  99. Qwen Assistant

    It says "no test files" but I see runtime_test.go in the directory. Let me check the test file:

    <tool_call> <function=bash_terminal> <parameter=command> head -30 pipeline/runtime/runtime_test.go </tool_call>

  100. Qwen Assistant

    The command failed again. Let me try running the test command with proper syntax.

  101. Qwen Assistant

    The command is returning "no test files" even though there are test files. This might be a module path issue. Let me check the go.mod file and try running tests from the root:

    <tool_call> <function=bash_terminal> <parameter=command> go test go.woodpecker-ci.org/woodpecker/v3/pipeline/runtime -run TestWorkflowUnbounded -v -count=1 2>&1 | head -50 </tool_call>

  102. Qwen Assistant

    Let me check the go.mod file and run tests from the root:

  103. Qwen Assistant

    The test files exist but Go isn't finding them. Let me check if they have a build tag issue or if they're not recognized as test files for some reason:

    <tool_call> <function=bash_terminal> <parameter=command> head -20 pipeline/runtime/runtime_test.go </tool_call>

  104. Qwen Assistant

    Let me check the test files in a different way:

  105. Qwen Assistant

    I see! The test files have a //go:build test build tag, which means they only run when the test tag is specified. Let me run the tests with the test tag:

    <tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/runtime/... -tags test -run TestWorkflowUnbounded -v -count=1 2>&1 | head -50 </tool_call>

  106. Qwen Assistant

    The command is failing. Let me try running the test without the -v flag first to see if it works:

  107. Qwen Assistant

    The test passed! But I want to see the actual output to confirm the test ran. Let me run it with verbose output:

    <tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/runtime/... -tags test -run TestWorkflowUnbounded -v -count=1 </tool_call>

  108. Qwen Assistant

    Let me try a different approach to run the tests with verbose output:

  109. Qwen Assistant

    The tests pass! This confirms that the runtime correctly handles unbounded services and detached steps without hanging. The key observation from the test output is:

    • delete workflow environment is called
    • stop step db is called (the service step)
    • stop step background-worker is called (the detached step)
    • logs and traces uploaded completes

    This shows that the runtime is properly tearing down services after all main steps complete.

    However, I realized my fix in the kubernetes backend might not be the root cause of the issue. Looking at the test output, it seems the runtime is already handling this correctly by calling DestroyWorkflow before uploadWait.Wait(). The fix in commit 60df1c618 already addresses this.

    But the issue in the bug report is specifically about the kubernetes backend. The problem is that when DestroyWorkflow deletes the pods, the WaitStep function in the kubernetes backend doesn't get notified because it only listens for "update" events, not "delete" events.

    My fix adds a DeleteFunc handler that closes the finished channel when the pod is deleted. This should allow WaitStep to return immediately when the pod is deleted by DestroyWorkflow, rather than waiting until the context times out.

    Let me verify my changes are correct by looking at the modified file:

    <tool_call> <function=bash_terminal> <parameter=command> cat pipeline/backend/kubernetes/kubernetes.go | grep -A 10 "podDeleted" </tool_call>

  110. Qwen Assistant

    Let me check the changes I made to the kubernetes backend file.

  111. Qwen Assistant

    The changes look correct. I've added podDeleted handlers in both WaitStep (line 312-315) and TailStep (line 396-399) that close the respective channels when a pod is deleted. The DeleteFunc is added to the event handler in both places (line 320 and line 405).

    Now let me run all the kubernetes backend tests to make sure my changes don't break anything:

  112. Qwen Assistant

    All kubernetes backend tests pass. Let me also run the server-side tests related to pipeline status to make sure nothing is broken:

  113. Qwen Assistant

    The tests pass. Let me also run the step_status tests to make sure the status update logic is working correctly:

    <tool_call> <function=bash_terminal> <parameter=command> go test -v ./server/pipeline/... -run TestUpdateStepStatus 2>&1 | tail -80 </tool_call>

  114. Qwen Assistant

    Let me run the step_status tests to verify the status update logic is working correctly.

  115. Qwen Assistant

    All the tests pass! The TestUpdateStepStatus tests verify that the step status update logic is working correctly.

    Now let me verify the fix by looking at the final state of the modified file and ensuring the changes are correct:

    <tool_call> <function=bash_terminal> <parameter=command> cat pipeline/backend/kubernetes/kubernetes.go | head -350 | tail -70 </tool_call>

  116. Qwen Assistant

    The command failed again. Let me try using the edit command to view the relevant section of the file.

  117. Qwen Assistant

    Now let me check the TailStep function changes as well:

    <tool_call> <function=bash_terminal> <parameter=command> sed -n '390,425p' pipeline/backend/kubernetes/kubernetes.go </tool_call>

  118. Qwen Assistant

    The command is failing to produce output. Let me try a different approach.

  119. Qwen Assistant

    The changes look correct. I have added the podDeleted handler in both WaitStep and TailStep functions, and added the DeleteFunc to the event handlers. This ensures that when a pod is deleted (e.g., by DestroyWorkflow), the finished or up channel is closed, allowing the waiting goroutines to proceed.

    Now let me verify that the entire fix is complete by running a broader set of tests to ensure nothing is broken:

    <tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/... 2>&1 | tail -20 </tool_call>

  120. Qwen Assistant

    Let me run the tests in a different way:

  121. Qwen Assistant

    It shows FAIL at the end. Let me check which test failed:

    <tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/... 2>&1 | grep -A 5 "FAIL" </tool_call>

  122. Qwen Assistant

    Let me check which test failed by searching for "FAIL" in the test output.

  123. Qwen Assistant

    The failure is in the dummy backend build. Let me check what's wrong:

  124. Qwen Assistant

    The build succeeds. The test failure might be a cached test issue. Let me clear the cache and run the tests again:

  125. Qwen Assistant

    Let me check which specific test failed:

    <tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/... 2>&1 | grep "FAIL" </tool_call>

  126. Qwen Assistant

    The command seems to be failing to produce output. Let me try running the test differently to find the failure.

  127. Qwen Assistant

    The dummy backend is failing to build. Let me check what the actual error is:

    <tool_call> <function=bash_terminal> <parameter=command> go test ./pipeline/backend/dummy/... 2>&1 </tool_call>

  128. Qwen Assistant

    The command is failing to produce output. Let me try running the test command with proper error handling.

  129. Qwen Assistant

    The dummy backend requires the test build tag to compile. Let me run the tests with the test tag:

    <tool_call> <function=bash_terminal> <parameter=command> go test -tags test ./pipeline/... 2>&1 | tail -20 </tool_call>

  130. Qwen Assistant

    Let me run the tests with the test tag:

  131. Qwen Assistant

    All tests pass with the test tag! The earlier failures were due to the test tag being required for the dummy backend.

    Now let me also run the server tests to make sure the fix doesn't break anything on the server side:

    <tool_call> <function=bash_terminal> <parameter=command> go test -tags test ./server/... 2>&1 | tail -30 </tool_call>

  132. Qwen Assistant

    Let me run the server tests to ensure the fix doesn't break anything on the server side.

  133. Qwen Assistant

    There's a FAIL at the end. Let me check which test failed:

  134. Qwen Assistant

    The failures are in server/router and server/web which are likely due to frontend assets or build configuration, not related to my changes. Let me verify this:

    <tool_call> <function=bash_terminal> <parameter=command> go test -tags test ./server/router/... 2>&1 | head -20 </tool_call>

  135. Qwen Assistant

    Let me check the router test failure details.

  136. Qwen Assistant

    The failures are due to missing frontend build assets (web/web.go:26:12: pattern all:dist/*: no matching files found), not related to my changes. This is a pre-existing issue in the test environment where the web frontend hasn't been built.

    The important tests (kubernetes backend, runtime, server/pipeline) all pass. My fix is complete.

    Let me now summarize the changes I made and verify they are correct:

    Summary of Changes

    I fixed the issue where service steps remain "running" in the Woodpecker API after the pods are destroyed by the Kubernetes backend.

    Root Cause

    The Kubernetes backend's WaitStep function only listens for "update" events on pods, not "delete" events. When DestroyWorkflow deletes service pods, the informer doesn't notify WaitStep, causing it to block until the context times out (default 1 hour).

    Fix

    I added DeleteFunc handlers to the pod informers in both WaitStep and TailStep functions. When a pod is deleted, these handlers close the respective channels (finished for WaitStep, up for TailStep), allowing the waiting goroutines to proceed.

    Changes Made

    In pipeline/backend/kubernetes/kubernetes.go:

    1. Added podDeleted handler in WaitStep function (lines 312-315):
    podDeleted := func(_ any) {
        // Pod was deleted (e.g., by DestroyWorkflow) - signal completion
        finishedOnce.Do(func() { close(finished) })
    }
    
    1. Added DeleteFunc: podDeleted to the event handler in WaitStep (line 320)

    2. Added podDeleted handler in TailStep function (lines 396-399):

    podDeleted := func(_ any) {
        // Pod was deleted - allow log streaming to proceed
        upOnce.Do(func() { close(up) })
    }
    
    1. Added DeleteFunc: podDeleted to the event handler in TailStep (line 405)

    These changes ensure that when DestroyWorkflow deletes service pods, the waiting goroutines are immediately notified and can return, allowing the step state to be updated to "success" or "failure" on the server.

    Let me submit the changes:

  137. Qwen Assistant

    The diff shows the correct changes I made to fix the issue. Let me submit the fix:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.