Skip to content
TheChatArchiveThe Chat Archive
xiaomi/mimo-v2.5-pro· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…

  1. User

    ISSUE DESCRIPTION

    Woodpecker services appear to be running on UI & API after pipeline completes

    Component

    server

    Describe the bug

    Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.

    These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).

    Steps to reproduce

    • Install woodpecker next-d05c597374
    • Use kubernetes backend
    • Define pipeline with services running infinitely, like running postgres
    services:
      - name: postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        ports:
          - 5432
        environment:
          POSTGRES_PASSWORD: somepass
          POSTGRES_USER: someuser
          POSTGRES_DB: somedb
    steps:
      - name: wait-for-postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        environment:
          DB_HOST: postgres
          DB_PORT: 5432
          MAX_WAIT: 60
        commands:
          - ./scripts/wait-for-postgres.sh
    
    • Run the pipeline, observe pods start for the service and the step
    • once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
    pr-build > clone (#2):
    Step: clone
    Started: 1779195580
    Stopped: 1779195594
    Type: clone
    State: success
    
    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    
    pr-build > wait-for-postgres (#4):
    Step: wait-for-postgres
    Started: 1779195596
    Stopped: 1779195602
    Type: commands
    State: success
    

    Expected behavior

    Pipeline services are stopped once the pipeline steps complete, the pipeline completes.

    System Info

    next-d05c597374
    kubernetes backend, bitbucket datacenter forge
    

    Additional context

    No response

    Validations

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /woodpecker, current file: none) bash-$

  2. Assistant

    I'll start by exploring the repository structure to understand the codebase and locate relevant files for the issue. The bug is about services not being cleaned up after pipeline completes, showing as running in API/UI. This seems to be a server-side issue with the pipeline execution state management.

    Let me first get an overview of the repository.

  3. Assistant

    Let me explore the repository structure more broadly.

  4. Assistant

    Let me search for files related to pipeline workflow/step execution and state management.

  5. Assistant

    Let me explore the pipeline and server directories more to understand how workflows and services are managed.

  6. Assistant

    Let me look at the key areas of the code. The issue is about services not being cleaned up after the pipeline completes. The services continue to show as "running" in the API/UI even after the pipeline's steps complete. Let me look at:

    1. How services/workflows are managed in the pipeline
    2. The agent's workflow execution logic
    3. How step completion is tracked and services are stopped

    Let me start by looking at the pipeline directory structure and the agent's workflow handling.

  7. Assistant

    Let me explore the runtime directory and look at workflow execution.

  8. Assistant

    Let me look at the workflow.go file to understand how workflows execute services.

  9. Assistant

    Now let's look at step execution to understand how service steps are handled.

  10. Assistant

    Let me examine the tracer implementation to understand how step states are reported.

  11. Assistant

    Let me check the agent-side code to see how the tracer communicates state back to the server.

  12. Assistant

    Now let me understand the execution flow. Let me check how services are structured in the backend types:

  13. Assistant

    Let me look at the pipeline backend types to understand step types:

  14. Assistant

    Let me look at the compiler to understand how services are compiled into stages:

  15. Assistant

    Let me look at how the compiler creates stages from services and steps:

  16. Assistant

    Now let me look at how services are organized into stages and how the workflow stages are set up:

  17. Assistant

    Let me look at the compiler to understand how services are organized into stages:

  18. Assistant

    Let me look at the runtime structure and how context is managed:

  19. Assistant

    Now let me look at the agent runner to understand the full lifecycle:

  20. Assistant

    Now let me look at how the detached steps (services) are handled. I need to understand the lifecycle. Let me check the WaitStep implementation for kubernetes backend:

  21. Assistant

    Let me understand the flow more clearly. Let me check the server-side RPC handler for step updates:

  22. Assistant

    Let me look at the server-side step status handling:

  23. Assistant

    Let me look at the checkWorkflowAllowsStepUpdate function that sanitizes agent input:

  24. Assistant

    Now let me look at the Done function in detail to see what happens when the workflow completes:

  25. Assistant

    Now let me look at the kubernetes backend's WaitStep implementation:

  26. Assistant

    Let me look at how DestroyWorkflow works for kubernetes backend:

  27. Assistant

    Now let me look at the stopPod function to understand how pods are deleted:

  28. Assistant

    Let me search for stopPod function.

  29. Assistant

    Let me look at the stopPod function and how delete options are configured:

  30. Assistant

    Let me look at the default delete options and how they affect the pod lifecycle:

  31. Assistant

    Now I need to understand the key issue. Let me look at the WaitStep for kubernetes more carefully:

  32. Assistant

    Now let me check how destroyWorkflowFunc and service cleanup interact. Let me look at the DestroyStep implementation for kubernetes:

  33. Assistant

    Let me look more carefully at the WaitStep function - particularly what happens when a pod is deleted (not just updated):

  34. Assistant

    Let me look at the default delete options:

  35. Assistant

    Now I understand the issue. Let me trace the flow carefully:

    1. Service steps run as Detached in runDetachedStep
    2. runDetachedStep spawns a background goroutine calling completeStep → WaitStep(r.ctx, ...) and returns immediately
    3. After all stages complete, destroyWorkflowFunc() deletes all pods
    4. But WaitStep only registers UpdateFunc on the kubernetes informer, not DeleteFunc
    5. When the pod is deleted, the informer may not see a terminal phase update, so WaitStep blocks until workflow timeout

    The key issue is in WaitStep — it only handles pod UpdateFunc, missing DeleteFunc. When pods are force-deleted or deleted before reaching a terminal state, the service step's WaitStep never returns.

  36. Assistant

    Let me check the StopTimeout configuration:

  37. Assistant

    Now let me look at what the backend_types.State looks like:

  38. Assistant

    Let me look at how the Docker backend handles WaitStep for comparison:

  39. Assistant

    Now I understand the full picture. Let me look at how the stages are organized with services and regular steps:

  40. Assistant

    Now I understand the full picture. Let me trace the execution flow:

    1. Service stage runs → services start as Detached steps → stage completes immediately (background goroutine for each service keeps running WaitStep)
    2. Steps stage runs → regular steps complete → stage completes
    3. destroyWorkflowFunc() is called → DestroyWorkflow deletes ALL pods (including service pods)
    4. r.uploadWait.Wait() blocks waiting for service goroutine to finish

    The problem: In step 3, when the service pod is deleted, the WaitStep function in the service goroutine only has an UpdateFunc handler on the informer - no DeleteFunc. When a pod is deleted (especially with short/zero grace period), it may be removed from Kubernetes without transitioning through Succeeded/Failed phase. The UpdateFunc never fires, so WaitStep blocks until the workflow context timeout.

    This is exactly the reported behavior: "services are not cleaned properly... only cleaned after the pipeline times out."

    The fix: Add a DeleteFunc handler to the informer in WaitStep to detect pod deletion.

  41. Assistant

    Now let me also check how kubeerrors is used to check for not-found errors:

  42. Assistant

    Let me check the imports in kubernetes.go:

  43. Assistant

    Now let me also check the errors import to make sure kubeerrors is available:

  44. Assistant

    Now I understand the full problem. The WaitStep in the Kubernetes backend only watches for UpdateFunc on the pod informer. When a service pod is deleted by destroyWorkflowFunc(), the pod may transition to a terminal phase (which triggers UpdateFunc), but there's also a race where the pod could be removed before the terminal state update reaches the informer, or the informer might not fire UpdateFunc with the right phase. More importantly, when destroyWorkflowFunc() is called, it deletes the pod. The pod will be deleted with a grace period (StopTimeout), during which SIGTERM is sent to the containers. However, there's a critical path issue:

    1. destroyWorkflowFunc() calls stopPod() with the delete options
    2. The pod deletion happens, which eventually causes the container to terminate
    3. WaitStep should detect the pod reaching a terminal state via the informer
    4. After finished is closed, WaitStep tries to Get() the pod
    5. If the pod is already deleted by then, the Get() call returns a 404 error

    The key issue: When destroyWorkflowFunc() calls DestroyWorkflow, it deletes all pods. The service pod is still running. After deletion, the pod goes through termination. But WaitStep only watches for UpdateFunc. When the pod is fully removed (DELETE event), there's no DeleteFunc handler to close the finished channel.

    Additionally, there's a more subtle issue. When completeStep() finishes for the service step (after WaitStep returns), it calls DestroyStep(). But destroyWorkflowFunc() already destroyed the workflow (including all pods). The DestroyStep() call might fail because the pod is already gone, and the error from DestroyStep() causes completeStep() to return nil, err without setting the Started field. This means traceStep gets called with processState=nil and err != nil, which should still mark the step as exited.

    But the main issue is that WaitStep doesn't handle the pod being deleted. When the pod is deleted and there's no UpdateFunc callback with a terminal state, the informer never closes the finished channel. The WaitStep only returns when ctx.Done() is triggered (i.e., context timeout or cancellation).

    Let me now implement the fix. I need to:

    1. Add a DeleteFunc handler to the pod informer that also closes the finished channel
    2. Handle the case where the pod is deleted and Get() returns a 404 error - instead of returning an error, return a state with the information we have

    Actually, let me re-think. The main flow is:

    1. Services stage runs: service step starts, runDetachedStep spawns a goroutine that calls completeStep -> WaitStep(r.ctx, step, r.taskUUID)
    2. Steps stage runs: blocking steps complete
    3. destroyWorkflowFunc() is called: deletes all pods (including service pods)
    4. r.uploadWait.Wait() waits for the service goroutine

    When destroyWorkflowFunc() deletes the service pod:

    • The pod gets a deletion timestamp
    • SIGTERM is sent to containers
    • The container should terminate, transitioning the pod to Succeeded/Failed
    • WaitStep's informer should see the Update and close finished

    But what if the container doesn't terminate cleanly within the grace period? Then SIGKILL is sent. But the pod should still transition to Failed.

    Actually, I think the real issue might be different. Let me look at the timing more carefully.

    When destroyWorkflowFunc() calls stopPod(), it calls Delete() on the pod. The pod will be deleted asynchronously (with grace period). But destroyWorkflowFunc() doesn't wait for the pods to actually terminate - it just issues the delete request.

    Then r.uploadWait.Wait() waits for the service goroutine. The service goroutine is in WaitStep, waiting for the pod to reach a terminal state.

    Now, the pod deletion should cause the container to terminate, which should cause the pod to reach a terminal state. But there's a timing issue: what if the pod is fully removed from Kubernetes before the informer sees the terminal state transition? In that case, the UpdateFunc would never fire with a terminal state.

    When a pod is deleted in Kubernetes:

    1. The pod gets a deletion timestamp (UPDATE event)
    2. The kubelet sends SIGTERM to containers
    3. After grace period, SIGKILL is sent
    4. The pod transitions to terminal state (Succeeded/Failed) (UPDATE event)
    5. The pod is removed from the API server (DELETE event)

    Steps 2-4 happen asynchronously. The informer should see the UPDATE at step 4. But if the pod is removed quickly (e.g., if StopTimeout is 0), the pod might go directly from step 1 to step 5 without a terminal state update.

    Actually, looking at the default delete options:

    func (c *config) newDefaultDeleteOptions() kube_meta_v1.DeleteOptions {
        propagationPolicy := kube_meta_v1.DeletePropagationBackground
        return kube_meta_v1.DeleteOptions{
            GracePeriodSeconds: &c.StopTimeout,
            PropagationPolicy:  &propagationPolicy,
        }
    }
    

    The GracePeriodSeconds is c.StopTimeout. If this is 0, the pod is force-deleted immediately, which means the informer might only see a DELETE event, not an UPDATE with terminal state.

    Even if StopTimeout is not 0, there's still a race: the informer might not process the terminal state update before the pod is removed.

    I think the fix should be:

    1. Add a DeleteFunc handler to close the finished channel when the pod is deleted
    2. Handle the case where the pod is deleted and we can't get its state (return a reasonable default state with Exited=true)

    Let me implement this now.Now I have a clear understanding of the bug. Let me implement the fix. The issue is in WaitStep - it only watches for UpdateFunc but not DeleteFunc on the pod informer. When destroyWorkflowFunc() deletes the service pod, the pod might be fully removed from Kubernetes (DELETE event) without the informer seeing a terminal state update, causing WaitStep to block until the context timeout.

    Additionally, when the pod is deleted and WaitStep tries to Get() it after detecting the event, it gets a 404 error, which propagates incorrectly.

    Let me fix both issues:

  45. Assistant

    Now I'll implement the fix. I need to:

    1. Add a DeleteFunc handler to the informer that also closes the finished channel
    2. Handle the case where the pod is deleted and Get() returns a 404 error - instead of propagating the error, return a state indicating the step was exited
  46. Assistant

    Now I need to add the kubeerrors import:

  47. Assistant

    I see there's a formatting issue in the imports. Let me fix it:

  48. Assistant

    Good. Let me verify the complete WaitStep function looks correct:

  49. Assistant

    Now let me verify the code compiles:

  50. Assistant

    Let me try setting up the Go environment properly.

  51. Assistant

    I need to set GOPATH and GOMODCACHE.

  52. Assistant

    I need to set HOME and GOCACHE too.

  53. Assistant

    Good, the code compiles. Let me also run the existing tests:

  54. Assistant

    Let me run all the kubernetes backend tests:

  55. Assistant

    All tests pass. Let me also run the runtime tests:

  56. Assistant

    Let me also run the agent tests:

  57. Assistant

    Let me also run the RPC server tests:

  58. Assistant

    Let me also check the completeChildrenIfParentCompleted function in the server's RPC handler, which is important for understanding how the server handles the workflow completion:

  59. Assistant

    Now let me check the existing tests for WaitStep to see if we should add a test for the delete case:

  60. Assistant

    Let me also run a broader set of tests to make sure nothing is broken:

  61. Assistant

    Let me check if the dummy test failure is pre-existing:

  62. Assistant

    Good - the dummy backend build failure is pre-existing (build constraints exclude all Go files). All kubernetes backend tests pass.

    Now let me verify the full file looks correct:

  63. Assistant

    The fix looks correct. Let me also run the server pipeline tests to make sure nothing is broken:

  64. Assistant

    Let me verify the complete diff of my changes:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.