Skip to content
TheChatArchiveThe Chat Archive
xiaomi/mimo-v2.5-pro· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…

  1. User

    ISSUE DESCRIPTION

    Woodpecker services appear to be running on UI & API after pipeline completes

    Component

    server

    Describe the bug

    Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.

    These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).

    Steps to reproduce

    • Install woodpecker next-d05c597374
    • Use kubernetes backend
    • Define pipeline with services running infinitely, like running postgres
    services:
      - name: postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        ports:
          - 5432
        environment:
          POSTGRES_PASSWORD: somepass
          POSTGRES_USER: someuser
          POSTGRES_DB: somedb
    steps:
      - name: wait-for-postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        environment:
          DB_HOST: postgres
          DB_PORT: 5432
          MAX_WAIT: 60
        commands:
          - ./scripts/wait-for-postgres.sh
    
    • Run the pipeline, observe pods start for the service and the step
    • once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
    pr-build > clone (#2):
    Step: clone
    Started: 1779195580
    Stopped: 1779195594
    Type: clone
    State: success
    
    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    
    pr-build > wait-for-postgres (#4):
    Step: wait-for-postgres
    Started: 1779195596
    Stopped: 1779195602
    Type: commands
    State: success
    

    Expected behavior

    Pipeline services are stopped once the pipeline steps complete, the pipeline completes.

    System Info

    next-d05c597374
    kubernetes backend, bitbucket datacenter forge
    

    Additional context

    No response

    Validations

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /woodpecker, current file: none) bash-$

  2. Assistant

    I'll start by exploring the repository structure to understand the codebase and locate relevant files related to services and pipeline completion.

  3. Assistant

    Let me look at the issue more carefully. The problem is that services are not being stopped after the pipeline completes. The services are showing as state=running in the API even though the pods are destroyed. Let me look at the pipeline execution and service cleanup logic.

  4. Assistant

    The issue is that after pipeline steps complete, service steps are not being marked as stopped/completed in the API/UI. The pods are destroyed but the state is still showing as "running".

    Let me explore how the workflow/step execution works and how states are updated.

  5. Assistant

    Let me look at the server pipeline code and the RPC service to understand how steps are updated.

  6. Assistant

    Let me look at the tracer.go file, as it seems relevant for updating step states.

  7. Assistant

    Let me look at the pipeline runtime workflow execution:

  8. Assistant

    Let me look at executeStep to understand how steps are executed and cleaned up.

  9. Assistant

    Now let me understand how services are configured. Let me look at the backend types:

  10. Assistant

    Now let me understand the flow better. Let me look at how the compiler sets up services:

  11. Assistant

    Let me look at the compiler to see how service steps are created and whether they're marked as detached.

  12. Assistant

    Now let me look at how createProcess handles service steps and detached flag:

  13. Assistant

    Let me find the createProcess method.

  14. Assistant

    Let me search more broadly for createProcess:

  15. Assistant

    Let me look at how the service steps are compiled. Let me check the createProcess function.

  16. Assistant

    Now let me look at the kubernetes backend to understand how services are handled:

  17. Assistant

    Now let me look at the runDetachedStep function and the WaitStep implementation for kubernetes more carefully:

  18. Assistant

    Let me look at the Runtime struct and how the context is used:

  19. Assistant

    Let me check the Runtime struct and its context.

  20. Assistant

    Now let me look at how the runner works:

  21. Assistant

    Let me now look at how service step states are tracked on the server side and how they are updated when the pipeline completes:

  22. Assistant

    Let me look at the workflow status update logic to understand how the overall pipeline status is computed and updated.

  23. Assistant

    Now let me look at how the RPC client handles the Update call:

  24. Assistant

    Now let me look at the pipeline runtime to understand how service steps complete their lifecycle:

  25. Assistant

    Let me look at how services are handled as "detached" steps and how the step state transitions work:

  26. Assistant

    Let me look at the key issue. Let me trace the lifecycle more carefully. Let me look at how destroyWorkflowFunc interacts with WaitStep:

  27. Assistant

    Now I need to understand the exact issue. Let me look at the WaitStep implementation more carefully and trace through what happens when a service step's pod gets destroyed:

  28. Assistant

    Now I can see the problem! Let me trace the flow:

    1. Service steps are compiled as Detached: true steps
    2. runDetachedStep starts the step, then spawns a goroutine calling completeStep (which calls WaitStep)
    3. WaitStep uses a kubernetes informer with only an UpdateFunc handler watching for pod phase changes to Succeeded/Failed/Unknown
    4. When all pipeline stages finish, destroyWorkflowFunc() calls DestroyWorkflow which calls stopPod to delete the pods
    5. When a pod is deleted, the pod's phase doesn't necessarily change to Failed or Succeeded - it can be deleted while still in Running phase
    6. The informer only has UpdateFunc, no DeleteFunc - so when the pod is deleted, finished channel never gets closed
    7. WaitStep eventually returns when r.ctx times out (pipeline timeout)

    Let me verify this hypothesis by looking at stopPod:

  29. Assistant

    Let me look at the stopPod function to understand how pods are deleted.

  30. Assistant

    Let me check recent git history for the relevant files to see if there was a regression:

  31. Assistant

    Let me look at the recent "Fix workflow hang on services" commit and the one that moved wait for log uploads:

  32. Assistant

    This is very relevant! Let me look at this commit in detail.

  33. Assistant

    Now I understand the issue! Let me trace through the exact flow:

    1. Services are Detached: true steps, so runDetachedStep starts them and spawns a background goroutine calling completeStep → WaitStep
    2. WaitStep in kubernetes only watches for UpdateFunc events, checking for pod phases Succeeded/Failed/Unknown
    3. After all stages complete, destroyWorkflowFunc() calls DestroyWorkflow → stopPod which deletes the pod
    4. When the pod is deleted in kubernetes, it can be removed without going through Succeeded/Failed/Unknown phase (pod gets deleted from etcd directly)
    5. The informer's UpdateFunc handler doesn't fire with a terminal phase, and there's no DeleteFunc handler
    6. WaitStep blocks until the context times out
    7. The service step is stuck in "running" state until pipeline timeout

    Let me verify this by looking at what happens after the pod deletion:

  34. Assistant

    Let me search for those functions in the kubernetes package.

  35. Assistant

    Let me check the backend interface to understand what WaitStep should return:

  36. Assistant

    Let me look at the StopTimeout configuration:

  37. Assistant

    Let me also check how the workflow context is set and what triggers its cancellation:

  38. Assistant

    Let me now understand the full picture. Let me look at the workflow runner to understand the timing:

  39. Assistant

    Now I have a clear understanding of the bug. Let me trace through the exact issue:

    1. Services are Detached: true steps → runDetachedStep spawns a goroutine that calls completeStep → WaitStep
    2. WaitStep in kubernetes creates an informer that only handles UpdateFunc (pod status changes)
    3. After all stages complete, destroyWorkflowFunc() calls DestroyWorkflow → stopPod which deletes the pods
    4. When a pod is deleted, the informer receives a Delete event, NOT an Update event
    5. Since there's no DeleteFunc handler, the finished channel never gets closed
    6. WaitStep blocks until ctx.Done() (pipeline timeout)
    7. The service step remains in "running" state until the pipeline timeout

    Let me fix this by adding a DeleteFunc handler to the informer in WaitStep:

  40. Assistant

    Now I understand the issue! Let me look at how the service step state flows through the system:

  41. Assistant

    Let me now look at the exact sequence of events to confirm my understanding. Let me also check how the TailStep informer is implemented for comparison:

  42. Assistant

    Now I have a clear understanding of the bug. Let me trace through the exact flow:

    1. Service steps are Detached: true, so runDetachedStep starts them and spawns a goroutine calling completeStep → WaitStep
    2. WaitStep in kubernetes creates an informer that only has UpdateFunc handler
    3. When all stages complete, destroyWorkflowFunc() calls DestroyWorkflow → stopPod which deletes pods
    4. The critical bug: When a pod is deleted from the API server, the informer receives a Delete event, NOT an Update event. There's no DeleteFunc handler, so the finished channel never gets closed
    5. WaitStep falls through to ctx.Done() which only fires on pipeline timeout
    6. The service step remains in "running" state until the pipeline timeout

    The fix is to add a DeleteFunc handler to the informer in WaitStep, and handle the case where the pod has already been deleted when we try to fetch its status afterwards.

    Let me implement the fix:

  43. Assistant

    Now let me implement the fix. I need to add a DeleteFunc handler and handle the "pod not found" case:

  44. Assistant

    Now let me implement the fix. I need to:

    1. Add a DeleteFunc handler to close the finished channel when the pod is deleted
    2. Handle the "not found" case when fetching the pod status after deletion
  45. Assistant

    Let me verify the change by viewing the complete function:

  46. Assistant

    Now let me check if the kubernetes API error handling is correct. The "not found" error from kubernetes client-go should be handled properly:

  47. Assistant

    Let me check how the codebase handles "not found" errors from the k8s API:

  48. Assistant

    I need to use the kubernetes API error handling for "not found" errors. Let me update the fix:

  49. Assistant

    Now I need to add the import for kube_errors. Let me check the current imports:

  50. Assistant

    I need to add the kube_errors import for k8s.io/apimachinery/pkg/api/errors. Let me add it.

  51. Assistant

    Now let me verify the fix compiles:

  52. Assistant

    Let me check the go environment.

  53. Assistant

    The GOPATH and GOMODCACHE are empty. Let me set them.

  54. Assistant

    Let me set up the environment properly:

  55. Assistant

    Good, the build succeeds! Now let me also run the existing tests to make sure nothing is broken:

  56. Assistant

    Let me see the rest of the test output:

  57. Assistant

    All tests pass. Now let me also check if there are any existing tests that specifically test the WaitStep behavior:

  58. Assistant

    All tests pass. Now let me also run the pipeline runtime tests to make sure nothing is broken there:

  59. Assistant

    Let me check if the tests are in a different package structure:

  60. Assistant

    Let me check why the tests didn't run.

  61. Assistant

    Let me check the package declaration of the test files:

  62. Assistant

    The tests are in the runtime package, not the runtime_test package. Let me try running them differently.

  63. Assistant

    Let me run the tests from the right directory.

  64. Assistant

    The test files use package runtime (not package runtime_test), and go test with a path that doesn't have external test files will show "no test files". Let me check why...

  65. Assistant

    The test files are internal (package runtime, not runtime_test), so they should be found by go test. Let me check if the build tag is the issue.

  66. Assistant

    Let me look at the file more carefully:

  67. Assistant

    The test files have //go:build test build tag. Let me run the tests with that tag.

  68. Assistant

    The test passes. Let me also run all the pipeline runtime tests to make sure nothing is broken:

  69. Assistant

    All tests pass. Let me verify the changes by reviewing the final state of the file:

  70. Assistant

    Let me now check the git diff to verify the changes:

  71. Assistant

    Let me now review the full file to make sure the import for standard errors is still used (and not orphaned):

  72. Assistant

    Good, the errors import is still used. Let me also verify there are no unused imports:

  73. Assistant

    All clean. Let me also look at the existing WaitStep tests to see if there's a test for pod deletion that I should verify:

  74. Assistant

    The existing tests only test context cancellation, not pod deletion. Let me verify the existing tests still pass:

  75. Assistant

    All tests pass. Let me run a final comprehensive check on all tests in the repository that could be affected:

  76. Assistant

    All tests pass. Let me now verify the complete diff one final time:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.