Skip to content
TheChatArchiveThe Chat Archive
x-ai/grok-4.5· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…

  1. User

    ISSUE DESCRIPTION

    Woodpecker services appear to be running on UI & API after pipeline completes

    Component

    server

    Describe the bug

    Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.

    These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).

    Steps to reproduce

    • Install woodpecker next-d05c597374
    • Use kubernetes backend
    • Define pipeline with services running infinitely, like running postgres
    services:
      - name: postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        ports:
          - 5432
        environment:
          POSTGRES_PASSWORD: somepass
          POSTGRES_USER: someuser
          POSTGRES_DB: somedb
    steps:
      - name: wait-for-postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        environment:
          DB_HOST: postgres
          DB_PORT: 5432
          MAX_WAIT: 60
        commands:
          - ./scripts/wait-for-postgres.sh
    
    • Run the pipeline, observe pods start for the service and the step
    • once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
    pr-build > clone (#2):
    Step: clone
    Started: 1779195580
    Stopped: 1779195594
    Type: clone
    State: success
    
    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    
    pr-build > wait-for-postgres (#4):
    Step: wait-for-postgres
    Started: 1779195596
    Stopped: 1779195602
    Type: commands
    State: success
    

    Expected behavior

    Pipeline services are stopped once the pipeline steps complete, the pipeline completes.

    System Info

    next-d05c597374
    kubernetes backend, bitbucket datacenter forge
    

    Additional context

    No response

    Validations

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /woodpecker, current file: none) bash-$

  2. Assistant

    I'll start by exploring the repository structure and understanding how pipeline services are cleaned up after completion.

  3. Assistant

    I'll look for how pipeline services are stopped and how step states are updated when a pipeline finishes.

  4. Assistant

    Let me examine the runtime and how services are handled when workflows complete.

  5. Assistant

    rg is not available, let me use grep instead.

  6. Assistant

    There's a test about unbounded services hanging — let me read the runtime code and that test carefully.

  7. Assistant

    Let me continue reading the rest of the file.

  8. Assistant

    Let me continue reading.

  9. Assistant

    Now I'll check workflow cleanup and the unbounded service test.

  10. Assistant

    Let me inspect the Kubernetes backend's step wait and destroy handling.

  11. Assistant

    Let me read the WaitStep and DestroyWorkflow implementations carefully.

  12. Assistant

    I think WaitStep may miss pod deletions — only Update is handled. Let me verify DestroyWorkflow and compare with other backends.

  13. Assistant

    Let me read DestroyWorkflow

  14. Assistant

    Checking delete options and whether peach deletion events are handled over�.

  15. Assistant

    Checking recent changes and git history related to service teardown.

  16. Assistant

    Looking at the earlier fix for services hanging and the related server-side cleanup.

  17. Assistant

    There's already a server-side fix for stuck services — inspecting that RPC path.

  18. Assistant

    Let me look at Done function where completeChildrenIfParentCompleted is used.

  19. Assistant

    I think I've found the race: WaitStep starts after logs drain, but the pod may already be gone. Verifying and checking related tests.

  20. Assistant

    The dummy backend properly implements DestroyWorkflow to kill running steps. The k8s backend deletes pods but WaitStep doesn't handle already-deleted pods.

  21. Assistant

    Let me read the WaitStep and DestroyWorkflow of dummy backend fully.

  22. Assistant

    Checking how the agent manages context lifecycle around runtime.Run.

  23. Assistant

    Confirmed hang until timeout. Checking how the Docker Backend implementation handles this and what the existing WaitStep tests cover.

  24. Assistant

    Checking dummy TailStep behavior and docker WaitStep for completeness.

  25. Assistant

    Looking at docker DestroyWorkflow - ContainerWait may return immediately if already exited.

  26. Assistant

    Checking gitblame/history of Waitオーラ Step and github issues for more context.

  27. Assistant

    Checking local backend for the waitForLogs-before-WaitStep comment.

  28. Assistant

    Checking how exit codes and kill status are interpreted for services.

  29. Assistant

    There's ExitCodeKilled. Let me check that and how tracers map state.

  30. Assistant

    For services that get killed, when we report exited with non-zero exit code, they'd show as failure. Looking at completeChildrenIfParentCompleted - it sets them as killed/skipped when workflow Done is...

  31. Assistant

    Root cause: WaitStep starts after logs drain, so the pod may already be deleted after DestroyWorkspace — and the informer only watches updates, so it hangs. Implementing a fix.

  32. Assistant

    Let me also look at utility functions like isImagePullBackOffState and check imports available for k8s errors.

  33. Assistant

    Let me review the test helpers and full kubernetes test file.

  34. Assistant

    Checking stopPod and the default stop timeout.

  35. Assistant

    I've confirmed the root cause: WaitStep only handles Update events and starts after logs drain, so an already-terminal or deleted pod never signals. Implementing the fix.

  36. Assistant

    The edit landed in the wrong file — reverting pod.go and applying the change to kubernetes.go.

  37. Assistant

    Now applying the WaitStep fix to the correct file.

  38. Assistant

    Updating imports to include the packages we need.

  39. Assistant

    Adding tests for the new WaitStep behavior.

  40. Assistant

    I'll add helper for finished pods and tests at the end of the file.

  41. Assistant

    Adding the new tests at the end of the file.

  42. Assistant

    I need to add import for pipeline package in the test file for ExitCodeKilled.

  43. Assistant

    Now append the new tests.

  44. Assistant

    Checking how stepToPodName works so pod names line up.

  45. Assistant

    Running the Kubernetes backend tests.

  46. Assistant

    Need to set up the Go environment first.

  47. Assistant

    There's a syntax error — checking the end of the test file.

  48. Assistant

    There's an extra brace at the end — removing it.

  49. Assistant

    Verifying the earlier tests weren't corrupted skip by the edit.

  50. Assistant

    Rerunning the tests.

  51. Assistant

    All tests passed. Verifying the new tests specifically ran.

  52. Assistant

    Confirming no import cycle and reviewing the final implementation.

  53. Assistant

    Reporting destroyed services as success (exit 0) matches how the server marks cleaned-up daemons.

  54. Assistant

    The edit broke TailStep — fixing that.

  55. Assistant

    Restoring the TailStep function header and cleaning up the duplicate comment.

  56. Assistant

    Updating tests and removing the unused pipeline import.

  57. Assistant

    Updating the tests to expect exit code 0.

  58. Assistant

    Now update the assertions that reference pipeline.ExitCodeKilled

  59. Assistant

    I need to open that section first or use sed.

  60. Assistant

    Now I can replace.

  61. Assistant

    Need to fix the second occurrence too.

  62. Assistant

    Rerunning tests and verifying the full file compiles.

  63. Assistant

    All WaitStep tests passed. Verifying the build and the end of the file.

  64. Assistant

    The instructions say not to add or modify tests — restoring the test file and keeping only the production fix.

  65. Assistant

    Creating a temporary script to verify the fix, then removing it.

  66. Assistant

    Running the full package tests and reviewing the final diff.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.