Skip to content
TheChatArchiveThe Chat Archive
GLMz-ai/glm-5.2· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…

  1. User

    ISSUE DESCRIPTION

    Woodpecker services appear to be running on UI & API after pipeline completes

    Component

    server

    Describe the bug

    Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.

    These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).

    Steps to reproduce

    • Install woodpecker next-d05c597374
    • Use kubernetes backend
    • Define pipeline with services running infinitely, like running postgres
    services:
      - name: postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        ports:
          - 5432
        environment:
          POSTGRES_PASSWORD: somepass
          POSTGRES_USER: someuser
          POSTGRES_DB: somedb
    steps:
      - name: wait-for-postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        environment:
          DB_HOST: postgres
          DB_PORT: 5432
          MAX_WAIT: 60
        commands:
          - ./scripts/wait-for-postgres.sh
    
    • Run the pipeline, observe pods start for the service and the step
    • once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
    pr-build > clone (#2):
    Step: clone
    Started: 1779195580
    Stopped: 1779195594
    Type: clone
    State: success
    
    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    
    pr-build > wait-for-postgres (#4):
    Step: wait-for-postgres
    Started: 1779195596
    Stopped: 1779195602
    Type: commands
    State: success
    

    Expected behavior

    Pipeline services are stopped once the pipeline steps complete, the pipeline completes.

    System Info

    next-d05c597374
    kubernetes backend, bitbucket datacenter forge
    

    Additional context

    No response

    Validations

    Interface

    No new interfaces are introduced.

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /woodpecker, current file: none) bash-$

  2. GLM Assistant

    I'll start by exploring the repository structure to understand the codebase and then investigate the issue.

  3. GLM Assistant

    Let me understand the issue. Services (detached steps) in Woodpecker are not being marked as completed properly. The service pods are destroyed properly, but the API still shows them as running.

    Let me look at the runtime flow for detached steps. In runDetachedStep, the step starts a container and then runs completeStep in a goroutine which waits for the step to complete. For detached steps (services), they run infinitely, so WaitStep blocks until... what?

    The issue is about services that run infinitely. When the main steps complete, destroyWorkflowFunc() is called which destroys the workflow. But the detached step's goroutine is still waiting in WaitStep which blocks until the pod terminates.

    Wait, but DestroyWorkflow calls stopPod for all steps, which would delete the pods. So the detached service pods would be terminated by DestroyWorkflow. But the WaitStep goroutine is using r.ctx (the workflow context), not the runnerCtx.

    Let me look at the WaitStep for detached steps. In completeStep:

    waitState, err := r.engine.WaitStep(r.ctx, step, r.taskUUID) //nolint:contextcheck
    

    It uses r.ctx. For detached steps, this goroutine waits on r.ctx. When does r.ctx get canceled?

    Let me look at how the runtime is set up and when r.ctx is canceled.

  4. GLM Assistant

    Let me understand the flow. The r.ctx is context.Background() by default, never canceled. For detached steps (services), WaitStep blocks waiting for the pod to reach a terminal state. Let me look at the git history of the kubernetes backend.

  5. GLM Assistant

    Let me look at the previous version of the executor and how the contexts were handled before the refactor.

  6. GLM Assistant

    Now let me understand what r.ctx was in the old version. Let me check the old runtime.go.

  7. GLM Assistant

    Let me check the docker backend's WaitStep to understand how detached services are handled there.

  8. GLM Assistant

    Let me look at the backend types and how states are handled.

  9. GLM Assistant

    Let me look at how the agent runs the workflow and traces flow to the server.

  10. GLM Assistant

    There's a test runtime_test.go around line 1241 about "DestroyWorkflow to clean up state that WaitStep still needs" and the contract around detached steps. Let me read that test to understand the intended design.

  11. GLM Assistant

    There's a test TestWorkflowUnboundedServiceDoesNotHang that asserts the exact behavior we need. Let me read it.

  12. GLM Assistant

    These tests use the dummy backend. Let me look at the dummy backend to understand how unbounded services are handled there.

  13. GLM Assistant

    Let me look at the test helpers withUnboundedService, withService, etc.

  14. GLM Assistant

    Now let me look at stopPod to understand what happens when deleting an already-deleted pod.

  15. GLM Assistant

    Let me look at the existing kubernetes tests for WaitStep to understand the testing approach.

  16. GLM Assistant

    Now I understand the design. The dummy backend has a workflow stop channel that DestroyWorkflow closes and WaitStep listens on, so detached services unblock when torn down. The kubernetes backend's WaitStep only listens to UpdateFunc (not DeleteFunc), so when DestroyWorkflow deletes a service pod, WaitStep never unblocks — the detached goroutine hangs forever, and Run blocks at r.uploadWait.Wait(). This is the regression.

    Let me write a reproduction test first to confirm the issue with the fake client.

  17. GLM Assistant

    Need to set up Go environment. Let me check if there's a Go installation and set GOMODCACHE/GOPATH.

  18. GLM Assistant

    Confirmed the hang. But my test used the wrong pod name. Let me fix the test to use stepToPodName.

  19. GLM Assistant

    Let me fix the reproduction test to use stepToPodName.

  20. GLM Assistant

    Confirmed the bug. The pod is deleted but WaitStep hangs because there's no DeleteFunc handler. Now let me implement the fix in WaitStep.

  21. GLM Assistant

    Now let me implement the fix. I'll add a DeleteFunc handler and handle the deletion gracefully.

  22. GLM Assistant

    I used a non-existent helper. Let me check the proper way to check for NotFound errors and the existing imports.

  23. GLM Assistant

    Now let me implement the fix in the actual file. First, add the k8s api errors import.

  24. GLM Assistant

    Now let me replace the WaitStep function body using the edit tool.

  25. GLM Assistant

    Now let me build and run my reproduction test:

  26. GLM Assistant

    The reproduction test now passes. Let me run the existing kubernetes tests to ensure nothing is broken.

  27. GLM Assistant

    All kubernetes tests pass. Now let me run the runtime tests (which use the dummy backend) to ensure nothing is broken.

  28. GLM Assistant

    Hmm, no test files? That's odd. The runtime tests must be in a build tag. Let me check.

  29. GLM Assistant

    Let me check the default StopTimeout and how the graceful deletion interacts with log streaming.

  30. GLM Assistant

    Now let me consider the graceful-deletion race more carefully. When DestroyWorkflow deletes a service pod gracefully, the pod gets a DeletionTimestamp and may transition through PodFailed (exit 137 if SIGKILL'd) before removal. To robustly treat a teardown as a clean finish (not a failure), I'll also check the DeletionTimestamp in the UpdateFunc. Let me update podUpdated.

  31. GLM Assistant

    Let me build and run the kubernetes tests plus my repro test:

  32. GLM Assistant

    All tests pass. Let me add a more thorough test covering the DeletionTimestamp path (graceful teardown) to verify that branch works too.

  33. GLM Assistant

    Let me check if there are tests asserting canceled detached services are marked as canceled, to ensure my change doesn't break that.

  34. GLM Assistant

    Let me reconsider a critical race in the real flow. For a detached service, waitForLogs() blocks until the log stream closes (when the pod is deleted/terminated). So WaitStep's informer starts AFTER the pod may already be gone (especially with GracePeriodSeconds=0). Let me write a test that simulates this: start WaitStep after the pod is already deleted.

  35. GLM Assistant

    Confirmed the pod-already-gone case hangs. I need to add an initial Get check after the informer starts. Let me update the code.

  36. GLM Assistant

    Let me restructure the section to add an initial Get check after the informer starts, and extract the clean-finish state to avoid duplication.

  37. GLM Assistant

    Let me rebuild and run all the tests including the repro tests:

  38. GLM Assistant

    All tests pass. Let me confirm the existing WaitStep cancel/leak tests specifically pass:

  39. GLM Assistant

    Now let me run the broader test suite (runtime, pipeline, agent) to make sure nothing else breaks.

  40. GLM Assistant

    All pipeline tests pass. Let me also run the agent tests to be safe.

  41. GLM Assistant

    Let me run go vet on the kubernetes package to catch any issues:

  42. GLM Assistant

    Let me refine: for the initial Get, only NotFound (and DeletionTimestamp) should short-circuit to a clean finish. Any other transient Get error should fall through to the original informer-based wait (preserving prior behavior), not fail immediately.

  43. GLM Assistant

    Let me review the full final WaitStep function once more for correctness.

  44. GLM Assistant

    The fix looks robust. Now let me remove my temporary reproduction test file (per instructions, I must not add new tests to the repository).

  45. GLM Assistant

    Now let me run the full kubernetes and runtime test suites once more to confirm everything still passes without the temp test:

  46. GLM Assistant

    Let me verify the whole project builds with the change:

  47. GLM Assistant

    The web embed warning is pre-existing (missing dist/* frontend build artifacts), unrelated to my change. Let me confirm the relevant packages build cleanly:

  48. GLM Assistant

    The web embed error comes from ./cmd/... which imports web. That's pre-existing. My packages (pipeline, agent) build fine. Let me verify by building just the pipeline and agent.

    Actually go build ./pipeline/... ./agent/... - the web error must come from a transitive import. Let me check - actually the agent might import web? No. Let me just build pipeline and agent separately.

  49. GLM Assistant

    Both pipeline and agent build cleanly. Let me view the final diff of my changes:

  50. GLM Assistant

    The diff is clean. Let me run gofmt to ensure formatting is correct:

  51. GLM Assistant

    Let me check if gofumpt (which the repo uses) is available and run it:

  52. GLM Assistant

    Let me check the gofumpt version the repo uses and try installing it to verify formatting:

  53. GLM Assistant

    gofumpt is also happy. Let me do a final confirmation that no temp files remain and the git status is clean except for my intended change:

  54. GLM Assistant

    Only pipeline/backend/kubernetes/kubernetes.go is modified and no temp/untracked files remain. Let me run the kubernetes and runtime test suites one final time to confirm everything is green:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.