Skip to content
TheChatArchiveThe Chat Archive
DeepSeekdeepseek/deepseek-v4-pro· deepseekHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

ISSUE DESCRIPTION Woodpecker services appear to be running on UI & API after pipeline…

  1. User

    ISSUE DESCRIPTION

    Woodpecker services appear to be running on UI & API after pipeline completes

    Component

    server

    Describe the bug

    Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.

    These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).

    Steps to reproduce

    • Install woodpecker next-d05c597374
    • Use kubernetes backend
    • Define pipeline with services running infinitely, like running postgres
    services:
      - name: postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        ports:
          - 5432
        environment:
          POSTGRES_PASSWORD: somepass
          POSTGRES_USER: someuser
          POSTGRES_DB: somedb
    steps:
      - name: wait-for-postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        environment:
          DB_HOST: postgres
          DB_PORT: 5432
          MAX_WAIT: 60
        commands:
          - ./scripts/wait-for-postgres.sh
    
    • Run the pipeline, observe pods start for the service and the step
    • once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
    pr-build > clone (#2):
    Step: clone
    Started: 1779195580
    Stopped: 1779195594
    Type: clone
    State: success
    
    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    
    pr-build > wait-for-postgres (#4):
    Step: wait-for-postgres
    Started: 1779195596
    Stopped: 1779195602
    Type: commands
    State: success
    

    Expected behavior

    Pipeline services are stopped once the pipeline steps complete, the pipeline completes.

    System Info

    next-d05c597374
    kubernetes backend, bitbucket datacenter forge
    

    Additional context

    No response

    Validations

    ADDITIONAL ADVICE

    Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use git checkout <file> to restore the file to its original state. You can also reset all changes in the repository using git reset --hard command. Additionally, you can use git stash and git stash pop to temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you call submit) using git status or git diff commands. However you don't need to use git add or git commit before submitting your changes. In fact, submit only submits currently unstaged changes because it uses git diff to get the changes that need to be submitted.

    CHECKLIST

    Before submitting your solution, please go over the following checklist and make sure you've done everything:

    • If an error was reported in the issue description, I have successfully reproduced it.
    • If an error was reported in the issue description, I have confirmed that my fix resolves the error.
    • I have thought about edge cases and made sure my fix handles them as well.
    • I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.

    Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!

    (Current directory: /woodpecker, current file: none) bash-$

  2. DeepSeek Assistant

    Let me start by understanding the repository structure and the issue. The bug is about Woodpecker CI services not being properly cleaned up - the pods are destroyed on Kubernetes but the API still shows them as "running" state. This seems like a regression in 3.14.x.

    Let me explore the repository structure first, understand the codebase, and then find the relevant code that handles service completion/cleanup.

  3. DeepSeek Assistant

    Let me explore the relevant directories. I'll start with the pipeline directory and look for code related to service state

  4. DeepSeek Assistant

    I need to make only one tool call at a time.

  5. DeepSeek Assistant

    Let me look at the pipeline state directory and the backend directory since this is about service state management.

  6. DeepSeek Assistant

    Let me look at the backend files and also the kubernetes backend:

  7. DeepSeek Assistant

    Let me look at the kubernetes backend:

  8. DeepSeek Assistant

    Let me look at the main kubernetes backend file and the pod.go

  9. DeepSeek Assistant

    Let me look at the pipeline runtime code - this is likely where the issue is since it manages the pipeline execution

  10. DeepSeek Assistant

    Let me look at the runtime directory and the server pipeline code:

  11. DeepSeek Assistant

    Let me look at the runtime files

  12. DeepSeek Assistant

    Let me look at workflow.go and step.go:

  13. DeepSeek Assistant

    Let me look at step.go

  14. DeepSeek Assistant

    Now let me look at shutdown.go:

  15. DeepSeek Assistant

    Let me look at the server pipeline handling code:

  16. DeepSeek Assistant

    Let me search for files that handle pipeline steps and services:

  17. DeepSeek Assistant

    Let me look for the main pipeline step handling in the server:

  18. DeepSeek Assistant

    Let me also search for where the pipeline execution/running logic is in the server, and where service steps are handled:

  19. DeepSeek Assistant

    Let me look at the

  20. DeepSeek Assistant

    Let me look at how pipeline steps are managed in server code, especially the pipeline creation and step state updates

  21. DeepSeek Assistant

    Let me look at the pipeline step_status.go

  22. DeepSeek Assistant

    Let me look at the RPC handler to see how step states are communicated:

  23. DeepSeek Assistant

    Now let me look at the agent code to see how it handles service steps:

  24. DeepSeek Assistant

    Let me look at the agent runner.go

  25. DeepSeek Assistant

    Let me look at the tracer.go and state.go:

  26. DeepSeek Assistant

    Let me look at the pipeline frontend yaml to see how services are handled

  27. DeepSeek Assistant

    Let me look at the compiler to see how services are compiled into steps

  28. DeepSeek Assistant

    Let me look at the compiler.go to understand how services are compiled

  29. DeepSeek Assistant

    Let me look at the convert.go to see how createProcess works

  30. DeepSeek Assistant

    Let me look at the kubernetes pod.go to understand how pods are stopped

  31. DeepSeek Assistant

    Let me look at the git log for recent changes to the pipeline runtime files

  32. DeepSeek Assistant

    Let me look at the commit that fixed workflow hang on services

  33. DeepSeek Assistant

    Let me look at the full diff of this commit

  34. DeepSeek Assistant

    Now I understand the issue. The problem is in the Kubernetes backend's WaitStep function. When DestroyWorkflow deletes the service pod, the WaitStep function (which is running in a goroutine for the detached service step) has an informer that only registers UpdateFunc, not DeleteFunc. When a pod is deleted, the delete event is ignored, and the finished channel is never closed. This causes WaitStep to block until the context times out.

    Let me also check the TailStep function for the same issue

  35. DeepSeek Assistant

    Let me look at the exact code around both handlers

  36. DeepSeek Assistant

    Let me look at the TailStep handler

  37. DeepSeek Assistant

    Now I understand the issue. Let me verify my understanding by looking at the full flow once more. The issue is in the Kubernetes backend's WaitStep function. When DestroyWorkflow deletes the service pod, the WaitStep function's informer only has UpdateFunc registered, not DeleteFunc. When a pod is deleted, the delete event is ignored, and the finished channel is never closed. This causes WaitStep to block until the workflow context times out.

    But wait, let me also check if there's an issue with the WaitStep being called after the pod is already deleted — the pod might not exist at all when the informer starts.

    Let me also look at the TailStep function which has the same pattern

  38. DeepSeek Assistant

    Now I have a clear understanding of the bug. Let me implement the fix.

    The issue is in the Kubernetes backend's WaitStep function. When DestroyWorkflow deletes the service pod, the WaitStep function's pod informer only has UpdateFunc registered. When a pod is deleted:

    1. The delete event is sent to the informer
    2. But since only UpdateFunc is registered, the delete event is ignored
    3. The finished channel is never closed
    4. WaitStep blocks until the workflow context times out

    The fix is to add a DeleteFunc handler in both WaitStep and TailStep that closes the signal channels when the pod is deleted

  39. DeepSeek Assistant

    Let me use the open command correctly.

  40. DeepSeek Assistant

    Let me see the full WaitStep function

  41. DeepSeek Assistant

    Now I understand the issue fully. Let me implement the fix. The WaitStep function needs to handle the case where the pod is deleted (by DestroyWorkflow) by adding a DeleteFunc handler. I also need to handle the case where the pod no longer exists when WaitStep starts.

    Let me also look at the TailStep function to see if it needs the same fix:

  42. DeepSeek Assistant

    Now let me implement the fix. I'll edit both WaitStep and TailStep to add DeleteFunc handlers

  43. DeepSeek Assistant

    Let me use the replace command instead

  44. DeepSeek Assistant

    Let me check the raw content of the file around the area I want to edit

  45. DeepSeek Assistant

    I see the file uses tabs. Let me use the edit command with proper tab indentation

  46. DeepSeek Assistant

    Good, the WaitStep function is updated. Now let me also update the TailStep function. Let me also check if errors package has IsNotFound:

  47. DeepSeek Assistant

    Let me check how errors.IsNotFound is used elsewhere in the kubernetes backend

  48. DeepSeek Assistant

    Let me check the imports in pod.go

  49. DeepSeek Assistant

    Let me check the imports in kubernetes.go

  50. DeepSeek Assistant

    I need to add the k8s errors import to kubernetes.go since I'm using errors.IsNotFound. But the standard library errors is already imported. I need to use an alias

  51. DeepSeek Assistant

    Now I need to update the reference from errors.IsNotFound to kube_errors.IsNotFound in the WaitStep function

  52. DeepSeek Assistant

    Let me navigate to that line first

  53. DeepSeek Assistant

    Now I can replace errors.IsNotFound with kube_errors.IsNotFound

  54. DeepSeek Assistant

    I need to also handle the case where the pod is already deleted before the informer starts. Let me add a pre-check and also add the DeleteFunc to TailStep. Let me first add the pre-check in WaitStep:

  55. DeepSeek Assistant

    Now let me also add the DeleteFunc to TailStep for consistency. Let me view the current state of TailStep first

  56. DeepSeek Assistant

    Let me also update TailStep to add a DeleteFunc handler

  57. DeepSeek Assistant

    Let me use the heredoc approach

  58. DeepSeek Assistant

    Now let me verify the code compiles correctly

  59. DeepSeek Assistant

    Let me check the Go environment

  60. DeepSeek Assistant

    Let me try setting up the Go module cache

  61. DeepSeek Assistant

    Let me set up GOCACHE as well

  62. DeepSeek Assistant

    The code compiles. Let me now run the tests for the kubernetes backend and the pipeline runtime

  63. DeepSeek Assistant

    Let me continue looking at the test output

  64. DeepSeek Assistant

    All kubernetes tests pass. Let me also run the pipeline runtime tests

  65. DeepSeek Assistant

    Let me check the test files in the runtime directory

  66. DeepSeek Assistant

    Let me try running tests with the correct path

  67. DeepSeek Assistant

    Let me check the package declaration in the test files

  68. DeepSeek Assistant

    Let me check the package declaration.

  69. DeepSeek Assistant

    Let me check if there are build tags

  70. DeepSeek Assistant

    Let me run with the test build tag

  71. DeepSeek Assistant

    Both tests pass. Let me now review the full changes

  72. DeepSeek Assistant

    The diff looks good. Let me run the full runtime test suite to ensure nothing is broken

  73. DeepSeek Assistant

    All tests pass. Let me verify the file looks correct in its entirety

  74. DeepSeek Assistant

    Let me check if there are other places in the codebase with similar patterns that might need the

  75. DeepSeek Assistant

    Good - only the two places I already fixed use AddEventHandler. Let me run the kubernetes tests one more time to confirm everything passes

  76. DeepSeek Assistant

    All tests pass. Let me also run the runtime tests once

  77. DeepSeek Assistant

    All tests pass. Let me verify the final state of

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.