Skip to content
TheChatArchiveThe Chat Archive
composer-2.5· unknownHugging Face ibragim-bad/swe_rebench_07_2026_trajectories

Woodpecker services appear to be running on UI & API after pipeline completes Component…

  1. User

    Woodpecker services appear to be running on UI & API after pipeline completes

    Component

    server

    Describe the bug

    Some regression with newer releases (3.14.x) With kubernetes backend and pipelines defining services, the services are not cleaned properly. The pods from kubernetes backend are destroyed properly, but the woodpecker API still show these steps as state=running.

    These are only cleaned after the pipeline times out. This prevents timely deploys as pipeline executions are not getting green status (although, as a mitigation, the pipeline waiting services can be cancelled, after which the deploy can be made).

    Steps to reproduce

    • Install woodpecker next-d05c597374
    • Use kubernetes backend
    • Define pipeline with services running infinitely, like running postgres
    services:
      - name: postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        ports:
          - 5432
        environment:
          POSTGRES_PASSWORD: somepass
          POSTGRES_USER: someuser
          POSTGRES_DB: somedb
    steps:
      - name: wait-for-postgres
        image: public.ecr.aws/docker/library/postgres:16-alpine3.21
        environment:
          DB_HOST: postgres
          DB_PORT: 5432
          MAX_WAIT: 60
        commands:
          - ./scripts/wait-for-postgres.sh
    
    • Run the pipeline, observe pods start for the service and the step
    • once wait-for-postgres step completes, the pods are destroyed, but API still shows the service as running, also visible on Web UI.
    pr-build > clone (#2):
    Step: clone
    Started: 1779195580
    Stopped: 1779195594
    Type: clone
    State: success
    
    pr-build > postgres (#3):
    Step: postgres
    Started: 1779195594
    Stopped: 0
    Type: service
    State: running
    
    pr-build > wait-for-postgres (#4):
    Step: wait-for-postgres
    Started: 1779195596
    Stopped: 1779195602
    Type: commands
    State: success
    

    Expected behavior

    Pipeline services are stopped once the pipeline steps complete, the pipeline completes.

    System Info

    next-d05c597374
    kubernetes backend, bitbucket datacenter forge
    

    Additional context

    No response

    Validations

    Interface

    No new interfaces are introduced.

  2. Assistant

    Investigating how Woodpecker cleans up pipeline services after steps complete — likely a regression in service state handling.

  3. Assistant

    Identifying the issue: DestroyWorkflow deletes pods, but Kubernetes WaitStep only unblocks on terminal pod phases—not on deletion—so detached service goroutines hang and the API never gets the completion trace. Implementing a fix.

  4. Assistant

    Implementing the fix: Kubernetes WaitStep doesn't detect pod deletion, so detached service goroutines hang after DestroyWorkflow removes pods.

  5. Assistant

    Root Cause

    This is a regression from the v3.14 change that waits for detached steps/services to report status before the workflow finishes (#6471, fixed partially in #6507).

    The sequence for a long-running service (e.g. Postgres):

    1. All regular steps finish successfully.
    2. destroyWorkflowFunc() runs and deletes Kubernetes pods via DestroyWorkflow.
    3. The detached service goroutine is still blocked in WaitStep, waiting for a terminal pod phase (Succeeded/Failed/Unknown).
    4. Deleted pods never reach those phases, so WaitStep never returns.
    5. uploadWait.Wait() blocks → agent never calls Done() → API keeps the service step in running until pipeline timeout.

    The server-side fallback in completeChildrenIfParentCompleted only runs when Done() is called, so it never triggers here.

    Fix

    Updated the Kubernetes backend WaitStep to:

    1. Listen for pod deletion via the informer's DeleteFunc.
    2. Treat a missing pod as a successful exit when Get returns NotFound (pod was torn down by DestroyWorkflow).
    func (e *kube) WaitStep(ctx context.Context, step *types.Step, taskUUID string) (*types.State, error) {
    	// ...
    	if _, err := si.Core().V1().Pods().Informer().AddEventHandler(
    		cache.ResourceEventHandlerFuncs{
    			UpdateFunc: podUpdated,
    			DeleteFunc: podDeleted,
    		},
    	); err != nil {
    		return nil, err
    	}
    	// ...
    	pod, err := e.client.CoreV1().Pods(e.config.GetNamespace(step.OrgID)).Get(ctx, podName, kube_meta_v1.GetOptions{})
    	if kube_errors.IsNotFound(err) {
    		return &types.State{
    			ExitCode: 0,
    			Exited:   true,
    		}, nil
    	}
    	// ...
    }
    

    Added TestWaitStepReturnsOnPodDeletion to cover this path.

    After this change, when workflow teardown deletes service pods, the agent traces the service as finished, the workflow completes green, and deploys are no longer blocked.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.