Skip to content
TheChatArchiveThe Chat Archive
gpt-4.1-mini-2025-04-14· OpenAIAi2 WildChat-4.8M (AI2)

antix@antix:~$ kubectl -n longhorn-system get pod NAME READY STATUS RESTARTS AGE…

  1. User

    antix@antix:~kubectl−nlonghorn−systemgetpodNAMEREADYSTATUSRESTARTSAGElonghorn−driver−deployer−7f95558b85−wflrt0/1Pending03m55slonghorn−driver−deployer−7f95558b85−ztg2l0/1Init:Error07m24slonghorn−manager−h5m5l0/2Evicted082slonghorn−manager−ngr660/2Evicted082slonghorn−ui−7ff79dfb4−5rvl70/1Error07m24slonghorn−ui−7ff79dfb4−7scm40/1Error07m24slonghorn−ui−7ff79dfb4−9mrgh0/1Pending04m8slonghorn−ui−7ff79dfb4−fplx60/1Pending04m37santix@antix:  kubectl -n longhorn-system get pod NAME READY STATUS RESTARTS AGE longhorn-driver-deployer-7f95558b85-wflrt 0/1 Pending 0 3m55s longhorn-driver-deployer-7f95558b85-ztg2l 0/1 Init:Error 0 7m24s longhorn-manager-h5m5l 0/2 Evicted 0 82s longhorn-manager-ngr66 0/2 Evicted 0 82s longhorn-ui-7ff79dfb4-5rvl7 0/1 Error 0 7m24s longhorn-ui-7ff79dfb4-7scm4 0/1 Error 0 7m24s longhorn-ui-7ff79dfb4-9mrgh 0/1 Pending 0 4m8s longhorn-ui-7ff79dfb4-fplx6 0/1 Pending 0 4m37s antix@antix:~

  2. Assistant

    Your Longhorn pods are in problematic states: some are Pending, some Init:Error, some Evicted, and some Error.

    Let's break down what these statuses typically mean and how you can troubleshoot each:

    1. Pods in Pending

    • Meaning: Pod is waiting to be scheduled to a node.
    • Common causes:
      • No node meets the resource requests or tolerations.
      • Insufficient resources (CPU, Memory).
      • NodeSelector or affinity rules can't be satisfied.

    Check:

    kubectl describe pod <pod-name> -n longhorn-system
    

    Look under Events for reasons why pod scheduling failed.


    2. Pods in Init:Error

    • Meaning: An init container failed to start properly.

    Check:

    kubectl logs <pod-name> -n longhorn-system -c <init-container-name>
    

    Replace <init-container-name> with the init container(s) listed in the pod spec.

    Common reasons can be:

    • Missing images.
    • Permission errors.
    • Issues with volume mounts.

    3. Pods in Evicted

    • Meaning: Pod was terminated by Kubernetes node eviction mechanisms.
    • Cause: Usually due to node resource pressure (memory, disk).

    Check:

    • Why were the pods evicted? Check node metrics.
    • More info in pod events:
    kubectl describe pod <evicted-pod-name> -n longhorn-system
    

    Look specifically for reason: "Evicted" and messages explaining the cause.

    Likely cause with Longhorn:

    • Disk pressure or memory pressure on the node(s).
    • Longhorn often requires a lot of disk space and memory.

    4. Pods in Error

    • Meaning: Pod container(s) have failed.

    Check:

    kubectl logs <pod-name> -n longhorn-system
    kubectl describe pod <pod-name> -n longhorn-system
    

    Look at container states, exit codes, and events.


    General Debug tips:

    1. Check node status and resources:
    kubectl get nodes
    kubectl describe node <node-name>
    

    Especially look for disk pressure, memory pressure, or NotReady nodes.

    1. Check storage: Longhorn heavily depends on available disk space; ensure your nodes are not running low on disk.

    2. Check Longhorn system logs:

    kubectl logs <longhorn-pod-name> -n longhorn-system
    
    1. Check Kubernetes events for related errors:
    kubectl get events -n longhorn-system
    

    Summary / Next steps:

    • For evicted pods: fix node resource pressure (free memory/disk).
    • For pending pods: inspect scheduling constraints or cluster resources.
    • For Init errors: check the init container logs for detailed error messages.
    • For error pods: check container logs.

    If you provide the output of kubectl describe pod <pod-name> -n longhorn-system or logs, I can help further analyze.


    Let me know if you want assistance fetching/decoding some logs!

  3. User

    antix@antix:~kubectl−nlonghorn−systemgetpodNAMEREADYSTATUSRESTARTSAGElonghorn−driver−deployer−7f95558b85−wflrt0/1Init:0/107m44slonghorn−driver−deployer−7f95558b85−ztg2l0/1Init:Error011mlonghorn−manager−8tvkr0/2Evicted03m2slonghorn−manager−b9w4d0/2Evicted03m2slonghorn−ui−7ff79dfb4−5rvl70/1Error011mlonghorn−ui−7ff79dfb4−7scm40/1Error011mlonghorn−ui−7ff79dfb4−9mrgh1/1Running1(21sago)7m57slonghorn−ui−7ff79dfb4−fplx61/1Running08m26santix@antix:  kubectl -n longhorn-system get pod NAME READY STATUS RESTARTS AGE longhorn-driver-deployer-7f95558b85-wflrt 0/1 Init:0/1 0 7m44s longhorn-driver-deployer-7f95558b85-ztg2l 0/1 Init:Error 0 11m longhorn-manager-8tvkr 0/2 Evicted 0 3m2s longhorn-manager-b9w4d 0/2 Evicted 0 3m2s longhorn-ui-7ff79dfb4-5rvl7 0/1 Error 0 11m longhorn-ui-7ff79dfb4-7scm4 0/1 Error 0 11m longhorn-ui-7ff79dfb4-9mrgh 1/1 Running 1 (21s ago) 7m57s longhorn-ui-7ff79dfb4-fplx6 1/1 Running 0 8m26s antix@antix:~

  4. Assistant

    Thanks for the update!

    Right now you have:

    • 1 longhorn-driver-deployer pod stuck in Init:Error
    • 2 longhorn-manager pods Evicted
    • 2 longhorn-ui pods in Error
    • 2 longhorn-ui pods are running fine

    Next steps for troubleshooting

    1. Check why longhorn-driver-deployer-7f95558b85-ztg2l has Init:Error

    Run:

    kubectl -n longhorn-system describe pod longhorn-driver-deployer-7f95558b85-ztg2l
    

    Look under Events for errors during the init container startup.

    Also check init container logs (replace init container name if different):

    kubectl -n longhorn-system logs longhorn-driver-deployer-7f95558b85-ztg2l -c <init-container-name>
    

    Usually, Longhorn uses an init container called init.

    This will give precise error info.

    2. Investigate Evicted longhorn-manager pods

    Check why pods were evicted by describing pod:

    kubectl -n longhorn-system describe pod longhorn-manager-8tvkr
    kubectl -n longhorn-system describe pod longhorn-manager-b9w4d
    

    You should find something like:

    Reason: Evicted
    Message: The node was low on resource: [MemoryPressure/DiskPressure].
    

    If you see resource pressure on nodes:

    • Check node status:
    kubectl describe node <node-name>
    
    • Check disk space on nodes:
    kubectl ssh <node>  # Or use your method to ssh
    df -h
    free -m
    
    • Free up disk or memory on nodes, or increase resources.

    3. Check Error longhorn-ui pods

    Describe one of the erroring UI pods:

    kubectl -n longhorn-system describe pod longhorn-ui-7ff79dfb4-5rvl7
    

    Then look at logs:

    kubectl -n longhorn-system logs longhorn-ui-7ff79dfb4-5rvl7
    

    This may show why UI container is failing (misconfiguration, image problems, or crashed process).


    General advice on resources:

    Longhorn is storage-intensive and memory-consuming. The evicted pods suggest your nodes do not have sufficient disk or memory resources.

    Check node health and free disk:

    1. Disk:

      • Longhorn requires disks with enough free space.
      • If /var/lib/longhorn or related mount points are full, Longhorn pods may evict.
    2. Memory:

      • If node faces memory pressure, kubelet evicts pods to relieve memory.

    Summary

    1. Describe problematic pods (kubectl describe pod ...) → get event reasons.
    2. Check init container logs for deployer pod.
    3. Check node disk/memory stats.
    4. Free up resources or add more node capacity.

    If you share output from:

    • kubectl describe pod longhorn-driver-deployer-7f95558b85-ztg2l -n longhorn-system
    • kubectl describe pod longhorn-manager-8tvkr -n longhorn-system
    • logs of init container and UI pods

    I can help analyze further.

  5. User

    longhorn-system 22m Warning FailedDaemonPod daemonset/longhorn-manager (combined from similar events): Found failed daemon pod longhorn-system/longhorn-manager-w9nh6 on node worker2, will try to kill it longhorn-system 22m Normal SuccessfulDelete daemonset/longhorn-manager (combined from similar events): Deleted pod: longhorn-manager-w9nh6 longhorn-system 13m Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-zjcw2 longhorn-system 13m Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-69hm8 longhorn-system 9m49s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-zjcw2 on node worker1, will try to kill it longhorn-system 9m49s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-zjcw2 longhorn-system 9m48s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-pbbr8 longhorn-system 9m48s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-pbbr8 on node worker1, will try to kill it longhorn-system 9m48s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-pbbr8 longhorn-system 9m47s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-g5nwb longhorn-system 9m45s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-g5nwb on node worker1, will try to kill it longhorn-system 9m45s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-g5nwb longhorn-system 9m45s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-9rhd7 longhorn-system 9m40s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-9rhd7 on node worker1, will try to kill it longhorn-system 9m40s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-9rhd7 longhorn-system 9m40s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-x6gdf longhorn-system 9m32s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-x6gdf on node worker1, will try to kill it longhorn-system 9m30s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-x6gdf longhorn-system 9m29s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-stpbl longhorn-system 9m22s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-69hm8 on node worker2, will try to kill it longhorn-system 9m22s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-69hm8 longhorn-system 9m22s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-psjmc longhorn-system 9m21s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-psjmc on node worker2, will try to kill it longhorn-system 9m21s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-psjmc longhorn-system 9m21s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-gqjwg longhorn-system 9m18s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-gqjwg on node worker2, will try to kill it longhorn-system 9m18s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-gqjwg longhorn-system 9m4s Normal SuccessfulCreate daemonset/longhorn-manager (combined from similar events): Created pod: longhorn-manager-fbnmk longhorn-system 9m14s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-stpbl on node worker1, will try to kill it longhorn-system 7m10s Warning FailedDaemonPod daemonset/longhorn-manager (combined from similar events): Found failed daemon pod longhorn-system/longhorn-manager-jn47p on node worker2, will try to kill it longhorn-system 9m13s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-stpbl longhorn-system 7m9s Normal SuccessfulDelete daemonset/longhorn-manager (combined from similar events): Deleted pod: longhorn-manager-jn47p longhorn-system 13m Normal Scheduled pod/longhorn-ui-7ff79dfb4-4mdp2 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-4mdp2 to worker2 longhorn-system 13m Normal Pulling pod/longhorn-ui-7ff79dfb4-4mdp2 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 10m Warning Evicted pod/longhorn-ui-7ff79dfb4-4mdp2 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 428128Ki. longhorn-system 10m Warning ExceededGracePeriod pod/longhorn-ui-7ff79dfb4-4mdp2 Container runtime did not kill the pod within specified grace period. longhorn-system 10m Normal Pulled pod/longhorn-ui-7ff79dfb4-4mdp2 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 40.674s (3m0.765s including waiting). Image size: 73388900 bytes. longhorn-system 10m Normal Created pod/longhorn-ui-7ff79dfb4-4mdp2 Created container: longhorn-ui longhorn-system 10m Normal Started pod/longhorn-ui-7ff79dfb4-4mdp2 Started container longhorn-ui longhorn-system 9m59s Normal Killing pod/longhorn-ui-7ff79dfb4-4mdp2 Stopping container longhorn-ui longhorn-system 43m Normal Scheduled pod/longhorn-ui-7ff79dfb4-5rvl7 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-5rvl7 to worker2 longhorn-system 43m Normal Pulling pod/longhorn-ui-7ff79dfb4-5rvl7 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 42m Warning Evicted pod/longhorn-ui-7ff79dfb4-5rvl7 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 575112Ki. longhorn-system 41m Warning ExceededGracePeriod pod/longhorn-ui-7ff79dfb4-5rvl7 Container runtime did not kill the pod within specified grace period. longhorn-system 41m Warning Evicted pod/longhorn-ui-7ff79dfb4-5rvl7 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 334952Ki. longhorn-system 41m Normal Pulled pod/longhorn-ui-7ff79dfb4-5rvl7 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 30.658s (2m32.851s including waiting). Image size: 73388900 bytes. longhorn-system 41m Normal Created pod/longhorn-ui-7ff79dfb4-5rvl7 Created container: longhorn-ui longhorn-system 41m Normal Started pod/longhorn-ui-7ff79dfb4-5rvl7 Started container longhorn-ui longhorn-system 41m Normal Killing pod/longhorn-ui-7ff79dfb4-5rvl7 Stopping container longhorn-ui longhorn-system 31m Normal Scheduled pod/longhorn-ui-7ff79dfb4-5sbrt Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-5sbrt to worker1 longhorn-system 31m Normal Pulled pod/longhorn-ui-7ff79dfb4-5sbrt Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 31m Normal Created pod/longhorn-ui-7ff79dfb4-5sbrt Created container: longhorn-ui longhorn-system 31m Normal Started pod/longhorn-ui-7ff79dfb4-5sbrt Started container longhorn-ui longhorn-system 29m Warning Evicted pod/longhorn-ui-7ff79dfb4-5sbrt The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 211616Ki. Container longhorn-ui was using 44Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 29m Normal Killing pod/longhorn-ui-7ff79dfb4-5sbrt Stopping container longhorn-ui longhorn-system 31m Normal Scheduled pod/longhorn-ui-7ff79dfb4-6njj2 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-6njj2 to worker1 longhorn-system 31m Normal Pulled pod/longhorn-ui-7ff79dfb4-6njj2 Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 31m Normal Created pod/longhorn-ui-7ff79dfb4-6njj2 Created container: longhorn-ui longhorn-system 31m Normal Started pod/longhorn-ui-7ff79dfb4-6njj2 Started container longhorn-ui longhorn-system 29m Warning Evicted pod/longhorn-ui-7ff79dfb4-6njj2 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 218544Ki. Container longhorn-ui was using 44Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 29m Normal Killing pod/longhorn-ui-7ff79dfb4-6njj2 Stopping container longhorn-ui longhorn-system 43m Normal Scheduled pod/longhorn-ui-7ff79dfb4-7scm4 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-7scm4 to worker1 longhorn-system 43m Normal Pulling pod/longhorn-ui-7ff79dfb4-7scm4 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 41m Warning Evicted pod/longhorn-ui-7ff79dfb4-7scm4 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 333748Ki. longhorn-system 40m Warning ExceededGracePeriod pod/longhorn-ui-7ff79dfb4-7scm4 Container runtime did not kill the pod within specified grace period. longhorn-system 40m Normal Pulled pod/longhorn-ui-7ff79dfb4-7scm4 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 34.115s (3m5.186s including waiting). Image size: 73388900 bytes. longhorn-system 40m Normal Created pod/longhorn-ui-7ff79dfb4-7scm4 Created container: longhorn-ui longhorn-system 40m Normal Started pod/longhorn-ui-7ff79dfb4-7scm4 Started container longhorn-ui longhorn-system 40m Normal Killing pod/longhorn-ui-7ff79dfb4-7scm4 Stopping container longhorn-ui longhorn-system 20m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-7tqgx 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 9m54s Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-8h2ng 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 13m Normal Scheduled pod/longhorn-ui-7ff79dfb4-8xtr9 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-8xtr9 to worker1 longhorn-system 13m Normal Pulling pod/longhorn-ui-7ff79dfb4-8xtr9 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 10m Warning Evicted pod/longhorn-ui-7ff79dfb4-8xtr9 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 589436Ki. longhorn-system 10m Warning ExceededGracePeriod pod/longhorn-ui-7ff79dfb4-8xtr9 Container runtime did not kill the pod within specified grace period. longhorn-system 10m Normal Pulled pod/longhorn-ui-7ff79dfb4-8xtr9 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 25.594s (2m22.686s including waiting). Image size: 73388900 bytes. longhorn-system 10m Normal Created pod/longhorn-ui-7ff79dfb4-8xtr9 Created container: longhorn-ui longhorn-system 10m Normal Started pod/longhorn-ui-7ff79dfb4-8xtr9 Started container longhorn-ui longhorn-system 10m Normal Killing pod/longhorn-ui-7ff79dfb4-8xtr9 Stopping container longhorn-ui longhorn-system 40m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-9mrgh 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 35m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-9mrgh 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 34m Normal Scheduled pod/longhorn-ui-7ff79dfb4-9mrgh Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-9mrgh to worker2 longhorn-system 34m Normal Pulling pod/longhorn-ui-7ff79dfb4-9mrgh Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 33m Normal Pulled pod/longhorn-ui-7ff79dfb4-9mrgh Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 1m3.141s (1m3.141s including waiting). Image size: 73388900 bytes. longhorn-system 32m Normal Created pod/longhorn-ui-7ff79dfb4-9mrgh Created container: longhorn-ui longhorn-system 32m Normal Started pod/longhorn-ui-7ff79dfb4-9mrgh Started container longhorn-ui longhorn-system 32m Normal Pulled pod/longhorn-ui-7ff79dfb4-9mrgh Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 31m Warning Evicted pod/longhorn-ui-7ff79dfb4-9mrgh The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 311140Ki. Container longhorn-ui was using 48Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 31m Normal Killing pod/longhorn-ui-7ff79dfb4-9mrgh Stopping container longhorn-ui longhorn-system 29m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-b2tlz 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 25m Normal Scheduled pod/longhorn-ui-7ff79dfb4-b2tlz Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-b2tlz to worker2 longhorn-system 25m Normal Pulling pod/longhorn-ui-7ff79dfb4-b2tlz Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 24m Normal Pulled pod/longhorn-ui-7ff79dfb4-b2tlz Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 1.79s (51.711s including waiting). Image size: 73388900 bytes. longhorn-system 23m Normal Created pod/longhorn-ui-7ff79dfb4-b2tlz Created container: longhorn-ui longhorn-system 23m Normal Started pod/longhorn-ui-7ff79dfb4-b2tlz Started container longhorn-ui longhorn-system 23m Normal Pulled pod/longhorn-ui-7ff79dfb4-b2tlz Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 23m Warning BackOff pod/longhorn-ui-7ff79dfb4-b2tlz Back-off restarting failed container longhorn-ui in pod longhorn-ui-7ff79dfb4-b2tlz_longhorn-system(fb7c99f7-1be8-4878-aeb1-fd871fa3f67d) longhorn-system 22m Warning Evicted pod/longhorn-ui-7ff79dfb4-b2tlz The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 310488Ki. Container longhorn-ui was using 48Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 22m Normal Killing pod/longhorn-ui-7ff79dfb4-b2tlz Stopping container longhorn-ui longhorn-system 41m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-fplx6 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 35m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-fplx6 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 34m Normal Scheduled pod/longhorn-ui-7ff79dfb4-fplx6 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-fplx6 to worker2 longhorn-system 34m Normal Pulling pod/longhorn-ui-7ff79dfb4-fplx6 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 33m Normal Pulled pod/longhorn-ui-7ff79dfb4-fplx6 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 1.696s (1m20.381s including waiting). Image size: 73388900 bytes. longhorn-system 33m Normal Created pod/longhorn-ui-7ff79dfb4-fplx6 Created container: longhorn-ui longhorn-system 33m Normal Started pod/longhorn-ui-7ff79dfb4-fplx6 Started container longhorn-ui longhorn-system 31m Warning Evicted pod/longhorn-ui-7ff79dfb4-fplx6 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 311568Ki. Container longhorn-ui was using 44Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 31m Normal Killing pod/longhorn-ui-7ff79dfb4-fplx6 Stopping container longhorn-ui longhorn-system 10m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-gxm47 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 20m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-lwkqz 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 22m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-qp8vp 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 22m Normal Scheduled pod/longhorn-ui-7ff79dfb4-qp8vp Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-qp8vp to worker1 longhorn-system 22m Normal Pulling pod/longhorn-ui-7ff79dfb4-qp8vp Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 21m Normal Pulled pod/longhorn-ui-7ff79dfb4-qp8vp Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 21.987s (21.987s including waiting). Image size: 73388900 bytes. longhorn-system 20m Normal Created pod/longhorn-ui-7ff79dfb4-qp8vp Created container: longhorn-ui longhorn-system 20m Normal Started pod/longhorn-ui-7ff79dfb4-qp8vp Started container longhorn-ui longhorn-system 20m Normal Pulled pod/longhorn-ui-7ff79dfb4-qp8vp Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 21m Warning BackOff pod/longhorn-ui-7ff79dfb4-qp8vp Back-off restarting failed container longhorn-ui in pod longhorn-ui-7ff79dfb4-qp8vp_longhorn-system(bbeae2ed-0157-4623-a28e-9738e93e4f40) longhorn-system 20m Warning Evicted pod/longhorn-ui-7ff79dfb4-qp8vp The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 393956Ki. Container longhorn-ui was using 48Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 20m Normal Killing pod/longhorn-ui-7ff79dfb4-qp8vp Stopping container longhorn-ui longhorn-system 29m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-v4qt2 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 25m Normal Scheduled pod/longhorn-ui-7ff79dfb4-v4qt2 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-v4qt2 to worker2 longhorn-system 25m Normal Pulling pod/longhorn-ui-7ff79dfb4-v4qt2 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 24m Normal Pulled pod/longhorn-ui-7ff79dfb4-v4qt2 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 50.205s (50.205s including waiting). Image size: 73388900 bytes. longhorn-system 23m Normal Created pod/longhorn-ui-7ff79dfb4-v4qt2 Created container: longhorn-ui longhorn-system 23m Normal Started pod/longhorn-ui-7ff79dfb4-v4qt2 Started container longhorn-ui longhorn-system 23m Normal Pulled pod/longhorn-ui-7ff79dfb4-v4qt2 Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 23m Warning BackOff pod/longhorn-ui-7ff79dfb4-v4qt2 Back-off restarting failed container longhorn-ui in pod longhorn-ui-7ff79dfb4-v4qt2_longhorn-system(78686ea7-8c8d-4123-9964-3a1c2c362139) longhorn-system 22m Warning Evicted pod/longhorn-ui-7ff79dfb4-v4qt2 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 311048Ki. Container longhorn-ui was using 48Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 22m Normal Killing pod/longhorn-ui-7ff79dfb4-v4qt2 Stopping container longhorn-ui longhorn-system 22m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-z9rtp 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 22m Normal Scheduled pod/longhorn-ui-7ff79dfb4-z9rtp Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-z9rtp to worker1 longhorn-system 22m Normal Pulling pod/longhorn-ui-7ff79dfb4-z9rtp Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 21m Normal Pulled pod/longhorn-ui-7ff79dfb4-z9rtp Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 1.837s (22.578s including waiting). Image size: 73388900 bytes. longhorn-system 20m Normal Created pod/longhorn-ui-7ff79dfb4-z9rtp Created container: longhorn-ui longhorn-system 20m Normal Started pod/longhorn-ui-7ff79dfb4-z9rtp Started container longhorn-ui longhorn-system 20m Normal Pulled pod/longhorn-ui-7ff79dfb4-z9rtp Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 21m Warning BackOff pod/longhorn-ui-7ff79dfb4-z9rtp Back-off restarting failed container longhorn-ui in pod longhorn-ui-7ff79dfb4-z9rtp_longhorn-system(4b243f8d-c00f-41ac-81b3-78afb9e66a48) longhorn-system 20m Warning Evicted pod/longhorn-ui-7ff79dfb4-z9rtp The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 394216Ki. Container longhorn-ui was using 48Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 20m Normal Killing pod/longhorn-ui-7ff79dfb4-z9rtp Stopping container longhorn-ui longhorn-system 43m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-5rvl7 longhorn-system 43m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-7scm4 longhorn-system 41m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-fplx6 longhorn-system 40m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-9mrgh longhorn-system 31m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-6njj2 longhorn-system 31m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-5sbrt longhorn-system 29m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-v4qt2 longhorn-system 29m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-b2tlz longhorn-system 22m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-qp8vp longhorn-system 20m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 (combined from similar events): Created pod: longhorn-ui-7ff79dfb4-lwkqz longhorn-system 13m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-8xtr9 longhorn-system 13m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-4mdp2 longhorn-system 10m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-gxm47 longhorn-system 9m54s Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-8h2ng longhorn-system 43m Normal ScalingReplicaSet deployment/longhorn-ui Scaled up replica set longhorn-ui-7ff79dfb4 from 0 to 2 longhorn-system 13m Normal ScalingReplicaSet deployment/longhorn-ui Scaled up replica set longhorn-ui-7ff79dfb4 from 0 to 2 longhorn-system 22m Warning FailedScheduling pod/longhorn-uninstall-jfqxc 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 22m Normal Scheduled pod/longhorn-uninstall-jfqxc Successfully assigned longhorn-system/longhorn-uninstall-jfqxc to worker1 longhorn-system 22m Normal Pulling pod/longhorn-uninstall-jfqxc Pulling image "longhornio/longhorn-manager:v1.8.1" longhorn-system 20m Normal Pulled pod/longhorn-uninstall-jfqxc Successfully pulled image "longhornio/longhorn-manager:v1.8.1" in 1.927s (1m25.744s including waiting). Image size: 119475837 bytes. longhorn-system 20m Normal Created pod/longhorn-uninstall-jfqxc Created container: longhorn-uninstall longhorn-system 20m Normal Started pod/longhorn-uninstall-jfqxc Started container longhorn-uninstall longhorn-system 24m Normal Scheduled pod/longhorn-uninstall-mbrvt Successfully assigned longhorn-system/longhorn-uninstall-mbrvt to worker2 longhorn-system 24m Normal Pulling pod/longhorn-uninstall-mbrvt Pulling image "longhornio/longhorn-manager:v1.8.1" longhorn-system 22m Normal Pulled pod/longhorn-uninstall-mbrvt Successfully pulled image "longhornio/longhorn-manager:v1.8.1" in 2.745s (1m8.689s including waiting). Image size: 119475837 bytes. longhorn-system 22m Normal Created pod/longhorn-uninstall-mbrvt Created container: longhorn-uninstall longhorn-system 22m Normal Started pod/longhorn-uninstall-mbrvt Started container longhorn-uninstall longhorn-system 22m Warning Evicted pod/longhorn-uninstall-mbrvt The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 310672Ki. longhorn-system 22m Warning ExceededGracePeriod pod/longhorn-uninstall-mbrvt Container runtime did not kill the pod within specified grace period. longhorn-system 24m Normal SuccessfulCreate job/longhorn-uninstall Created pod: longhorn-uninstall-mbrvt longhorn-system 22m Normal SuccessfulCreate job/longhorn-uninstall Created pod: longhorn-uninstall-jfqxc longhorn-system 20m Warning BackoffLimitExceeded job/longhorn-uninstall Job has reached the specified backoff limit metallb-system 20m Warning FailedScheduling pod/controller-bb5f47665-9wvd9 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 15m Normal Scheduled pod/controller-bb5f47665-9wvd9 Successfully assigned metallb-system/controller-bb5f47665-9wvd9 to worker2 metallb-system 15m Normal Pulling pod/controller-bb5f47665-9wvd9 Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 15m Normal Pulled pod/controller-bb5f47665-9wvd9 Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 24.231s (24.231s including waiting). Image size: 29163511 bytes. metallb-system 15m Normal Created pod/controller-bb5f47665-9wvd9 Created container: controller metallb-system 15m Normal Started pod/controller-bb5f47665-9wvd9 Started container controller metallb-system 9m57s Warning Evicted pod/controller-bb5f47665-9wvd9 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 310612Ki. Container controller was using 84Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 9m57s Normal Killing pod/controller-bb5f47665-9wvd9 Stopping container controller metallb-system 9m54s Warning FailedScheduling pod/controller-bb5f47665-bmx9v 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 4m52s Warning FailedScheduling pod/controller-bb5f47665-bmx9v 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 3m35s Normal Scheduled pod/controller-bb5f47665-bmx9v Successfully assigned metallb-system/controller-bb5f47665-bmx9v to worker1 metallb-system 3m32s Normal Pulling pod/controller-bb5f47665-bmx9v Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 2m50s Normal Pulled pod/controller-bb5f47665-bmx9v Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 24.545s (41.673s including waiting). Image size: 29163511 bytes. metallb-system 2m50s Normal Created pod/controller-bb5f47665-bmx9v Created container: controller metallb-system 2m49s Normal Started pod/controller-bb5f47665-bmx9v Started container controller metallb-system 31m Normal Scheduled pod/controller-bb5f47665-h7t84 Successfully assigned metallb-system/controller-bb5f47665-h7t84 to worker1 metallb-system 31m Normal Pulling pod/controller-bb5f47665-h7t84 Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 30m Warning Evicted pod/controller-bb5f47665-h7t84 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 592432Ki. metallb-system 30m Warning ExceededGracePeriod pod/controller-bb5f47665-h7t84 Container runtime did not kill the pod within specified grace period. metallb-system 30m Warning Evicted pod/controller-bb5f47665-h7t84 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 390424Ki. metallb-system 29m Normal Pulled pod/controller-bb5f47665-h7t84 Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 19.988s (1m42.564s including waiting). Image size: 29163511 bytes. metallb-system 29m Normal Created pod/controller-bb5f47665-h7t84 Created container: controller metallb-system 29m Normal Started pod/controller-bb5f47665-h7t84 Started container controller metallb-system 29m Normal Killing pod/controller-bb5f47665-h7t84 Stopping container controller metallb-system 29m Warning FailedScheduling pod/controller-bb5f47665-pcnsj 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 25m Normal Scheduled pod/controller-bb5f47665-pcnsj Successfully assigned metallb-system/controller-bb5f47665-pcnsj to worker2 metallb-system 25m Normal Pulling pod/controller-bb5f47665-pcnsj Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 23m Warning Evicted pod/controller-bb5f47665-pcnsj The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 438304Ki. metallb-system 23m Warning ExceededGracePeriod pod/controller-bb5f47665-pcnsj Container runtime did not kill the pod within specified grace period. metallb-system 22m Normal Pulled pod/controller-bb5f47665-pcnsj Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 21.521s (2m19.598s including waiting). Image size: 29163511 bytes. metallb-system 22m Normal Created pod/controller-bb5f47665-pcnsj Created container: controller metallb-system 22m Normal Started pod/controller-bb5f47665-pcnsj Started container controller metallb-system 22m Normal Killing pod/controller-bb5f47665-pcnsj Stopping container controller metallb-system 22m Warning FailedScheduling pod/controller-bb5f47665-rwlqr 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 22m Normal Scheduled pod/controller-bb5f47665-rwlqr Successfully assigned metallb-system/controller-bb5f47665-rwlqr to worker1 metallb-system 22m Normal Pulling pod/controller-bb5f47665-rwlqr Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 20m Warning Evicted pod/controller-bb5f47665-rwlqr The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 494544Ki. metallb-system 20m Warning ExceededGracePeriod pod/controller-bb5f47665-rwlqr Container runtime did not kill the pod within specified grace period. metallb-system 20m Normal Pulled pod/controller-bb5f47665-rwlqr Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 24.874s (1m50.589s including waiting). Image size: 29163511 bytes. metallb-system 20m Normal Created pod/controller-bb5f47665-rwlqr Created container: controller metallb-system 20m Normal Started pod/controller-bb5f47665-rwlqr Started container controller metallb-system 20m Normal Killing pod/controller-bb5f47665-rwlqr Stopping container controller metallb-system 40m Warning Evicted pod/controller-bb5f47665-vlj95 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 219300Ki. Container controller was using 96Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 40m Normal Killing pod/controller-bb5f47665-vlj95 Stopping container controller metallb-system 40m Warning FailedScheduling pod/controller-bb5f47665-vqqz2 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 35m Warning FailedScheduling pod/controller-bb5f47665-vqqz2 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 34m Normal Scheduled pod/controller-bb5f47665-vqqz2 Successfully assigned metallb-system/controller-bb5f47665-vqqz2 to worker2 metallb-system 34m Normal Pulling pod/controller-bb5f47665-vqqz2 Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 32m Normal Pulled pod/controller-bb5f47665-vqqz2 Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 22.913s (1m43.205s including waiting). Image size: 29163511 bytes. metallb-system 32m Normal Created pod/controller-bb5f47665-vqqz2 Created container: controller metallb-system 32m Normal Started pod/controller-bb5f47665-vqqz2 Started container controller metallb-system 31m Warning Evicted pod/controller-bb5f47665-vqqz2 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 310980Ki. Container controller was using 80Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 31m Normal Killing pod/controller-bb5f47665-vqqz2 Stopping container controller metallb-system 40m Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-vqqz2 metallb-system 31m Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-h7t84 metallb-system 29m Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-pcnsj metallb-system 22m Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-rwlqr metallb-system 20m Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-9wvd9 metallb-system 9m54s Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-bmx9v metallb-system 40m Normal Scheduled pod/speaker-9nqxs Successfully assigned metallb-system/speaker-9nqxs to worker1 metallb-system 40m Warning Evicted pod/speaker-9nqxs The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-9wwmv Successfully assigned metallb-system/speaker-9wwmv to worker1 metallb-system 40m Warning Evicted pod/speaker-9wwmv The node had condition: [DiskPressure]. metallb-system 8m2s Normal Scheduled pod/speaker-cdt7b Successfully assigned metallb-system/speaker-cdt7b to worker2 metallb-system 8m2s Warning Evicted pod/speaker-cdt7b The node had condition: [DiskPressure]. metallb-system 35m Normal Scheduled pod/speaker-ckcjq Successfully assigned metallb-system/speaker-ckcjq to worker2 metallb-system 35m Warning Evicted pod/speaker-ckcjq The node had condition: [DiskPressure]. metallb-system 23m Normal Scheduled pod/speaker-dmsmj Successfully assigned metallb-system/speaker-dmsmj to worker1 metallb-system 23m Warning Evicted pod/speaker-dmsmj The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-fvn6g Successfully assigned metallb-system/speaker-fvn6g to worker2 metallb-system 40m Warning Evicted pod/speaker-fvn6g The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-gvx65 Successfully assigned metallb-system/speaker-gvx65 to worker2 metallb-system 40m Warning Evicted pod/speaker-gvx65 The node had condition: [DiskPressure]. metallb-system 38m Normal Scheduled pod/speaker-hdhbl Successfully assigned metallb-system/speaker-hdhbl to worker2 metallb-system 38m Warning Evicted pod/speaker-hdhbl The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-j7klj Successfully assigned metallb-system/speaker-j7klj to worker2 metallb-system 40m Warning Evicted pod/speaker-j7klj The node had condition: [DiskPressure]. metallb-system 23m Normal Scheduled pod/speaker-jb9q6 Successfully assigned metallb-system/speaker-jb9q6 to worker2 metallb-system 23m Warning Evicted pod/speaker-jb9q6 The node had condition: [DiskPressure]. metallb-system 31m Normal Scheduled pod/speaker-l8ft6 Successfully assigned metallb-system/speaker-l8ft6 to worker1 metallb-system 31m Normal Pulling pod/speaker-l8ft6 Pulling image "quay.io/metallb/speaker:v0.14.9" metallb-system 30m Normal Pulled pod/speaker-l8ft6 Successfully pulled image "quay.io/metallb/speaker:v0.14.9" in 39.391s (39.391s including waiting). Image size: 55246333 bytes. metallb-system 30m Normal Created pod/speaker-l8ft6 Created container: speaker metallb-system 30m Normal Started pod/speaker-l8ft6 Started container speaker metallb-system 29m Warning Evicted pod/speaker-l8ft6 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 218324Ki. Container speaker was using 76Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 29m Normal Killing pod/speaker-l8ft6 Stopping container speaker metallb-system 31m Normal Scheduled pod/speaker-l9xcz Successfully assigned metallb-system/speaker-l9xcz to worker2 metallb-system 31m Warning Evicted pod/speaker-l9xcz The node had condition: [DiskPressure]. metallb-system 40m Warning Evicted pod/speaker-lbprs The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-lbprs Successfully assigned metallb-system/speaker-lbprs to worker2 metallb-system 40m Normal Scheduled pod/speaker-mzq9b Successfully assigned metallb-system/speaker-mzq9b to worker1 metallb-system 40m Warning Evicted pod/speaker-mzq9b The node had condition: [DiskPressure]. metallb-system 39m Warning Evicted pod/speaker-q4jh5 The node had condition: [DiskPressure]. metallb-system 39m Normal Scheduled pod/speaker-q4jh5 Successfully assigned metallb-system/speaker-q4jh5 to worker1 metallb-system 8m2s Normal Scheduled pod/speaker-q4v6n Successfully assigned metallb-system/speaker-q4v6n to worker1 metallb-system 8m2s Warning Evicted pod/speaker-q4v6n The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-qjvgb Successfully assigned metallb-system/speaker-qjvgb to worker1 metallb-system 40m Warning Evicted pod/speaker-qjvgb The node had condition: [DiskPressure]. metallb-system 35m Normal Scheduled pod/speaker-r2rsz Successfully assigned metallb-system/speaker-r2rsz to worker1 metallb-system 35m Warning Evicted pod/speaker-r2rsz The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-rg45w Successfully assigned metallb-system/speaker-rg45w to worker2 metallb-system 40m Warning Evicted pod/speaker-rg45w The node had condition: [DiskPressure]. metallb-system 40m Warning Evicted pod/speaker-rzcs7 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 217284Ki. Container speaker was using 80Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 40m Normal Killing pod/speaker-rzcs7 Stopping container speaker metallb-system 40m Normal Scheduled pod/speaker-t457m Successfully assigned metallb-system/speaker-t457m to worker2 metallb-system 40m Warning Evicted pod/speaker-t457m The node had condition: [DiskPressure]. metallb-system 39m Normal Scheduled pod/speaker-v9cmq Successfully assigned metallb-system/speaker-v9cmq to worker2 metallb-system 39m Warning Evicted pod/speaker-v9cmq The node had condition: [DiskPressure]. metallb-system 29m Normal Scheduled pod/speaker-vr2gt Successfully assigned metallb-system/speaker-vr2gt to worker1 metallb-system 29m Warning Evicted pod/speaker-vr2gt The node had condition: [DiskPressure]. metallb-system 38m Normal Scheduled pod/speaker-wkqmz Successfully assigned metallb-system/speaker-wkqmz to worker1 metallb-system 38m Warning Evicted pod/speaker-wkqmz The node had condition: [DiskPressure]. metallb-system 39m Normal Scheduled pod/speaker-wrp7x Successfully assigned metallb-system/speaker-wrp7x to worker1 metallb-system 39m Warning Evicted pod/speaker-wrp7x The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-xhjvq Successfully assigned metallb-system/speaker-xhjvq to worker1 metallb-system 40m Warning Evicted pod/speaker-xhjvq The node had condition: [DiskPressure]. metallb-system 41m Warning Evicted pod/speaker-z7bpd The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 230352Ki. Container speaker was using 80Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 41m Normal Killing pod/speaker-z7bpd Stopping container speaker metallb-system 40m Normal Scheduled pod/speaker-z9slq Successfully assigned metallb-system/speaker-z9slq to worker2 metallb-system 40m Warning Evicted pod/speaker-z9slq The node had condition: [DiskPressure]. metallb-system 40m Normal SucceededDaemonPod daemonset/speaker Found succeeded daemon pod metallb-system/speaker-z7bpd on node worker2, will try to delete it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-z7bpd metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-lbprs metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-lbprs on node worker2, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-lbprs metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-z9slq metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-z9slq on node worker2, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-z9slq metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-t457m metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-t457m on node worker2, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-t457m metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-j7klj metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-j7klj on node worker2, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-j7klj metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-rg45w metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-rg45w on node worker2, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-rg45w metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-gvx65 metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-rzcs7 on node worker1, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-rzcs7 metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-mzq9b metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-mzq9b on node worker1, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-mzq9b metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-qjvgb metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-qjvgb on node worker1, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-qjvgb metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-9nqxs metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-9nqxs on node worker1, will try to kill it metallb-system 23m Warning FailedDaemonPod daemonset/speaker (combined from similar events): Found failed daemon pod metallb-system/speaker-vr2gt on node worker1, will try to kill it metallb-system 23m Normal SuccessfulDelete daemonset/speaker (combined from similar events): Deleted pod: speaker-vr2gt metallb-system 40m Normal SuccessfulCreate daemonset/speaker (combined from similar events): Created pod: speaker-xhjvq metallb-system 29m Normal SucceededDaemonPod daemonset/speaker Found succeeded daemon pod metallb-system/speaker-l8ft6 on node worker1, will try to delete it metallb-system 8m2s Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-jb9q6 on node worker2, will try to kill it metallb-system 8m2s Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-dmsmj on node worker1, will try to kill it metallb-system 8m2s Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-jb9q6 metallb-system 8m2s Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-dmsmj metallb-system 8m2s Normal SuccessfulCreate daemonset/speaker Created pod: speaker-q4v6n antix@antix:~$

  6. Assistant

    The events you've posted clearly indicate:

    Root Cause

    • Your Kubernetes nodes are under disk pressure, specifically on ephemeral-storage.
    • Many pods (Longhorn's longhorn-manager, longhorn-ui, longhorn-uninstall, as well as MetalLB pods) are repeatedly getting evicted due to insufficient ephemeral storage.
    • Nodes have the taint node.kubernetes.io/disk-pressure which prevents pods from scheduling on them unless they tolerate that taint.
    • Pods also experience FailedDaemonPod and get continuously killed and recreated due to the underlying node resource issues.

    What is ephemeral-storage pressure?

    • Ephemeral storage is the node’s local available disk space used for container writable layers, logs, and emptyDir volumes.
    • When it runs low, kubelet taints the node with node.kubernetes.io/disk-pressure and evicts pods to free space.
    • Pods without proper tolerations and resource requests get unscheduled or rejected.

    How to Fix This

    1. Free up disk space on your nodes

    • SSH to each node:
    ssh user@worker1
    ssh user@worker2
    
    • Check disk usage:
    df -h
    
    • Identify which partitions are almost full, especially /var/lib/kubelet, /var/lib/docker, /var/lib/containerd, and /tmp

    • Remove unused images and containers:

    For Docker:

    docker system prune -af
    

    For containerd:

    crictl images            # List images
    crictl rmi <imageID>     # Remove unused images
    
    • Check for large log files and clean them up (e.g., /var/log/)

    2. Add more ephemeral storage to nodes or increase PV sizes if using hostPath based storage

    • If freeing disk space is not enough, consider expanding the disk size or adding additional storage to your worker nodes.

    3. Configure resource requests and limits for Longhorn pods (especially ephemeral-storage)

    • This helps Kubernetes properly schedule pods based on real resource needs.
    • Check your Longhorn deployment manifests and add requests/limits for ephemeral-storage.

    Example for a container spec:

    resources:
      requests:
        ephemeral-storage: "1Gi"
      limits:
        ephemeral-storage: "2Gi"
    

    4. Remove /taints or add tolerations as a short-term workaround (not recommended)

    • You could remove the node.kubernetes.io/disk-pressure taint with:
    kubectl taint nodes <node-name> node.kubernetes.io/disk-pressure:NoSchedule-
    
    • However, this will make pods run on nodes that are disk pressure-affected, and they will likely be evicted immediately again unless disk pressure is relieved.

    5. Monitor disk usage continuously

    Install tools like node-problem-detector, or monitor disk space with your favorite monitoring stack (Prometheus, Grafana).


    Summary

    Your Longhorn and other pods failing/evicting is due to nodes running low on ephemeral-storage, causing Kubernetes to taint the nodes with node.kubernetes.io/disk-pressure. Pods cannot be scheduled or are evicted when running on these nodes.

    You need to:

    • Free up disk space on your nodes
    • Add tolerations if needed (temporarily)
    • Configure Longhorn pods with proper ephemeral-storage requests/limits
    • Possibly expand the nodes' disk capacity

    If you want, I can also help with commands to check disk usage or cleaning up the nodes. Just ask!

  7. User

    antix@antix:~/k0s$ k0sctl apply --config k0s.yaml

    ⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███ ⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███ ███ ███ ⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███ ███ ███ ⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███ ███ ███ ⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████ ███ ██████████ k0sctl v0.23.0 Copyright 2023, k0sctl authors. By continuing to use k0sctl you agree to these terms: https://k0sproject.io/licenses/eula INFO ==> Running phase: Set k0s version
    INFO Looking up latest stable k0s version
    INFO Using k0s version v1.32.4+k0s.0
    INFO ==> Running phase: Connect to hosts INFO [ssh] 10.203.173.30:22: connected
    INFO [ssh] 10.203.173.74:22: connected
    INFO [ssh] 10.203.173.171:22: connected
    INFO ==> Running phase: Detect host operating systems INFO [ssh] 10.203.173.171:22: is running Ubuntu 24.04.2 LTS INFO [ssh] 10.203.173.30:22: is running Ubuntu 24.04.2 LTS INFO [ssh] 10.203.173.74:22: is running Ubuntu 24.04.2 LTS INFO ==> Running phase: Acquire exclusive host lock INFO ==> Running phase: Prepare hosts
    INFO ==> Running phase: Gather host facts INFO [ssh] 10.203.173.74:22: using Worker2 as hostname INFO [ssh] 10.203.173.171:22: using Master as hostname INFO [ssh] 10.203.173.30:22: using Worker1 as hostname INFO [ssh] 10.203.173.74:22: discovered ens3 as private interface INFO [ssh] 10.203.173.171:22: discovered ens3 as private interface INFO [ssh] 10.203.173.30:22: discovered ens3 as private interface INFO ==> Running phase: Validate hosts
    INFO ==> Running phase: Validate facts
    INFO ==> Running phase: Download k0s on hosts INFO [ssh] 10.203.173.74:22: downloading k0s v1.32.4+k0s.0 INFO [ssh] 10.203.173.171:22: downloading k0s v1.32.4+k0s.0 INFO [ssh] 10.203.173.30:22: downloading k0s v1.32.4+k0s.0 INFO ==> Running phase: Install k0s binaries on hosts INFO [ssh] 10.203.173.171:22: validating configuration INFO ==> Running phase: Configure k0s
    INFO [ssh] 10.203.173.171:22: installing new configuration INFO ==> Running phase: Initialize the k0s cluster INFO [ssh] 10.203.173.171:22: installing k0s controller INFO [ssh] 10.203.173.171:22: waiting for the k0s service to start INFO [ssh] 10.203.173.171:22: wait for kubernetes to reach ready state INFO ==> Running phase: Install workers
    INFO [ssh] 10.203.173.171:22: generating a join token for worker 1 INFO [ssh] 10.203.173.171:22: generating a join token for worker 2 INFO [ssh] 10.203.173.74:22: validating api connection to https://10.203.173.171:6443 using join token INFO [ssh] 10.203.173.30:22: validating api connection to https://10.203.173.171:6443 using join token INFO [ssh] 10.203.173.30:22: writing join token to /etc/k0s/k0stoken INFO [ssh] 10.203.173.74:22: writing join token to /etc/k0s/k0stoken INFO [ssh] 10.203.173.30:22: installing k0s worker INFO [ssh] 10.203.173.74:22: installing k0s worker INFO [ssh] 10.203.173.30:22: starting service
    INFO [ssh] 10.203.173.74:22: starting service
    INFO [ssh] 10.203.173.30:22: waiting for node to become ready INFO [ssh] 10.203.173.74:22: waiting for node to become ready INFO * Running clean-up for phase: Acquire exclusive host lock INFO * Running clean-up for phase: Install k0s binaries on hosts INFO [ssh] 10.203.173.74:22: cleaning up k0s binary tempfile INFO [ssh] 10.203.173.30:22: cleaning up k0s binary tempfile INFO [ssh] 10.203.173.171:22: cleaning up k0s binary tempfile INFO * Running clean-up for phase: Initialize the k0s cluster INFO [ssh] 10.203.173.171:22: cleaning up
    WARN [ssh] 10.203.173.171:22: k0s reset failed
    INFO * Running clean-up for phase: Install workers INFO [ssh] 10.203.173.74:22: cleaning up
    INFO [ssh] 10.203.173.30:22: cleaning up
    WARN [ssh] 10.203.173.74:22: k0s reset failed
    WARN [ssh] 10.203.173.30:22: k0s reset failed
    INFO ==> Apply failed
    FATA apply failed - log file saved to /home/antix/.cache/k0sctl/k0sctl.log: failed on 2 hosts:

    • [ssh] 10.203.173.30:22: context deadline exceeded node Worker1 is not ready
    • [ssh] 10.203.173.74:22: context deadline exceeded node Worker2 is not ready antix@antix:~/k0s$
  8. Assistant

    The k0sctl apply failed because the two worker nodes (Worker1 at 10.203.173.30 and Worker2 at 10.203.173.74) did not become ready in the expected timeout, causing:

    context deadline exceeded
    node Worker1 is not ready
    node Worker2 is not ready
    

    What happened?

    • The controller node (Master) appears to have installed k0s and started successfully.
    • The join tokens were generated and sent to the workers.
    • The k0s worker services started on the worker nodes.
    • But the nodes did not transition to Ready state within the timeout period, so k0sctl timed out.

    Common causes for workers not becoming ready in k0s cluster

    1. Network connectivity issues between nodes
      Workers must connect to the controller API server (https://10.203.173.171:6443). If there is a firewall, security group, or routing blocking communication, the join will fail.

    2. Wrong kubelet or container runtime configuration on workers
      Sometimes workers have incompatible runtimes, or kubelet fails to register.

    3. Resources (CPU/memory/disk) on worker nodes are insufficient
      If the nodes are constrained (e.g., disk pressure or memory pressure as before), kubelet might fail to start or stay ready.

    4. Version mismatch or k0sctl bugs
      Version mismatches or bugs can cause worker fail to register.

    5. k0s reset failed warnings
      The logs says k0s reset failed on worker nodes during cleanup. Sometimes stale state or resources interfere with proper restart.


    What to do next?

    Check the status on Worker nodes manually:

    1. SSH into worker nodes:
    ssh 10.203.173.30
    ssh 10.203.173.74
    
    1. Check the k0s service status:
    sudo systemctl status k0sworker.service
    
    1. Check k0s logs on worker:
    sudo journalctl -u k0sworker -f
    

    Look for errors like:

    • API server connection errors
    • Cert issues
    • Runtime failures
    1. Check kubelet (if running separately):
    sudo journalctl -u kubelet -f
    
    1. Check network connectivity:

    From worker node, try:

    curl -k https://10.203.173.171:6443
    

    or

    telnet 10.203.173.171 6443
    

    If it fails, network/firewall is blocking it.

    1. Check disk space & node resources on workers, since you previously had disk pressure issues.
    df -h
    free -m
    

    If there’s stale state or reset failures

    Try manually resetting k0s on worker nodes:

    sudo k0s reset
    

    Then restart k0sworker:

    sudo systemctl restart k0sworker
    

    Summary

    • Workers fail to become Ready because they cannot successfully join controller or start kubelet properly.
    • Check k0sworker services and logs on the two workers for specific errors.
    • Verify network connectivity to controller on port 6443.
    • Verify disk/memory/CPU resource availability.
    • Fix any network or resource issues.
    • Retry deployment.

    If you want, share:

    • Output of sudo systemctl status k0sworker on the workers,
    • Logs via sudo journalctl -u k0sworker -n 50,
    • Network test results,

    and I can help pinpoint the issue further.

  9. User

    equivalent k0s

  10. Assistant

    If you want to diagnose and troubleshoot a k0s cluster without using k0sctl (i.e., directly using k0s CLI and systemctl commands), here’s how to do the equivalent manual checks and operations on your nodes:


    Manual checks and commands equivalent to k0sctl apply and troubleshooting:

    1. Check k0s service status on each node

    On the controller node:

    sudo systemctl status k0scontroller
    

    On worker nodes:

    sudo systemctl status k0sworker
    

    This is to confirm the k0s controller or worker services are running.


    2. Check k0s logs

    On controller node:

    sudo journalctl -u k0scontroller -f
    

    On worker node:

    sudo journalctl -u k0sworker -f
    

    Look for errors during startup or connecting to controller.


    3. Verify node readiness from inside cluster

    Once kubectl is configured (usually on controller), check node status:

    kubectl get nodes
    

    To look deeper:

    kubectl describe node <worker-node-name>
    

    4. Check network connectivity from worker nodes to controller kube-apiserver

    On worker nodes:

    curl -k https://<controller_ip>:6443
    

    Or check port connectivity:

    telnet <controller_ip> 6443
    

    or

    nc -vz <controller_ip> 6443
    

    5. Manual join workers to cluster

    If you want to manually join a worker node to the k0s cluster:

    • On controller, generate a join token:
    sudo k0s token create --role=worker
    

    Copy the token output.

    • On worker node, run:
    sudo k0s install worker --token <join-token-from-controller>
    sudo systemctl start k0sworker
    sudo systemctl enable k0sworker
    

    This will join the worker to the cluster.


    6. Reset k0s manually if needed

    On any node:

    sudo k0s reset
    

    This cleans up cluster state and is useful if a node is stuck.


    7. Check cluster info and kubeconfig

    Usually k0s places the kubeconfig at /var/lib/k0s/pki/admin.conf on controller.

    You can export this as:

    export KUBECONFIG=/var/lib/k0s/pki/admin.conf
    kubectl get nodes
    

    Summary

    Using k0sctlEquivalent manual k0s usage
    k0sctl apply --config k0s.yamlManual install k0s controller and workers via k0s install and systemctl start
    k0sctl ssh <node>ssh <node>
    Check node status with kubectlUse kubectl with KUBECONFIG pointed to controller's kubeconfig
    Troubleshoot servicessystemctl status k0scontroller and systemctl status k0sworker and journalctl logs
    Reset nodek0s reset
    Generate join tokenk0s token create --role=worker

    If you want me to provide step-by-step manual instructions to install or fix workers and controller without k0sctl, just ask!