antix@antix:~$ kubectl -n longhorn-system get pod NAME READY STATUS RESTARTS AGE…
- User
antix@antix:~
- Assistant
Your Longhorn pods are in problematic states: some are
Pending, someInit:Error, someEvicted, and someError.Let's break down what these statuses typically mean and how you can troubleshoot each:
1. Pods in
Pending- Meaning: Pod is waiting to be scheduled to a node.
- Common causes:
- No node meets the resource requests or tolerations.
- Insufficient resources (CPU, Memory).
- NodeSelector or affinity rules can't be satisfied.
Check:
kubectl describe pod <pod-name> -n longhorn-systemLook under Events for reasons why pod scheduling failed.
2. Pods in
Init:Error- Meaning: An init container failed to start properly.
Check:
kubectl logs <pod-name> -n longhorn-system -c <init-container-name>Replace
<init-container-name>with the init container(s) listed in the pod spec.Common reasons can be:
- Missing images.
- Permission errors.
- Issues with volume mounts.
3. Pods in
Evicted- Meaning: Pod was terminated by Kubernetes node eviction mechanisms.
- Cause: Usually due to node resource pressure (memory, disk).
Check:
- Why were the pods evicted? Check node metrics.
- More info in pod events:
kubectl describe pod <evicted-pod-name> -n longhorn-systemLook specifically for reason:
"Evicted"and messages explaining the cause.Likely cause with Longhorn:
- Disk pressure or memory pressure on the node(s).
- Longhorn often requires a lot of disk space and memory.
4. Pods in
Error- Meaning: Pod container(s) have failed.
Check:
kubectl logs <pod-name> -n longhorn-system kubectl describe pod <pod-name> -n longhorn-systemLook at container states, exit codes, and events.
General Debug tips:
- Check node status and resources:
kubectl get nodes kubectl describe node <node-name>Especially look for disk pressure, memory pressure, or NotReady nodes.
-
Check storage: Longhorn heavily depends on available disk space; ensure your nodes are not running low on disk.
-
Check Longhorn system logs:
kubectl logs <longhorn-pod-name> -n longhorn-system- Check Kubernetes events for related errors:
kubectl get events -n longhorn-system
Summary / Next steps:
- For evicted pods: fix node resource pressure (free memory/disk).
- For pending pods: inspect scheduling constraints or cluster resources.
- For Init errors: check the init container logs for detailed error messages.
- For error pods: check container logs.
If you provide the output of
kubectl describe pod <pod-name> -n longhorn-systemor logs, I can help further analyze.
Let me know if you want assistance fetching/decoding some logs!
- User
antix@antix:~
- Assistant
Thanks for the update!
Right now you have:
- 1 longhorn-driver-deployer pod stuck in
Init:Error - 2 longhorn-manager pods
Evicted - 2 longhorn-ui pods in
Error - 2 longhorn-ui pods are running fine
Next steps for troubleshooting
1. Check why
longhorn-driver-deployer-7f95558b85-ztg2lhasInit:ErrorRun:
kubectl -n longhorn-system describe pod longhorn-driver-deployer-7f95558b85-ztg2lLook under Events for errors during the init container startup.
Also check init container logs (replace init container name if different):
kubectl -n longhorn-system logs longhorn-driver-deployer-7f95558b85-ztg2l -c <init-container-name>Usually, Longhorn uses an init container called
init.This will give precise error info.
2. Investigate
Evictedlonghorn-manager podsCheck why pods were evicted by describing pod:
kubectl -n longhorn-system describe pod longhorn-manager-8tvkr kubectl -n longhorn-system describe pod longhorn-manager-b9w4dYou should find something like:
Reason: Evicted Message: The node was low on resource: [MemoryPressure/DiskPressure].If you see resource pressure on nodes:
- Check node status:
kubectl describe node <node-name>- Check disk space on nodes:
kubectl ssh <node> # Or use your method to ssh df -h free -m- Free up disk or memory on nodes, or increase resources.
3. Check
Errorlonghorn-ui podsDescribe one of the erroring UI pods:
kubectl -n longhorn-system describe pod longhorn-ui-7ff79dfb4-5rvl7Then look at logs:
kubectl -n longhorn-system logs longhorn-ui-7ff79dfb4-5rvl7This may show why UI container is failing (misconfiguration, image problems, or crashed process).
General advice on resources:
Longhorn is storage-intensive and memory-consuming. The evicted pods suggest your nodes do not have sufficient disk or memory resources.
Check node health and free disk:
-
Disk:
- Longhorn requires disks with enough free space.
- If
/var/lib/longhornor related mount points are full, Longhorn pods may evict.
-
Memory:
- If node faces memory pressure, kubelet evicts pods to relieve memory.
Summary
- Describe problematic pods (
kubectl describe pod ...) → get event reasons. - Check init container logs for deployer pod.
- Check node disk/memory stats.
- Free up resources or add more node capacity.
If you share output from:
kubectl describe pod longhorn-driver-deployer-7f95558b85-ztg2l -n longhorn-systemkubectl describe pod longhorn-manager-8tvkr -n longhorn-system- logs of init container and UI pods
I can help analyze further.
- 1 longhorn-driver-deployer pod stuck in
- User
longhorn-system 22m Warning FailedDaemonPod daemonset/longhorn-manager (combined from similar events): Found failed daemon pod longhorn-system/longhorn-manager-w9nh6 on node worker2, will try to kill it longhorn-system 22m Normal SuccessfulDelete daemonset/longhorn-manager (combined from similar events): Deleted pod: longhorn-manager-w9nh6 longhorn-system 13m Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-zjcw2 longhorn-system 13m Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-69hm8 longhorn-system 9m49s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-zjcw2 on node worker1, will try to kill it longhorn-system 9m49s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-zjcw2 longhorn-system 9m48s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-pbbr8 longhorn-system 9m48s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-pbbr8 on node worker1, will try to kill it longhorn-system 9m48s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-pbbr8 longhorn-system 9m47s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-g5nwb longhorn-system 9m45s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-g5nwb on node worker1, will try to kill it longhorn-system 9m45s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-g5nwb longhorn-system 9m45s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-9rhd7 longhorn-system 9m40s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-9rhd7 on node worker1, will try to kill it longhorn-system 9m40s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-9rhd7 longhorn-system 9m40s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-x6gdf longhorn-system 9m32s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-x6gdf on node worker1, will try to kill it longhorn-system 9m30s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-x6gdf longhorn-system 9m29s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-stpbl longhorn-system 9m22s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-69hm8 on node worker2, will try to kill it longhorn-system 9m22s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-69hm8 longhorn-system 9m22s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-psjmc longhorn-system 9m21s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-psjmc on node worker2, will try to kill it longhorn-system 9m21s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-psjmc longhorn-system 9m21s Normal SuccessfulCreate daemonset/longhorn-manager Created pod: longhorn-manager-gqjwg longhorn-system 9m18s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-gqjwg on node worker2, will try to kill it longhorn-system 9m18s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-gqjwg longhorn-system 9m4s Normal SuccessfulCreate daemonset/longhorn-manager (combined from similar events): Created pod: longhorn-manager-fbnmk longhorn-system 9m14s Warning FailedDaemonPod daemonset/longhorn-manager Found failed daemon pod longhorn-system/longhorn-manager-stpbl on node worker1, will try to kill it longhorn-system 7m10s Warning FailedDaemonPod daemonset/longhorn-manager (combined from similar events): Found failed daemon pod longhorn-system/longhorn-manager-jn47p on node worker2, will try to kill it longhorn-system 9m13s Normal SuccessfulDelete daemonset/longhorn-manager Deleted pod: longhorn-manager-stpbl longhorn-system 7m9s Normal SuccessfulDelete daemonset/longhorn-manager (combined from similar events): Deleted pod: longhorn-manager-jn47p longhorn-system 13m Normal Scheduled pod/longhorn-ui-7ff79dfb4-4mdp2 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-4mdp2 to worker2 longhorn-system 13m Normal Pulling pod/longhorn-ui-7ff79dfb4-4mdp2 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 10m Warning Evicted pod/longhorn-ui-7ff79dfb4-4mdp2 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 428128Ki. longhorn-system 10m Warning ExceededGracePeriod pod/longhorn-ui-7ff79dfb4-4mdp2 Container runtime did not kill the pod within specified grace period. longhorn-system 10m Normal Pulled pod/longhorn-ui-7ff79dfb4-4mdp2 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 40.674s (3m0.765s including waiting). Image size: 73388900 bytes. longhorn-system 10m Normal Created pod/longhorn-ui-7ff79dfb4-4mdp2 Created container: longhorn-ui longhorn-system 10m Normal Started pod/longhorn-ui-7ff79dfb4-4mdp2 Started container longhorn-ui longhorn-system 9m59s Normal Killing pod/longhorn-ui-7ff79dfb4-4mdp2 Stopping container longhorn-ui longhorn-system 43m Normal Scheduled pod/longhorn-ui-7ff79dfb4-5rvl7 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-5rvl7 to worker2 longhorn-system 43m Normal Pulling pod/longhorn-ui-7ff79dfb4-5rvl7 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 42m Warning Evicted pod/longhorn-ui-7ff79dfb4-5rvl7 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 575112Ki. longhorn-system 41m Warning ExceededGracePeriod pod/longhorn-ui-7ff79dfb4-5rvl7 Container runtime did not kill the pod within specified grace period. longhorn-system 41m Warning Evicted pod/longhorn-ui-7ff79dfb4-5rvl7 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 334952Ki. longhorn-system 41m Normal Pulled pod/longhorn-ui-7ff79dfb4-5rvl7 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 30.658s (2m32.851s including waiting). Image size: 73388900 bytes. longhorn-system 41m Normal Created pod/longhorn-ui-7ff79dfb4-5rvl7 Created container: longhorn-ui longhorn-system 41m Normal Started pod/longhorn-ui-7ff79dfb4-5rvl7 Started container longhorn-ui longhorn-system 41m Normal Killing pod/longhorn-ui-7ff79dfb4-5rvl7 Stopping container longhorn-ui longhorn-system 31m Normal Scheduled pod/longhorn-ui-7ff79dfb4-5sbrt Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-5sbrt to worker1 longhorn-system 31m Normal Pulled pod/longhorn-ui-7ff79dfb4-5sbrt Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 31m Normal Created pod/longhorn-ui-7ff79dfb4-5sbrt Created container: longhorn-ui longhorn-system 31m Normal Started pod/longhorn-ui-7ff79dfb4-5sbrt Started container longhorn-ui longhorn-system 29m Warning Evicted pod/longhorn-ui-7ff79dfb4-5sbrt The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 211616Ki. Container longhorn-ui was using 44Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 29m Normal Killing pod/longhorn-ui-7ff79dfb4-5sbrt Stopping container longhorn-ui longhorn-system 31m Normal Scheduled pod/longhorn-ui-7ff79dfb4-6njj2 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-6njj2 to worker1 longhorn-system 31m Normal Pulled pod/longhorn-ui-7ff79dfb4-6njj2 Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 31m Normal Created pod/longhorn-ui-7ff79dfb4-6njj2 Created container: longhorn-ui longhorn-system 31m Normal Started pod/longhorn-ui-7ff79dfb4-6njj2 Started container longhorn-ui longhorn-system 29m Warning Evicted pod/longhorn-ui-7ff79dfb4-6njj2 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 218544Ki. Container longhorn-ui was using 44Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 29m Normal Killing pod/longhorn-ui-7ff79dfb4-6njj2 Stopping container longhorn-ui longhorn-system 43m Normal Scheduled pod/longhorn-ui-7ff79dfb4-7scm4 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-7scm4 to worker1 longhorn-system 43m Normal Pulling pod/longhorn-ui-7ff79dfb4-7scm4 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 41m Warning Evicted pod/longhorn-ui-7ff79dfb4-7scm4 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 333748Ki. longhorn-system 40m Warning ExceededGracePeriod pod/longhorn-ui-7ff79dfb4-7scm4 Container runtime did not kill the pod within specified grace period. longhorn-system 40m Normal Pulled pod/longhorn-ui-7ff79dfb4-7scm4 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 34.115s (3m5.186s including waiting). Image size: 73388900 bytes. longhorn-system 40m Normal Created pod/longhorn-ui-7ff79dfb4-7scm4 Created container: longhorn-ui longhorn-system 40m Normal Started pod/longhorn-ui-7ff79dfb4-7scm4 Started container longhorn-ui longhorn-system 40m Normal Killing pod/longhorn-ui-7ff79dfb4-7scm4 Stopping container longhorn-ui longhorn-system 20m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-7tqgx 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 9m54s Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-8h2ng 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 13m Normal Scheduled pod/longhorn-ui-7ff79dfb4-8xtr9 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-8xtr9 to worker1 longhorn-system 13m Normal Pulling pod/longhorn-ui-7ff79dfb4-8xtr9 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 10m Warning Evicted pod/longhorn-ui-7ff79dfb4-8xtr9 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 589436Ki. longhorn-system 10m Warning ExceededGracePeriod pod/longhorn-ui-7ff79dfb4-8xtr9 Container runtime did not kill the pod within specified grace period. longhorn-system 10m Normal Pulled pod/longhorn-ui-7ff79dfb4-8xtr9 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 25.594s (2m22.686s including waiting). Image size: 73388900 bytes. longhorn-system 10m Normal Created pod/longhorn-ui-7ff79dfb4-8xtr9 Created container: longhorn-ui longhorn-system 10m Normal Started pod/longhorn-ui-7ff79dfb4-8xtr9 Started container longhorn-ui longhorn-system 10m Normal Killing pod/longhorn-ui-7ff79dfb4-8xtr9 Stopping container longhorn-ui longhorn-system 40m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-9mrgh 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 35m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-9mrgh 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 34m Normal Scheduled pod/longhorn-ui-7ff79dfb4-9mrgh Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-9mrgh to worker2 longhorn-system 34m Normal Pulling pod/longhorn-ui-7ff79dfb4-9mrgh Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 33m Normal Pulled pod/longhorn-ui-7ff79dfb4-9mrgh Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 1m3.141s (1m3.141s including waiting). Image size: 73388900 bytes. longhorn-system 32m Normal Created pod/longhorn-ui-7ff79dfb4-9mrgh Created container: longhorn-ui longhorn-system 32m Normal Started pod/longhorn-ui-7ff79dfb4-9mrgh Started container longhorn-ui longhorn-system 32m Normal Pulled pod/longhorn-ui-7ff79dfb4-9mrgh Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 31m Warning Evicted pod/longhorn-ui-7ff79dfb4-9mrgh The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 311140Ki. Container longhorn-ui was using 48Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 31m Normal Killing pod/longhorn-ui-7ff79dfb4-9mrgh Stopping container longhorn-ui longhorn-system 29m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-b2tlz 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 25m Normal Scheduled pod/longhorn-ui-7ff79dfb4-b2tlz Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-b2tlz to worker2 longhorn-system 25m Normal Pulling pod/longhorn-ui-7ff79dfb4-b2tlz Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 24m Normal Pulled pod/longhorn-ui-7ff79dfb4-b2tlz Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 1.79s (51.711s including waiting). Image size: 73388900 bytes. longhorn-system 23m Normal Created pod/longhorn-ui-7ff79dfb4-b2tlz Created container: longhorn-ui longhorn-system 23m Normal Started pod/longhorn-ui-7ff79dfb4-b2tlz Started container longhorn-ui longhorn-system 23m Normal Pulled pod/longhorn-ui-7ff79dfb4-b2tlz Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 23m Warning BackOff pod/longhorn-ui-7ff79dfb4-b2tlz Back-off restarting failed container longhorn-ui in pod longhorn-ui-7ff79dfb4-b2tlz_longhorn-system(fb7c99f7-1be8-4878-aeb1-fd871fa3f67d) longhorn-system 22m Warning Evicted pod/longhorn-ui-7ff79dfb4-b2tlz The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 310488Ki. Container longhorn-ui was using 48Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 22m Normal Killing pod/longhorn-ui-7ff79dfb4-b2tlz Stopping container longhorn-ui longhorn-system 41m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-fplx6 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 35m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-fplx6 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 34m Normal Scheduled pod/longhorn-ui-7ff79dfb4-fplx6 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-fplx6 to worker2 longhorn-system 34m Normal Pulling pod/longhorn-ui-7ff79dfb4-fplx6 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 33m Normal Pulled pod/longhorn-ui-7ff79dfb4-fplx6 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 1.696s (1m20.381s including waiting). Image size: 73388900 bytes. longhorn-system 33m Normal Created pod/longhorn-ui-7ff79dfb4-fplx6 Created container: longhorn-ui longhorn-system 33m Normal Started pod/longhorn-ui-7ff79dfb4-fplx6 Started container longhorn-ui longhorn-system 31m Warning Evicted pod/longhorn-ui-7ff79dfb4-fplx6 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 311568Ki. Container longhorn-ui was using 44Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 31m Normal Killing pod/longhorn-ui-7ff79dfb4-fplx6 Stopping container longhorn-ui longhorn-system 10m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-gxm47 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 20m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-lwkqz 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 22m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-qp8vp 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 22m Normal Scheduled pod/longhorn-ui-7ff79dfb4-qp8vp Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-qp8vp to worker1 longhorn-system 22m Normal Pulling pod/longhorn-ui-7ff79dfb4-qp8vp Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 21m Normal Pulled pod/longhorn-ui-7ff79dfb4-qp8vp Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 21.987s (21.987s including waiting). Image size: 73388900 bytes. longhorn-system 20m Normal Created pod/longhorn-ui-7ff79dfb4-qp8vp Created container: longhorn-ui longhorn-system 20m Normal Started pod/longhorn-ui-7ff79dfb4-qp8vp Started container longhorn-ui longhorn-system 20m Normal Pulled pod/longhorn-ui-7ff79dfb4-qp8vp Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 21m Warning BackOff pod/longhorn-ui-7ff79dfb4-qp8vp Back-off restarting failed container longhorn-ui in pod longhorn-ui-7ff79dfb4-qp8vp_longhorn-system(bbeae2ed-0157-4623-a28e-9738e93e4f40) longhorn-system 20m Warning Evicted pod/longhorn-ui-7ff79dfb4-qp8vp The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 393956Ki. Container longhorn-ui was using 48Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 20m Normal Killing pod/longhorn-ui-7ff79dfb4-qp8vp Stopping container longhorn-ui longhorn-system 29m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-v4qt2 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 25m Normal Scheduled pod/longhorn-ui-7ff79dfb4-v4qt2 Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-v4qt2 to worker2 longhorn-system 25m Normal Pulling pod/longhorn-ui-7ff79dfb4-v4qt2 Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 24m Normal Pulled pod/longhorn-ui-7ff79dfb4-v4qt2 Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 50.205s (50.205s including waiting). Image size: 73388900 bytes. longhorn-system 23m Normal Created pod/longhorn-ui-7ff79dfb4-v4qt2 Created container: longhorn-ui longhorn-system 23m Normal Started pod/longhorn-ui-7ff79dfb4-v4qt2 Started container longhorn-ui longhorn-system 23m Normal Pulled pod/longhorn-ui-7ff79dfb4-v4qt2 Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 23m Warning BackOff pod/longhorn-ui-7ff79dfb4-v4qt2 Back-off restarting failed container longhorn-ui in pod longhorn-ui-7ff79dfb4-v4qt2_longhorn-system(78686ea7-8c8d-4123-9964-3a1c2c362139) longhorn-system 22m Warning Evicted pod/longhorn-ui-7ff79dfb4-v4qt2 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 311048Ki. Container longhorn-ui was using 48Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 22m Normal Killing pod/longhorn-ui-7ff79dfb4-v4qt2 Stopping container longhorn-ui longhorn-system 22m Warning FailedScheduling pod/longhorn-ui-7ff79dfb4-z9rtp 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 22m Normal Scheduled pod/longhorn-ui-7ff79dfb4-z9rtp Successfully assigned longhorn-system/longhorn-ui-7ff79dfb4-z9rtp to worker1 longhorn-system 22m Normal Pulling pod/longhorn-ui-7ff79dfb4-z9rtp Pulling image "longhornio/longhorn-ui:v1.8.1" longhorn-system 21m Normal Pulled pod/longhorn-ui-7ff79dfb4-z9rtp Successfully pulled image "longhornio/longhorn-ui:v1.8.1" in 1.837s (22.578s including waiting). Image size: 73388900 bytes. longhorn-system 20m Normal Created pod/longhorn-ui-7ff79dfb4-z9rtp Created container: longhorn-ui longhorn-system 20m Normal Started pod/longhorn-ui-7ff79dfb4-z9rtp Started container longhorn-ui longhorn-system 20m Normal Pulled pod/longhorn-ui-7ff79dfb4-z9rtp Container image "longhornio/longhorn-ui:v1.8.1" already present on machine longhorn-system 21m Warning BackOff pod/longhorn-ui-7ff79dfb4-z9rtp Back-off restarting failed container longhorn-ui in pod longhorn-ui-7ff79dfb4-z9rtp_longhorn-system(4b243f8d-c00f-41ac-81b3-78afb9e66a48) longhorn-system 20m Warning Evicted pod/longhorn-ui-7ff79dfb4-z9rtp The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 394216Ki. Container longhorn-ui was using 48Ki, request is 0, has larger consumption of ephemeral-storage. longhorn-system 20m Normal Killing pod/longhorn-ui-7ff79dfb4-z9rtp Stopping container longhorn-ui longhorn-system 43m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-5rvl7 longhorn-system 43m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-7scm4 longhorn-system 41m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-fplx6 longhorn-system 40m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-9mrgh longhorn-system 31m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-6njj2 longhorn-system 31m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-5sbrt longhorn-system 29m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-v4qt2 longhorn-system 29m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-b2tlz longhorn-system 22m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-qp8vp longhorn-system 20m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 (combined from similar events): Created pod: longhorn-ui-7ff79dfb4-lwkqz longhorn-system 13m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-8xtr9 longhorn-system 13m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-4mdp2 longhorn-system 10m Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-gxm47 longhorn-system 9m54s Normal SuccessfulCreate replicaset/longhorn-ui-7ff79dfb4 Created pod: longhorn-ui-7ff79dfb4-8h2ng longhorn-system 43m Normal ScalingReplicaSet deployment/longhorn-ui Scaled up replica set longhorn-ui-7ff79dfb4 from 0 to 2 longhorn-system 13m Normal ScalingReplicaSet deployment/longhorn-ui Scaled up replica set longhorn-ui-7ff79dfb4 from 0 to 2 longhorn-system 22m Warning FailedScheduling pod/longhorn-uninstall-jfqxc 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. longhorn-system 22m Normal Scheduled pod/longhorn-uninstall-jfqxc Successfully assigned longhorn-system/longhorn-uninstall-jfqxc to worker1 longhorn-system 22m Normal Pulling pod/longhorn-uninstall-jfqxc Pulling image "longhornio/longhorn-manager:v1.8.1" longhorn-system 20m Normal Pulled pod/longhorn-uninstall-jfqxc Successfully pulled image "longhornio/longhorn-manager:v1.8.1" in 1.927s (1m25.744s including waiting). Image size: 119475837 bytes. longhorn-system 20m Normal Created pod/longhorn-uninstall-jfqxc Created container: longhorn-uninstall longhorn-system 20m Normal Started pod/longhorn-uninstall-jfqxc Started container longhorn-uninstall longhorn-system 24m Normal Scheduled pod/longhorn-uninstall-mbrvt Successfully assigned longhorn-system/longhorn-uninstall-mbrvt to worker2 longhorn-system 24m Normal Pulling pod/longhorn-uninstall-mbrvt Pulling image "longhornio/longhorn-manager:v1.8.1" longhorn-system 22m Normal Pulled pod/longhorn-uninstall-mbrvt Successfully pulled image "longhornio/longhorn-manager:v1.8.1" in 2.745s (1m8.689s including waiting). Image size: 119475837 bytes. longhorn-system 22m Normal Created pod/longhorn-uninstall-mbrvt Created container: longhorn-uninstall longhorn-system 22m Normal Started pod/longhorn-uninstall-mbrvt Started container longhorn-uninstall longhorn-system 22m Warning Evicted pod/longhorn-uninstall-mbrvt The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 310672Ki. longhorn-system 22m Warning ExceededGracePeriod pod/longhorn-uninstall-mbrvt Container runtime did not kill the pod within specified grace period. longhorn-system 24m Normal SuccessfulCreate job/longhorn-uninstall Created pod: longhorn-uninstall-mbrvt longhorn-system 22m Normal SuccessfulCreate job/longhorn-uninstall Created pod: longhorn-uninstall-jfqxc longhorn-system 20m Warning BackoffLimitExceeded job/longhorn-uninstall Job has reached the specified backoff limit metallb-system 20m Warning FailedScheduling pod/controller-bb5f47665-9wvd9 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 15m Normal Scheduled pod/controller-bb5f47665-9wvd9 Successfully assigned metallb-system/controller-bb5f47665-9wvd9 to worker2 metallb-system 15m Normal Pulling pod/controller-bb5f47665-9wvd9 Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 15m Normal Pulled pod/controller-bb5f47665-9wvd9 Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 24.231s (24.231s including waiting). Image size: 29163511 bytes. metallb-system 15m Normal Created pod/controller-bb5f47665-9wvd9 Created container: controller metallb-system 15m Normal Started pod/controller-bb5f47665-9wvd9 Started container controller metallb-system 9m57s Warning Evicted pod/controller-bb5f47665-9wvd9 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 310612Ki. Container controller was using 84Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 9m57s Normal Killing pod/controller-bb5f47665-9wvd9 Stopping container controller metallb-system 9m54s Warning FailedScheduling pod/controller-bb5f47665-bmx9v 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 4m52s Warning FailedScheduling pod/controller-bb5f47665-bmx9v 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 3m35s Normal Scheduled pod/controller-bb5f47665-bmx9v Successfully assigned metallb-system/controller-bb5f47665-bmx9v to worker1 metallb-system 3m32s Normal Pulling pod/controller-bb5f47665-bmx9v Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 2m50s Normal Pulled pod/controller-bb5f47665-bmx9v Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 24.545s (41.673s including waiting). Image size: 29163511 bytes. metallb-system 2m50s Normal Created pod/controller-bb5f47665-bmx9v Created container: controller metallb-system 2m49s Normal Started pod/controller-bb5f47665-bmx9v Started container controller metallb-system 31m Normal Scheduled pod/controller-bb5f47665-h7t84 Successfully assigned metallb-system/controller-bb5f47665-h7t84 to worker1 metallb-system 31m Normal Pulling pod/controller-bb5f47665-h7t84 Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 30m Warning Evicted pod/controller-bb5f47665-h7t84 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 592432Ki. metallb-system 30m Warning ExceededGracePeriod pod/controller-bb5f47665-h7t84 Container runtime did not kill the pod within specified grace period. metallb-system 30m Warning Evicted pod/controller-bb5f47665-h7t84 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 390424Ki. metallb-system 29m Normal Pulled pod/controller-bb5f47665-h7t84 Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 19.988s (1m42.564s including waiting). Image size: 29163511 bytes. metallb-system 29m Normal Created pod/controller-bb5f47665-h7t84 Created container: controller metallb-system 29m Normal Started pod/controller-bb5f47665-h7t84 Started container controller metallb-system 29m Normal Killing pod/controller-bb5f47665-h7t84 Stopping container controller metallb-system 29m Warning FailedScheduling pod/controller-bb5f47665-pcnsj 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 25m Normal Scheduled pod/controller-bb5f47665-pcnsj Successfully assigned metallb-system/controller-bb5f47665-pcnsj to worker2 metallb-system 25m Normal Pulling pod/controller-bb5f47665-pcnsj Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 23m Warning Evicted pod/controller-bb5f47665-pcnsj The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 438304Ki. metallb-system 23m Warning ExceededGracePeriod pod/controller-bb5f47665-pcnsj Container runtime did not kill the pod within specified grace period. metallb-system 22m Normal Pulled pod/controller-bb5f47665-pcnsj Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 21.521s (2m19.598s including waiting). Image size: 29163511 bytes. metallb-system 22m Normal Created pod/controller-bb5f47665-pcnsj Created container: controller metallb-system 22m Normal Started pod/controller-bb5f47665-pcnsj Started container controller metallb-system 22m Normal Killing pod/controller-bb5f47665-pcnsj Stopping container controller metallb-system 22m Warning FailedScheduling pod/controller-bb5f47665-rwlqr 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 22m Normal Scheduled pod/controller-bb5f47665-rwlqr Successfully assigned metallb-system/controller-bb5f47665-rwlqr to worker1 metallb-system 22m Normal Pulling pod/controller-bb5f47665-rwlqr Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 20m Warning Evicted pod/controller-bb5f47665-rwlqr The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 494544Ki. metallb-system 20m Warning ExceededGracePeriod pod/controller-bb5f47665-rwlqr Container runtime did not kill the pod within specified grace period. metallb-system 20m Normal Pulled pod/controller-bb5f47665-rwlqr Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 24.874s (1m50.589s including waiting). Image size: 29163511 bytes. metallb-system 20m Normal Created pod/controller-bb5f47665-rwlqr Created container: controller metallb-system 20m Normal Started pod/controller-bb5f47665-rwlqr Started container controller metallb-system 20m Normal Killing pod/controller-bb5f47665-rwlqr Stopping container controller metallb-system 40m Warning Evicted pod/controller-bb5f47665-vlj95 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 219300Ki. Container controller was using 96Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 40m Normal Killing pod/controller-bb5f47665-vlj95 Stopping container controller metallb-system 40m Warning FailedScheduling pod/controller-bb5f47665-vqqz2 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 35m Warning FailedScheduling pod/controller-bb5f47665-vqqz2 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling. metallb-system 34m Normal Scheduled pod/controller-bb5f47665-vqqz2 Successfully assigned metallb-system/controller-bb5f47665-vqqz2 to worker2 metallb-system 34m Normal Pulling pod/controller-bb5f47665-vqqz2 Pulling image "quay.io/metallb/controller:v0.14.9" metallb-system 32m Normal Pulled pod/controller-bb5f47665-vqqz2 Successfully pulled image "quay.io/metallb/controller:v0.14.9" in 22.913s (1m43.205s including waiting). Image size: 29163511 bytes. metallb-system 32m Normal Created pod/controller-bb5f47665-vqqz2 Created container: controller metallb-system 32m Normal Started pod/controller-bb5f47665-vqqz2 Started container controller metallb-system 31m Warning Evicted pod/controller-bb5f47665-vqqz2 The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 310980Ki. Container controller was using 80Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 31m Normal Killing pod/controller-bb5f47665-vqqz2 Stopping container controller metallb-system 40m Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-vqqz2 metallb-system 31m Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-h7t84 metallb-system 29m Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-pcnsj metallb-system 22m Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-rwlqr metallb-system 20m Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-9wvd9 metallb-system 9m54s Normal SuccessfulCreate replicaset/controller-bb5f47665 Created pod: controller-bb5f47665-bmx9v metallb-system 40m Normal Scheduled pod/speaker-9nqxs Successfully assigned metallb-system/speaker-9nqxs to worker1 metallb-system 40m Warning Evicted pod/speaker-9nqxs The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-9wwmv Successfully assigned metallb-system/speaker-9wwmv to worker1 metallb-system 40m Warning Evicted pod/speaker-9wwmv The node had condition: [DiskPressure]. metallb-system 8m2s Normal Scheduled pod/speaker-cdt7b Successfully assigned metallb-system/speaker-cdt7b to worker2 metallb-system 8m2s Warning Evicted pod/speaker-cdt7b The node had condition: [DiskPressure]. metallb-system 35m Normal Scheduled pod/speaker-ckcjq Successfully assigned metallb-system/speaker-ckcjq to worker2 metallb-system 35m Warning Evicted pod/speaker-ckcjq The node had condition: [DiskPressure]. metallb-system 23m Normal Scheduled pod/speaker-dmsmj Successfully assigned metallb-system/speaker-dmsmj to worker1 metallb-system 23m Warning Evicted pod/speaker-dmsmj The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-fvn6g Successfully assigned metallb-system/speaker-fvn6g to worker2 metallb-system 40m Warning Evicted pod/speaker-fvn6g The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-gvx65 Successfully assigned metallb-system/speaker-gvx65 to worker2 metallb-system 40m Warning Evicted pod/speaker-gvx65 The node had condition: [DiskPressure]. metallb-system 38m Normal Scheduled pod/speaker-hdhbl Successfully assigned metallb-system/speaker-hdhbl to worker2 metallb-system 38m Warning Evicted pod/speaker-hdhbl The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-j7klj Successfully assigned metallb-system/speaker-j7klj to worker2 metallb-system 40m Warning Evicted pod/speaker-j7klj The node had condition: [DiskPressure]. metallb-system 23m Normal Scheduled pod/speaker-jb9q6 Successfully assigned metallb-system/speaker-jb9q6 to worker2 metallb-system 23m Warning Evicted pod/speaker-jb9q6 The node had condition: [DiskPressure]. metallb-system 31m Normal Scheduled pod/speaker-l8ft6 Successfully assigned metallb-system/speaker-l8ft6 to worker1 metallb-system 31m Normal Pulling pod/speaker-l8ft6 Pulling image "quay.io/metallb/speaker:v0.14.9" metallb-system 30m Normal Pulled pod/speaker-l8ft6 Successfully pulled image "quay.io/metallb/speaker:v0.14.9" in 39.391s (39.391s including waiting). Image size: 55246333 bytes. metallb-system 30m Normal Created pod/speaker-l8ft6 Created container: speaker metallb-system 30m Normal Started pod/speaker-l8ft6 Started container speaker metallb-system 29m Warning Evicted pod/speaker-l8ft6 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 218324Ki. Container speaker was using 76Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 29m Normal Killing pod/speaker-l8ft6 Stopping container speaker metallb-system 31m Normal Scheduled pod/speaker-l9xcz Successfully assigned metallb-system/speaker-l9xcz to worker2 metallb-system 31m Warning Evicted pod/speaker-l9xcz The node had condition: [DiskPressure]. metallb-system 40m Warning Evicted pod/speaker-lbprs The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-lbprs Successfully assigned metallb-system/speaker-lbprs to worker2 metallb-system 40m Normal Scheduled pod/speaker-mzq9b Successfully assigned metallb-system/speaker-mzq9b to worker1 metallb-system 40m Warning Evicted pod/speaker-mzq9b The node had condition: [DiskPressure]. metallb-system 39m Warning Evicted pod/speaker-q4jh5 The node had condition: [DiskPressure]. metallb-system 39m Normal Scheduled pod/speaker-q4jh5 Successfully assigned metallb-system/speaker-q4jh5 to worker1 metallb-system 8m2s Normal Scheduled pod/speaker-q4v6n Successfully assigned metallb-system/speaker-q4v6n to worker1 metallb-system 8m2s Warning Evicted pod/speaker-q4v6n The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-qjvgb Successfully assigned metallb-system/speaker-qjvgb to worker1 metallb-system 40m Warning Evicted pod/speaker-qjvgb The node had condition: [DiskPressure]. metallb-system 35m Normal Scheduled pod/speaker-r2rsz Successfully assigned metallb-system/speaker-r2rsz to worker1 metallb-system 35m Warning Evicted pod/speaker-r2rsz The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-rg45w Successfully assigned metallb-system/speaker-rg45w to worker2 metallb-system 40m Warning Evicted pod/speaker-rg45w The node had condition: [DiskPressure]. metallb-system 40m Warning Evicted pod/speaker-rzcs7 The node was low on resource: ephemeral-storage. Threshold quantity: 408388818, available: 217284Ki. Container speaker was using 80Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 40m Normal Killing pod/speaker-rzcs7 Stopping container speaker metallb-system 40m Normal Scheduled pod/speaker-t457m Successfully assigned metallb-system/speaker-t457m to worker2 metallb-system 40m Warning Evicted pod/speaker-t457m The node had condition: [DiskPressure]. metallb-system 39m Normal Scheduled pod/speaker-v9cmq Successfully assigned metallb-system/speaker-v9cmq to worker2 metallb-system 39m Warning Evicted pod/speaker-v9cmq The node had condition: [DiskPressure]. metallb-system 29m Normal Scheduled pod/speaker-vr2gt Successfully assigned metallb-system/speaker-vr2gt to worker1 metallb-system 29m Warning Evicted pod/speaker-vr2gt The node had condition: [DiskPressure]. metallb-system 38m Normal Scheduled pod/speaker-wkqmz Successfully assigned metallb-system/speaker-wkqmz to worker1 metallb-system 38m Warning Evicted pod/speaker-wkqmz The node had condition: [DiskPressure]. metallb-system 39m Normal Scheduled pod/speaker-wrp7x Successfully assigned metallb-system/speaker-wrp7x to worker1 metallb-system 39m Warning Evicted pod/speaker-wrp7x The node had condition: [DiskPressure]. metallb-system 40m Normal Scheduled pod/speaker-xhjvq Successfully assigned metallb-system/speaker-xhjvq to worker1 metallb-system 40m Warning Evicted pod/speaker-xhjvq The node had condition: [DiskPressure]. metallb-system 41m Warning Evicted pod/speaker-z7bpd The node was low on resource: ephemeral-storage. Threshold quantity: 612583243, available: 230352Ki. Container speaker was using 80Ki, request is 0, has larger consumption of ephemeral-storage. metallb-system 41m Normal Killing pod/speaker-z7bpd Stopping container speaker metallb-system 40m Normal Scheduled pod/speaker-z9slq Successfully assigned metallb-system/speaker-z9slq to worker2 metallb-system 40m Warning Evicted pod/speaker-z9slq The node had condition: [DiskPressure]. metallb-system 40m Normal SucceededDaemonPod daemonset/speaker Found succeeded daemon pod metallb-system/speaker-z7bpd on node worker2, will try to delete it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-z7bpd metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-lbprs metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-lbprs on node worker2, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-lbprs metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-z9slq metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-z9slq on node worker2, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-z9slq metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-t457m metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-t457m on node worker2, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-t457m metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-j7klj metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-j7klj on node worker2, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-j7klj metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-rg45w metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-rg45w on node worker2, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-rg45w metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-gvx65 metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-rzcs7 on node worker1, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-rzcs7 metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-mzq9b metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-mzq9b on node worker1, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-mzq9b metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-qjvgb metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-qjvgb on node worker1, will try to kill it metallb-system 40m Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-qjvgb metallb-system 40m Normal SuccessfulCreate daemonset/speaker Created pod: speaker-9nqxs metallb-system 40m Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-9nqxs on node worker1, will try to kill it metallb-system 23m Warning FailedDaemonPod daemonset/speaker (combined from similar events): Found failed daemon pod metallb-system/speaker-vr2gt on node worker1, will try to kill it metallb-system 23m Normal SuccessfulDelete daemonset/speaker (combined from similar events): Deleted pod: speaker-vr2gt metallb-system 40m Normal SuccessfulCreate daemonset/speaker (combined from similar events): Created pod: speaker-xhjvq metallb-system 29m Normal SucceededDaemonPod daemonset/speaker Found succeeded daemon pod metallb-system/speaker-l8ft6 on node worker1, will try to delete it metallb-system 8m2s Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-jb9q6 on node worker2, will try to kill it metallb-system 8m2s Warning FailedDaemonPod daemonset/speaker Found failed daemon pod metallb-system/speaker-dmsmj on node worker1, will try to kill it metallb-system 8m2s Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-jb9q6 metallb-system 8m2s Normal SuccessfulDelete daemonset/speaker Deleted pod: speaker-dmsmj metallb-system 8m2s Normal SuccessfulCreate daemonset/speaker Created pod: speaker-q4v6n antix@antix:~$
- Assistant
The events you've posted clearly indicate:
Root Cause
- Your Kubernetes nodes are under disk pressure, specifically on ephemeral-storage.
- Many pods (Longhorn's
longhorn-manager,longhorn-ui,longhorn-uninstall, as well as MetalLB pods) are repeatedly getting evicted due to insufficient ephemeral storage. - Nodes have the taint
node.kubernetes.io/disk-pressurewhich prevents pods from scheduling on them unless they tolerate that taint. - Pods also experience FailedDaemonPod and get continuously killed and recreated due to the underlying node resource issues.
What is ephemeral-storage pressure?
- Ephemeral storage is the node’s local available disk space used for container writable layers, logs, and emptyDir volumes.
- When it runs low, kubelet taints the node with
node.kubernetes.io/disk-pressureand evicts pods to free space. - Pods without proper tolerations and resource requests get unscheduled or rejected.
How to Fix This
1. Free up disk space on your nodes
- SSH to each node:
ssh user@worker1 ssh user@worker2- Check disk usage:
df -h-
Identify which partitions are almost full, especially
/var/lib/kubelet,/var/lib/docker,/var/lib/containerd, and/tmp -
Remove unused images and containers:
For Docker:
docker system prune -afFor containerd:
crictl images # List images crictl rmi <imageID> # Remove unused images- Check for large log files and clean them up (e.g.,
/var/log/)
2. Add more ephemeral storage to nodes or increase PV sizes if using hostPath based storage
- If freeing disk space is not enough, consider expanding the disk size or adding additional storage to your worker nodes.
3. Configure resource requests and limits for Longhorn pods (especially ephemeral-storage)
- This helps Kubernetes properly schedule pods based on real resource needs.
- Check your Longhorn deployment manifests and add requests/limits for ephemeral-storage.
Example for a container spec:
resources: requests: ephemeral-storage: "1Gi" limits: ephemeral-storage: "2Gi"
4. Remove /taints or add tolerations as a short-term workaround (not recommended)
- You could remove the
node.kubernetes.io/disk-pressuretaint with:
kubectl taint nodes <node-name> node.kubernetes.io/disk-pressure:NoSchedule-- However, this will make pods run on nodes that are disk pressure-affected, and they will likely be evicted immediately again unless disk pressure is relieved.
5. Monitor disk usage continuously
Install tools like
node-problem-detector, or monitor disk space with your favorite monitoring stack (Prometheus, Grafana).
Summary
Your Longhorn and other pods failing/evicting is due to nodes running low on ephemeral-storage, causing Kubernetes to taint the nodes with
node.kubernetes.io/disk-pressure. Pods cannot be scheduled or are evicted when running on these nodes.You need to:
- Free up disk space on your nodes
- Add tolerations if needed (temporarily)
- Configure Longhorn pods with proper ephemeral-storage requests/limits
- Possibly expand the nodes' disk capacity
If you want, I can also help with commands to check disk usage or cleaning up the nodes. Just ask!
- User
antix@antix:~/k0s$ k0sctl apply --config k0s.yaml
⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███ ⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███ ███ ███ ⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███ ███ ███ ⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███ ███ ███ ⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████ ███ ██████████ k0sctl v0.23.0 Copyright 2023, k0sctl authors. By continuing to use k0sctl you agree to these terms: https://k0sproject.io/licenses/eula INFO ==> Running phase: Set k0s version
INFO Looking up latest stable k0s version
INFO Using k0s version v1.32.4+k0s.0
INFO ==> Running phase: Connect to hosts INFO [ssh] 10.203.173.30:22: connected
INFO [ssh] 10.203.173.74:22: connected
INFO [ssh] 10.203.173.171:22: connected
INFO ==> Running phase: Detect host operating systems INFO [ssh] 10.203.173.171:22: is running Ubuntu 24.04.2 LTS INFO [ssh] 10.203.173.30:22: is running Ubuntu 24.04.2 LTS INFO [ssh] 10.203.173.74:22: is running Ubuntu 24.04.2 LTS INFO ==> Running phase: Acquire exclusive host lock INFO ==> Running phase: Prepare hosts
INFO ==> Running phase: Gather host facts INFO [ssh] 10.203.173.74:22: using Worker2 as hostname INFO [ssh] 10.203.173.171:22: using Master as hostname INFO [ssh] 10.203.173.30:22: using Worker1 as hostname INFO [ssh] 10.203.173.74:22: discovered ens3 as private interface INFO [ssh] 10.203.173.171:22: discovered ens3 as private interface INFO [ssh] 10.203.173.30:22: discovered ens3 as private interface INFO ==> Running phase: Validate hosts
INFO ==> Running phase: Validate facts
INFO ==> Running phase: Download k0s on hosts INFO [ssh] 10.203.173.74:22: downloading k0s v1.32.4+k0s.0 INFO [ssh] 10.203.173.171:22: downloading k0s v1.32.4+k0s.0 INFO [ssh] 10.203.173.30:22: downloading k0s v1.32.4+k0s.0 INFO ==> Running phase: Install k0s binaries on hosts INFO [ssh] 10.203.173.171:22: validating configuration INFO ==> Running phase: Configure k0s
INFO [ssh] 10.203.173.171:22: installing new configuration INFO ==> Running phase: Initialize the k0s cluster INFO [ssh] 10.203.173.171:22: installing k0s controller INFO [ssh] 10.203.173.171:22: waiting for the k0s service to start INFO [ssh] 10.203.173.171:22: wait for kubernetes to reach ready state INFO ==> Running phase: Install workers
INFO [ssh] 10.203.173.171:22: generating a join token for worker 1 INFO [ssh] 10.203.173.171:22: generating a join token for worker 2 INFO [ssh] 10.203.173.74:22: validating api connection to https://10.203.173.171:6443 using join token INFO [ssh] 10.203.173.30:22: validating api connection to https://10.203.173.171:6443 using join token INFO [ssh] 10.203.173.30:22: writing join token to /etc/k0s/k0stoken INFO [ssh] 10.203.173.74:22: writing join token to /etc/k0s/k0stoken INFO [ssh] 10.203.173.30:22: installing k0s worker INFO [ssh] 10.203.173.74:22: installing k0s worker INFO [ssh] 10.203.173.30:22: starting service
INFO [ssh] 10.203.173.74:22: starting service
INFO [ssh] 10.203.173.30:22: waiting for node to become ready INFO [ssh] 10.203.173.74:22: waiting for node to become ready INFO * Running clean-up for phase: Acquire exclusive host lock INFO * Running clean-up for phase: Install k0s binaries on hosts INFO [ssh] 10.203.173.74:22: cleaning up k0s binary tempfile INFO [ssh] 10.203.173.30:22: cleaning up k0s binary tempfile INFO [ssh] 10.203.173.171:22: cleaning up k0s binary tempfile INFO * Running clean-up for phase: Initialize the k0s cluster INFO [ssh] 10.203.173.171:22: cleaning up
WARN [ssh] 10.203.173.171:22: k0s reset failed
INFO * Running clean-up for phase: Install workers INFO [ssh] 10.203.173.74:22: cleaning up
INFO [ssh] 10.203.173.30:22: cleaning up
WARN [ssh] 10.203.173.74:22: k0s reset failed
WARN [ssh] 10.203.173.30:22: k0s reset failed
INFO ==> Apply failed
FATA apply failed - log file saved to /home/antix/.cache/k0sctl/k0sctl.log: failed on 2 hosts:- [ssh] 10.203.173.30:22: context deadline exceeded node Worker1 is not ready
- [ssh] 10.203.173.74:22: context deadline exceeded node Worker2 is not ready antix@antix:~/k0s$
- Assistant
The k0sctl apply failed because the two worker nodes (
Worker1at 10.203.173.30 andWorker2at 10.203.173.74) did not become ready in the expected timeout, causing:context deadline exceeded node Worker1 is not ready node Worker2 is not ready
What happened?
- The controller node (
Master) appears to have installed k0s and started successfully. - The join tokens were generated and sent to the workers.
- The k0s worker services started on the worker nodes.
- But the nodes did not transition to
Readystate within the timeout period, so k0sctl timed out.
Common causes for workers not becoming ready in k0s cluster
-
Network connectivity issues between nodes
Workers must connect to the controller API server (https://10.203.173.171:6443). If there is a firewall, security group, or routing blocking communication, the join will fail. -
Wrong kubelet or container runtime configuration on workers
Sometimes workers have incompatible runtimes, or kubelet fails to register. -
Resources (CPU/memory/disk) on worker nodes are insufficient
If the nodes are constrained (e.g., disk pressure or memory pressure as before), kubelet might fail to start or stay ready. -
Version mismatch or k0sctl bugs
Version mismatches or bugs can cause worker fail to register. -
k0s reset failed warnings
The logs saysk0s reset failedon worker nodes during cleanup. Sometimes stale state or resources interfere with proper restart.
What to do next?
Check the status on Worker nodes manually:
- SSH into worker nodes:
ssh 10.203.173.30 ssh 10.203.173.74- Check the k0s service status:
sudo systemctl status k0sworker.service- Check k0s logs on worker:
sudo journalctl -u k0sworker -fLook for errors like:
- API server connection errors
- Cert issues
- Runtime failures
- Check
kubelet(if running separately):
sudo journalctl -u kubelet -f- Check network connectivity:
From worker node, try:
curl -k https://10.203.173.171:6443or
telnet 10.203.173.171 6443If it fails, network/firewall is blocking it.
- Check disk space & node resources on workers, since you previously had disk pressure issues.
df -h free -m
If there’s stale state or reset failures
Try manually resetting k0s on worker nodes:
sudo k0s resetThen restart k0sworker:
sudo systemctl restart k0sworker
Summary
- Workers fail to become
Readybecause they cannot successfully join controller or start kubelet properly. - Check k0sworker services and logs on the two workers for specific errors.
- Verify network connectivity to controller on port 6443.
- Verify disk/memory/CPU resource availability.
- Fix any network or resource issues.
- Retry deployment.
If you want, share:
- Output of
sudo systemctl status k0sworkeron the workers, - Logs via
sudo journalctl -u k0sworker -n 50, - Network test results,
and I can help pinpoint the issue further.
- The controller node (
- User
equivalent k0s
- Assistant
If you want to diagnose and troubleshoot a k0s cluster without using
k0sctl(i.e., directly using k0s CLI and systemctl commands), here’s how to do the equivalent manual checks and operations on your nodes:
Manual checks and commands equivalent to
k0sctl applyand troubleshooting:1. Check k0s service status on each node
On the controller node:
sudo systemctl status k0scontrollerOn worker nodes:
sudo systemctl status k0sworkerThis is to confirm the k0s controller or worker services are running.
2. Check k0s logs
On controller node:
sudo journalctl -u k0scontroller -fOn worker node:
sudo journalctl -u k0sworker -fLook for errors during startup or connecting to controller.
3. Verify node readiness from inside cluster
Once
kubectlis configured (usually on controller), check node status:kubectl get nodesTo look deeper:
kubectl describe node <worker-node-name>
4. Check network connectivity from worker nodes to controller kube-apiserver
On worker nodes:
curl -k https://<controller_ip>:6443Or check port connectivity:
telnet <controller_ip> 6443or
nc -vz <controller_ip> 6443
5. Manual join workers to cluster
If you want to manually join a worker node to the k0s cluster:
- On controller, generate a join token:
sudo k0s token create --role=workerCopy the token output.
- On worker node, run:
sudo k0s install worker --token <join-token-from-controller> sudo systemctl start k0sworker sudo systemctl enable k0sworkerThis will join the worker to the cluster.
6. Reset k0s manually if needed
On any node:
sudo k0s resetThis cleans up cluster state and is useful if a node is stuck.
7. Check cluster info and kubeconfig
Usually k0s places the kubeconfig at
/var/lib/k0s/pki/admin.confon controller.You can export this as:
export KUBECONFIG=/var/lib/k0s/pki/admin.conf kubectl get nodes
Summary
Using k0sctl Equivalent manual k0s usage k0sctl apply --config k0s.yamlManual install k0s controller and workers via k0s installandsystemctl startk0sctl ssh <node>ssh <node>Check node status with kubectlUse kubectlwith KUBECONFIG pointed to controller's kubeconfigTroubleshoot services systemctl status k0scontrollerandsystemctl status k0sworkerand journalctl logsReset node k0s resetGenerate join token k0s token create --role=worker
If you want me to provide step-by-step manual instructions to install or fix workers and controller without k0sctl, just ask!