vagrant@Master:~$ kubectl describe pod svclb-grafana-03ffa6f9-87fds Error from server…
- User
vagrant@Master:
kubectl logs pod svclb-grafana-03ffa6f9-87fds error: error from server (NotFound): pods "pod" not found in namespace "default" vagrant@Master:kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES grafana grafana-5744c94549-ksw8q 1/1 Running 4 (12m ago) 34h 10.42.1.33 worker kube-system coredns-ff8999cc5-txw22 1/1 Running 4 (13m ago) 36h 10.42.0.66 master kube-system local-path-provisioner-774c6665dc-7njhl 1/1 Running 5 (13m ago) 36h 10.42.0.57 master kube-system metrics-server-6f4c6675d5-qjlhq 1/1 Running 5 (13m ago) 36h 10.42.0.61 master kube-system svclb-grafana-03ffa6f9-87fds 0/1 Pending 0 21h kube-system svclb-grafana-03ffa6f9-9qsqq 0/1 Pending 0 21h kube-system svclb-grafana-03ffa6f9-f5nrh 0/1 Pending 0 21h kube-system svclb-prometheus-alertmanager-de124004-c6tk2 1/1 Running 4 (12m ago) 34h 10.42.2.30 worker2 kube-system svclb-prometheus-alertmanager-de124004-mtsh7 1/1 Running 4 (13m ago) 34h 10.42.0.62 master kube-system svclb-prometheus-alertmanager-de124004-sgfr2 1/1 Running 4 (12m ago) 34h 10.42.1.40 worker kube-system svclb-prometheus-kube-state-metrics-171aeb4a-7fsmj 1/1 Running 4 (12m ago) 34h 10.42.2.32 worker2 kube-system svclb-prometheus-kube-state-metrics-171aeb4a-l4hfh 1/1 Running 4 (12m ago) 34h 10.42.1.36 worker kube-system svclb-prometheus-kube-state-metrics-171aeb4a-v592b 1/1 Running 4 (13m ago) 34h 10.42.0.59 master kube-system svclb-prometheus-prometheus-node-exporter-2cf029b2-2jcj4 1/1 Running 4 (13m ago) 34h 10.42.0.63 master kube-system svclb-prometheus-prometheus-node-exporter-2cf029b2-bbsfj 1/1 Running 4 (12m ago) 34h 10.42.2.31 worker2 kube-system svclb-prometheus-prometheus-node-exporter-2cf029b2-fbht8 1/1 Running 4 (12m ago) 34h 10.42.1.37 worker kube-system svclb-prometheus-prometheus-pushgateway-aac15de8-c7wht 1/1 Running 4 (13m ago) 34h 10.42.0.65 master kube-system svclb-prometheus-prometheus-pushgateway-aac15de8-hkn47 1/1 Running 4 (12m ago) 34h 10.42.2.34 worker2 kube-system svclb-prometheus-prometheus-pushgateway-aac15de8-v88w8 1/1 Running 4 (12m ago) 34h 10.42.1.39 worker kube-system svclb-prometheus-server-544c4cf0-9s99f 0/1 Pending 0 21h kube-system svclb-prometheus-server-544c4cf0-hdhqr 0/1 Pending 0 21h kube-system svclb-prometheus-server-544c4cf0-mxw8f 0/1 Pending 0 21h kube-system svclb-traefik-7bfaae3b-27kj8 2/2 Running 8 (12m ago) 36h 10.42.1.38 worker kube-system svclb-traefik-7bfaae3b-95s7m 2/2 Running 8 (13m ago) 36h 10.42.0.64 master kube-system svclb-traefik-7bfaae3b-sxsn6 2/2 Running 8 (12m ago) 36h 10.42.2.33 worker2 kube-system traefik-67bfb46dcb-9q82m 1/1 Running 4 (12m ago) 36h 10.42.1.34 worker metallb-system controller-bb5f47665-grkt9 1/1 Running 5 (13m ago) 35h 10.42.0.60 master metallb-system speaker-cfqmq 1/1 Running 8 (11m ago) 35h 192.168.121.181 worker2 metallb-system speaker-h95d4 1/1 Running 7 (11m ago) 35h 192.168.121.80 worker metallb-system speaker-k8xhr 1/1 Running 5 (13m ago) 35h 192.168.121.41 master prometheus prometheus-alertmanager-0 1/1 Running 4 (12m ago) 34h 10.42.2.36 worker2 prometheus prometheus-kube-state-metrics-c7cbfcc5d-gbn76 1/1 Running 5 (13m ago) 34h 10.42.0.67 master prometheus prometheus-prometheus-node-exporter-2bs59 1/1 Running 2 (12m ago) 21h 10.42.1.35 worker prometheus prometheus-prometheus-node-exporter-dqp64 1/1 Running 2 (13m ago) 21h 10.42.0.58 master prometheus prometheus-prometheus-node-exporter-z7vcl 1/1 Running 2 (12m ago) 21h 10.42.2.35 worker2 prometheus prometheus-prometheus-pushgateway-79c99bd4b8-mrzqv 1/1 Running 4 (13m ago) 34h 10.42.0.69 master prometheus prometheus-server-564448f5dc-5svvd 2/2 Running 8 (13m ago) 34h 10.42.0.68 master vagrant@Master:~$ - Assistant
From your
kubectl get pods -A -o wideoutput, here are some observations relevant to your issue:- Pods named
svclb-grafana-03ffa6f9-87fdsare in thekube-systemnamespace, and their status isPending. - When you tried to describe or get logs from the pod, it failed with “not found”, likely because you didn't specify the namespace or mistyped the command.
- You tried:
kubectl describe pod svclb-grafana-03ffa6f9-87fds(default namespace assumed, pod not found)kubectl logs pod svclb-grafana-03ffa6f9-87fds -n grafana(wrong namespace)kubectl logs pod svclb-grafana-03ffa6f9-87fds(default namespace again, also invalid command in one trykubectl logs pod svclb-grafana-03ffa6f9-87fds- the error said: pods "pod" not found, which implies you passedpodincorrectly)
How to fix your command to inspect the pod
Since the pods are in
kube-systemnamespace, you should always specify-n kube-systemto operate on these pods.Describe pod
kubectl describe pod svclb-grafana-03ffa6f9-87fds -n kube-systemGet pod logs
kubectl logs svclb-grafana-03ffa6f9-87fds -n kube-systemNote: No need to prefix pod name with
podinkubectl logscommand, just:kubectl logs <pod-name>
Why are the pods Pending?
Your pods
svclb-grafana-03ffa6f9-87fdsare stuck in Pending state with no assigned IP or node. This often means:- There are not enough resources on nodes to schedule the pod.
- There may be issues with node taints or tolerations blocking scheduling.
- PVC or volume issues could prevent scheduling.
- Network plugin or CNI problems.
You can confirm scheduling problems by looking at the events in the output of
kubectl describe:kubectl describe pod svclb-grafana-03ffa6f9-87fds -n kube-systemLook for messages under
Events:at the bottom — for example, errors like:0/3 nodes are available: insufficient memory/cpu.pod has unbound immediate PersistentVolumeClaims.node(s) had taint {some-taint}, that the pod didn't tolerate
Summary and next steps:
- Use correct namespace (
-n kube-system) with your commands:
kubectl describe pod svclb-grafana-03ffa6f9-87fds -n kube-system kubectl logs svclb-grafana-03ffa6f9-87fds -n kube-system-
Check pod events and reason for
Pending. -
Check node resources:
kubectl get nodes -o wide kubectl describe node <node-name>- Check if any taints are applied on nodes:
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'- Check if pods require PVCs or other resources that are unavailable.
Once you identify why pods are stuck in Pending, take necessary actions (e.g., free up resources, fix PVCs, add tolerations, etc.) and pods should schedule successfully.
If you want, provide me with the output of:
kubectl describe pod svclb-grafana-03ffa6f9-87fds -n kube-systemso I can help you analyze the exact scheduling issue.
- Pods named
- User
Events: Type Reason Age From Message
Warning FailedScheduling 19h (x22 over 21h) default-scheduler 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. preemption: 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 Preemption is not helpful for scheduling. Warning FailedScheduling 9m48s (x2 over 14m) default-scheduler 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. preemption: 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 Preemption is not helpful for scheduling. vagrant@Master:~$
- Assistant
The key information from the pod event warning is:
0/3 nodes are available: - 1 node(s) didn't have free ports for the requested pod ports, - 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. preemption: 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 Preemption is not helpful for scheduling.What this means:
- Port conflict: One node is unable to schedule the pod because the pod requires specific ports that are already in use (
didn't have free ports for the requested pod ports). - NodeAffinity failure: Two nodes don't match the node affinity rules defined for this pod.
- Preemption isn't helping: The scheduler tried to evict lower priority pods to free resources but it is ineffective in this case.
Next steps to resolve:
1. Investigate
NodeAffinity- You need to check the pod's affinity rules to understand what constraints it's trying to meet.
- Run:
kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yaml | grep -A10 affinityor look at the full pod spec:
kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yamlLook especially for
nodeAffinityinspec.affinity.nodeAffinity, which limits scheduling to specific nodes or node labels.Check if the labels required actually exist on any nodes:
kubectl get nodes --show-labelsYou might find that certain nodes do not have the required label matching the affinity.
If the node affinity is too strict or incorrect, you may need to adjust or remove the affinity rules in your pod / deployment spec.
2. Investigate port conflicts on node(s)
- A node is rejecting this pod because it does not have free ports that the pod requests.
- This means the pod uses
hostPortsor a DaemonSet with fixed ports, or the service load balancer pods (since prefix issvclb-) expose ports that collide. - Check which ports are requested by the pod:
kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yaml | grep -A10 hostPort(or search for
hostPort:in pod spec).- Then check what pods are running on nodes and what ports they are using that may conflict:
kubectl get pods -A -o wideAnd describe suspicious pods that might be using the same ports.
3. Potential solutions
-
Port conflicts:
- Check if multiple pods request the same
hostPort. This is common with service load balancer pods (svclb-prefix) in K3s or similar setups. - Adjust the pod spec to avoid
hostPortconflicts (e.g., use different ports). - Or scale down/remove conflicting pods.
- Check your service type (LoadBalancer or
NodePort) settings that may trigger those hostPort requests.
- Check if multiple pods request the same
-
Node affinity problems:
- Either add the required labels to nodes or loosen the node affinity rules in pod specs.
- If node affinity isn’t required, remove it.
Why are these pods
svclb-pending?In K3s (lightweight Kubernetes),
svclb-prefixed pods are service load-balancer pods Emulator for LoadBalancer services via NodePorts. They create DaemonSet or deployment pods that bind to NodePorts viahostPorts.If two services or DaemonSets request the same ports on the same nodes, and node affinity restricts where pods can run, this leads to scheduling conflicts and Pending status.
Summary to fix your issue:
- Check pod
svclb-grafana-03ffa6f9-87fdsspec affinity andhostPortusage. - Check nodes’ labels to confirm affinity matches.
- Check for port conflicts from other pods.
- Adjust node affinity or ports to avoid conflicts.
- Possibly scale down conflicting pods or change service configuration.
If you want, share the output of:
- Pod spec affinity section:
kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yaml | grep -A20 affinity- All nodes with labels:
kubectl get nodes --show-labels- Pods requesting
hostPort:
kubectl get pods -A -o yaml | grep hostPort -B 10I can help analyze and suggest precise fixes.
- Port conflict: One node is unable to schedule the pod because the pod requires specific ports that are already in use (
- User
vagrant@Master:~
vagrant@Master:~
- Assistant
Thanks for the info! Let's analyze what you have:
Affinity snippet:
affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchFields: - key: metadata.name operator: In values: - workerInterpretation:
This means the pod can only be scheduled on a node with the exact nameworker. Kubernetes node names here are important.
Key insight about nodes:
Check the nodes you have:
kubectl get nodes --no-headers -o custom-columns=NAME:.metadata.nameYou probably have nodes named:
masterworkerworker2
HostPort snippet:
hostPort: 80 name: lb-tcp-80 protocol: TCPThis means the pod requests port 80 on the node’s host network.
What the scheduler error said:
- 1 node doesn't have free ports for the pod ports (hostPort 80 is used on that node).
- 2 nodes don’t satisfy nodeAffinity.
Since nodeAffinity restricts scheduling only to node
worker, but you have three nodes (master,worker,worker2), and 2 nodes fail affinity, it means these aremasterandworker2which are excluded.Only node
workermatches affinity, but port 80 onworkeris already occupied by some other pod.
How to diagnose further:
- Find which pod is using hostPort 80 on node
worker:
kubectl get pods -o wide -A --field-selector spec.nodeName=workerLook for pods that might be exposing
hostPort80.Or more directly:
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.nodeName=="worker") | {name: .metadata.name, namespace: .metadata.namespace, hostPorts: [.spec.containers[].ports[]?.hostPort] }'(If
jqis not available, just look manually.)
How to fix this:
Option A: Free port 80 on node
worker- Identify and delete or modify pod that occupies port 80 on
worker.
Option B: Loosen node affinity
- If you want the pod to be schedulable on nodes
worker2ormaster, then change affinity to include these nodes or remove affinity altogether.
Example to allow all nodes:
affinity: {}Or to include multiple nodes:
affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchFields: - key: metadata.name operator: In values: - worker - worker2 - master(Ensure node names are correctly specified and exist.)
Additional suggestions:
- Using
hostPort: 80limits pod placement because only one pod per node can bind that port. - Consider revising your architecture to avoid
hostPortusage if possible. For example, useNodePortservices or ingress with load balancer instead. - If these are load balancer pods (
svclb-prefix), they may come from k3s' service load balancer. You might need to configure the load balancer settings to use different ports or allow scheduling on more nodes.
Summary:
Your pod
svclb-grafana-03ffa6f9-87fdsis stuck in Pending because:- It can only schedule on node
worker(due to strict nodeAffinity) - Port 80 is already taken on node
workerby another pod, so no available port for this pod
Next recommended commands:
- See pods on
workerto find what’s using port 80:
kubectl get pods -o wide -A --field-selector spec.nodeName=worker- Find pods using hostPort 80:
for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do kubectl get pods -n $ns -o json | jq -r '.items[] | select(.spec.nodeName=="worker") | select(.spec.containers[].ports[]?.hostPort==80) | "\(.metadata.namespace)/\(.metadata.name)"' done- Consider updating node affinity or hostPort usage as explained above.
If you’d like, I can help you craft an updated pod spec or diagnose the pod blocking port 80 on worker node. Just share your findings or pod specs!