Skip to content
TheChatArchiveThe Chat Archive
gpt-4.1-mini-2025-04-14· OpenAIAi2 WildChat-4.8M (AI2)

vagrant@Master:~$ kubectl describe pod svclb-grafana-03ffa6f9-87fds Error from server…

  1. User

    vagrant@Master:kubectldescribepodsvclb−grafana−03ffa6f9−87fdsErrorfromserver(NotFound):pods"svclb−grafana−03ffa6f9−87fds"notfoundvagrant@Master:  kubectl describe pod svclb-grafana-03ffa6f9-87fds Error from server (NotFound): pods "svclb-grafana-03ffa6f9-87fds" not found vagrant@Master:~ kubectl logs pod svclb-grafana-03ffa6f9-87fds error: error from server (NotFound): pods "pod" not found in namespace "default" vagrant@Master:kubectllogspodsvclb−grafana−03ffa6f9−87fds−ngrafanaerror:errorfromserver(NotFound):pods"pod"notfoundinnamespace"grafana"vagrant@Master:  kubectl logs pod svclb-grafana-03ffa6f9-87fds -n grafana error: error from server (NotFound): pods "pod" not found in namespace "grafana" vagrant@Master:~ kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES grafana grafana-5744c94549-ksw8q 1/1 Running 4 (12m ago) 34h 10.42.1.33 worker kube-system coredns-ff8999cc5-txw22 1/1 Running 4 (13m ago) 36h 10.42.0.66 master kube-system local-path-provisioner-774c6665dc-7njhl 1/1 Running 5 (13m ago) 36h 10.42.0.57 master kube-system metrics-server-6f4c6675d5-qjlhq 1/1 Running 5 (13m ago) 36h 10.42.0.61 master kube-system svclb-grafana-03ffa6f9-87fds 0/1 Pending 0 21h kube-system svclb-grafana-03ffa6f9-9qsqq 0/1 Pending 0 21h kube-system svclb-grafana-03ffa6f9-f5nrh 0/1 Pending 0 21h kube-system svclb-prometheus-alertmanager-de124004-c6tk2 1/1 Running 4 (12m ago) 34h 10.42.2.30 worker2 kube-system svclb-prometheus-alertmanager-de124004-mtsh7 1/1 Running 4 (13m ago) 34h 10.42.0.62 master kube-system svclb-prometheus-alertmanager-de124004-sgfr2 1/1 Running 4 (12m ago) 34h 10.42.1.40 worker kube-system svclb-prometheus-kube-state-metrics-171aeb4a-7fsmj 1/1 Running 4 (12m ago) 34h 10.42.2.32 worker2 kube-system svclb-prometheus-kube-state-metrics-171aeb4a-l4hfh 1/1 Running 4 (12m ago) 34h 10.42.1.36 worker kube-system svclb-prometheus-kube-state-metrics-171aeb4a-v592b 1/1 Running 4 (13m ago) 34h 10.42.0.59 master kube-system svclb-prometheus-prometheus-node-exporter-2cf029b2-2jcj4 1/1 Running 4 (13m ago) 34h 10.42.0.63 master kube-system svclb-prometheus-prometheus-node-exporter-2cf029b2-bbsfj 1/1 Running 4 (12m ago) 34h 10.42.2.31 worker2 kube-system svclb-prometheus-prometheus-node-exporter-2cf029b2-fbht8 1/1 Running 4 (12m ago) 34h 10.42.1.37 worker kube-system svclb-prometheus-prometheus-pushgateway-aac15de8-c7wht 1/1 Running 4 (13m ago) 34h 10.42.0.65 master kube-system svclb-prometheus-prometheus-pushgateway-aac15de8-hkn47 1/1 Running 4 (12m ago) 34h 10.42.2.34 worker2 kube-system svclb-prometheus-prometheus-pushgateway-aac15de8-v88w8 1/1 Running 4 (12m ago) 34h 10.42.1.39 worker kube-system svclb-prometheus-server-544c4cf0-9s99f 0/1 Pending 0 21h kube-system svclb-prometheus-server-544c4cf0-hdhqr 0/1 Pending 0 21h kube-system svclb-prometheus-server-544c4cf0-mxw8f 0/1 Pending 0 21h kube-system svclb-traefik-7bfaae3b-27kj8 2/2 Running 8 (12m ago) 36h 10.42.1.38 worker kube-system svclb-traefik-7bfaae3b-95s7m 2/2 Running 8 (13m ago) 36h 10.42.0.64 master kube-system svclb-traefik-7bfaae3b-sxsn6 2/2 Running 8 (12m ago) 36h 10.42.2.33 worker2 kube-system traefik-67bfb46dcb-9q82m 1/1 Running 4 (12m ago) 36h 10.42.1.34 worker metallb-system controller-bb5f47665-grkt9 1/1 Running 5 (13m ago) 35h 10.42.0.60 master metallb-system speaker-cfqmq 1/1 Running 8 (11m ago) 35h 192.168.121.181 worker2 metallb-system speaker-h95d4 1/1 Running 7 (11m ago) 35h 192.168.121.80 worker metallb-system speaker-k8xhr 1/1 Running 5 (13m ago) 35h 192.168.121.41 master prometheus prometheus-alertmanager-0 1/1 Running 4 (12m ago) 34h 10.42.2.36 worker2 prometheus prometheus-kube-state-metrics-c7cbfcc5d-gbn76 1/1 Running 5 (13m ago) 34h 10.42.0.67 master prometheus prometheus-prometheus-node-exporter-2bs59 1/1 Running 2 (12m ago) 21h 10.42.1.35 worker prometheus prometheus-prometheus-node-exporter-dqp64 1/1 Running 2 (13m ago) 21h 10.42.0.58 master prometheus prometheus-prometheus-node-exporter-z7vcl 1/1 Running 2 (12m ago) 21h 10.42.2.35 worker2 prometheus prometheus-prometheus-pushgateway-79c99bd4b8-mrzqv 1/1 Running 4 (13m ago) 34h 10.42.0.69 master prometheus prometheus-server-564448f5dc-5svvd 2/2 Running 8 (13m ago) 34h 10.42.0.68 master vagrant@Master:~$

  2. Assistant

    From your kubectl get pods -A -o wide output, here are some observations relevant to your issue:

    1. Pods named svclb-grafana-03ffa6f9-87fds are in the kube-system namespace, and their status is Pending.
    2. When you tried to describe or get logs from the pod, it failed with “not found”, likely because you didn't specify the namespace or mistyped the command.
    3. You tried:
      • kubectl describe pod svclb-grafana-03ffa6f9-87fds (default namespace assumed, pod not found)
      • kubectl logs pod svclb-grafana-03ffa6f9-87fds -n grafana (wrong namespace)
      • kubectl logs pod svclb-grafana-03ffa6f9-87fds (default namespace again, also invalid command in one try kubectl logs pod svclb-grafana-03ffa6f9-87fds - the error said: pods "pod" not found, which implies you passed pod incorrectly)

    How to fix your command to inspect the pod

    Since the pods are in kube-system namespace, you should always specify -n kube-system to operate on these pods.

    Describe pod

    kubectl describe pod svclb-grafana-03ffa6f9-87fds -n kube-system
    

    Get pod logs

    kubectl logs svclb-grafana-03ffa6f9-87fds -n kube-system
    

    Note: No need to prefix pod name with pod in kubectl logs command, just:

    kubectl logs <pod-name>
    

    Why are the pods Pending?

    Your pods svclb-grafana-03ffa6f9-87fds are stuck in Pending state with no assigned IP or node. This often means:

    • There are not enough resources on nodes to schedule the pod.
    • There may be issues with node taints or tolerations blocking scheduling.
    • PVC or volume issues could prevent scheduling.
    • Network plugin or CNI problems.

    You can confirm scheduling problems by looking at the events in the output of kubectl describe:

    kubectl describe pod svclb-grafana-03ffa6f9-87fds -n kube-system
    

    Look for messages under Events: at the bottom — for example, errors like:

    • 0/3 nodes are available: insufficient memory/cpu.
    • pod has unbound immediate PersistentVolumeClaims.
    • node(s) had taint {some-taint}, that the pod didn't tolerate

    Summary and next steps:

    1. Use correct namespace (-n kube-system) with your commands:
    kubectl describe pod svclb-grafana-03ffa6f9-87fds -n kube-system
    kubectl logs svclb-grafana-03ffa6f9-87fds -n kube-system
    
    1. Check pod events and reason for Pending.

    2. Check node resources:

    kubectl get nodes -o wide
    kubectl describe node <node-name>
    
    1. Check if any taints are applied on nodes:
    kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints}{"\n"}{end}'
    
    1. Check if pods require PVCs or other resources that are unavailable.

    Once you identify why pods are stuck in Pending, take necessary actions (e.g., free up resources, fix PVCs, add tolerations, etc.) and pods should schedule successfully.


    If you want, provide me with the output of:

    kubectl describe pod svclb-grafana-03ffa6f9-87fds -n kube-system
    

    so I can help you analyze the exact scheduling issue.

  3. User

    Events: Type Reason Age From Message


    Warning FailedScheduling 19h (x22 over 21h) default-scheduler 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. preemption: 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 Preemption is not helpful for scheduling. Warning FailedScheduling 9m48s (x2 over 14m) default-scheduler 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. preemption: 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 Preemption is not helpful for scheduling. vagrant@Master:~$

  4. Assistant

    The key information from the pod event warning is:

    0/3 nodes are available:
    - 1 node(s) didn't have free ports for the requested pod ports,
    - 2 node(s) didn't satisfy plugin(s) [NodeAffinity].
    preemption: 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports,
    2 Preemption is not helpful for scheduling.
    

    What this means:

    • Port conflict: One node is unable to schedule the pod because the pod requires specific ports that are already in use (didn't have free ports for the requested pod ports).
    • NodeAffinity failure: Two nodes don't match the node affinity rules defined for this pod.
    • Preemption isn't helping: The scheduler tried to evict lower priority pods to free resources but it is ineffective in this case.

    Next steps to resolve:

    1. Investigate NodeAffinity

    • You need to check the pod's affinity rules to understand what constraints it's trying to meet.
    • Run:
    kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yaml | grep -A10 affinity
    

    or look at the full pod spec:

    kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yaml
    

    Look especially for nodeAffinity in spec.affinity.nodeAffinity, which limits scheduling to specific nodes or node labels.

    Check if the labels required actually exist on any nodes:

    kubectl get nodes --show-labels
    

    You might find that certain nodes do not have the required label matching the affinity.

    If the node affinity is too strict or incorrect, you may need to adjust or remove the affinity rules in your pod / deployment spec.


    2. Investigate port conflicts on node(s)

    • A node is rejecting this pod because it does not have free ports that the pod requests.
    • This means the pod uses hostPorts or a DaemonSet with fixed ports, or the service load balancer pods (since prefix is svclb-) expose ports that collide.
    • Check which ports are requested by the pod:
    kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yaml | grep -A10 hostPort
    

    (or search for hostPort: in pod spec).

    • Then check what pods are running on nodes and what ports they are using that may conflict:
    kubectl get pods -A -o wide
    

    And describe suspicious pods that might be using the same ports.


    3. Potential solutions

    • Port conflicts:

      • Check if multiple pods request the same hostPort. This is common with service load balancer pods (svclb- prefix) in K3s or similar setups.
      • Adjust the pod spec to avoid hostPort conflicts (e.g., use different ports).
      • Or scale down/remove conflicting pods.
      • Check your service type (LoadBalancer or NodePort) settings that may trigger those hostPort requests.
    • Node affinity problems:

      • Either add the required labels to nodes or loosen the node affinity rules in pod specs.
      • If node affinity isn’t required, remove it.

    Why are these pods svclb- pending?

    In K3s (lightweight Kubernetes), svclb- prefixed pods are service load-balancer pods Emulator for LoadBalancer services via NodePorts. They create DaemonSet or deployment pods that bind to NodePorts via hostPorts.

    If two services or DaemonSets request the same ports on the same nodes, and node affinity restricts where pods can run, this leads to scheduling conflicts and Pending status.


    Summary to fix your issue:

    1. Check pod svclb-grafana-03ffa6f9-87fds spec affinity and hostPort usage.
    2. Check nodes’ labels to confirm affinity matches.
    3. Check for port conflicts from other pods.
    4. Adjust node affinity or ports to avoid conflicts.
    5. Possibly scale down conflicting pods or change service configuration.

    If you want, share the output of:

    • Pod spec affinity section:
    kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yaml | grep -A20 affinity
    
    • All nodes with labels:
    kubectl get nodes --show-labels
    
    • Pods requesting hostPort:
    kubectl get pods -A -o yaml | grep hostPort -B 10
    

    I can help analyze and suggest precise fixes.

  5. User

    vagrant@Master:~kubectlgetpodsvclb−grafana−03ffa6f9−87fds−nkube−system−oyaml∣grep−A10affinityaffinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:−matchFields:−key:metadata.nameoperator:Invalues:−workerautomountServiceAccountToken:falsecontainers:vagrant@Master:  kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yaml | grep -A10 affinity affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchFields: - key: metadata.name operator: In values: - worker automountServiceAccountToken: false containers: vagrant@Master:~

    vagrant@Master:~kubectlgetpodsvclb−grafana−03ffa6f9−87fds−nkube−system−oyaml∣grep−A10hostPorthostPort:80name:lb−tcp−80protocol:TCPresources:securityContext:capabilities:add:−NETADMINterminationMessagePath:/dev/termination−logterminationMessagePolicy:FilednsPolicy:ClusterFirstvagrant@Master:  kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yaml | grep -A10 hostPort hostPort: 80 name: lb-tcp-80 protocol: TCP resources: {} securityContext: capabilities: add: - NET_ADMIN terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst vagrant@Master:~

  6. Assistant

    Thanks for the info! Let's analyze what you have:


    Affinity snippet:

    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchFields:
            - key: metadata.name
              operator: In
              values:
              - worker
    

    Interpretation:
    This means the pod can only be scheduled on a node with the exact name worker. Kubernetes node names here are important.


    Key insight about nodes:

    Check the nodes you have:

    kubectl get nodes --no-headers -o custom-columns=NAME:.metadata.name
    

    You probably have nodes named:

    • master
    • worker
    • worker2

    HostPort snippet:

    hostPort: 80
    name: lb-tcp-80
    protocol: TCP
    

    This means the pod requests port 80 on the node’s host network.


    What the scheduler error said:

    • 1 node doesn't have free ports for the pod ports (hostPort 80 is used on that node).
    • 2 nodes don’t satisfy nodeAffinity.

    Since nodeAffinity restricts scheduling only to node worker, but you have three nodes (master, worker, worker2), and 2 nodes fail affinity, it means these are master and worker2 which are excluded.

    Only node worker matches affinity, but port 80 on worker is already occupied by some other pod.


    How to diagnose further:

    1. Find which pod is using hostPort 80 on node worker:
    kubectl get pods -o wide -A --field-selector spec.nodeName=worker
    

    Look for pods that might be exposing hostPort 80.

    Or more directly:

    kubectl get pods -A -o json | jq -r '.items[] | select(.spec.nodeName=="worker") | {name: .metadata.name, namespace: .metadata.namespace, hostPorts: [.spec.containers[].ports[]?.hostPort] }'
    

    (If jq is not available, just look manually.)


    How to fix this:

    Option A: Free port 80 on node worker

    • Identify and delete or modify pod that occupies port 80 on worker.

    Option B: Loosen node affinity

    • If you want the pod to be schedulable on nodes worker2 or master, then change affinity to include these nodes or remove affinity altogether.

    Example to allow all nodes:

    affinity: {}
    

    Or to include multiple nodes:

    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchFields:
            - key: metadata.name
              operator: In
              values:
              - worker
              - worker2
              - master
    

    (Ensure node names are correctly specified and exist.)


    Additional suggestions:

    • Using hostPort: 80 limits pod placement because only one pod per node can bind that port.
    • Consider revising your architecture to avoid hostPort usage if possible. For example, use NodePort services or ingress with load balancer instead.
    • If these are load balancer pods (svclb- prefix), they may come from k3s' service load balancer. You might need to configure the load balancer settings to use different ports or allow scheduling on more nodes.

    Summary:

    Your pod svclb-grafana-03ffa6f9-87fds is stuck in Pending because:

    • It can only schedule on node worker (due to strict nodeAffinity)
    • Port 80 is already taken on node worker by another pod, so no available port for this pod

    Next recommended commands:

    1. See pods on worker to find what’s using port 80:
    kubectl get pods -o wide -A --field-selector spec.nodeName=worker
    
    1. Find pods using hostPort 80:
    for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
      kubectl get pods -n $ns -o json | jq -r '.items[] | select(.spec.nodeName=="worker") | select(.spec.containers[].ports[]?.hostPort==80) | "\(.metadata.namespace)/\(.metadata.name)"'
    done
    
    1. Consider updating node affinity or hostPort usage as explained above.

    If you’d like, I can help you craft an updated pod spec or diagnose the pod blocking port 80 on worker node. Just share your findings or pod specs!