Kubernetes errors and how to fix them

The image name or tag is wrong, or the tag was never pushed. The registry has nothing under that name: not found, and on Docker Hub often pull access denied, repository does not exist or may require authorization.

kubectl get pod <pod> -o jsonpath='{.spec.containers[*].image}'   # compare it letter by letter with what you pushed
docker manifest inspect <image>:<tag> >/dev/null && echo exists   # from a machine logged in to that registry: fails here too = the name or tag is wrong

Correct the image on the Deployment, not the Pod: kubectl set image deployment/<name> <container>=<image>:<tag>.

The image is private and no pull Secret reaches the Pod. The registry refuses an anonymous pull with 401 Unauthorized or 403 Forbidden; Docker Hub gives the same pull access denied as for a typo.

kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets}'   # empty: no pull Secret reaches this Pod
docker manifest inspect <image>:<tag> >/dev/null && echo exists   # works from a logged-in machine: the image exists, access is missing

Create a pull Secret in the Pod's namespace (kubectl create secret docker-registry <name> --docker-server=<registry> ...) and list it under imagePullSecrets, or on the namespace's ServiceAccount.

A pull Secret exists but is not used: it is in another namespace, or written for another registry host. The kubelet only sends credentials whose host matches the image's registry, so the pull is still anonymous.

kubectl get secret <pull-secret> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq -r '.auths | keys[]'   # registry hosts only, no passwords: none matches the image's registry = this cause
kubectl get secret <pull-secret> -n <pod-namespace>   # NotFound: it lives in another namespace

Recreate it in the Pod's namespace with --docker-server set to the exact registry host in the image name.

The registry is rate-limiting the node. Docker Hub answers with 429 Too Many Requests and toomanyrequests.

kubectl describe pod <pod> | grep -i toomanyrequests   # present: this cause

See ImagePullBackOff: 429 toomanyrequests: pull through a mirror, or authenticate the pulls.

The tag has no image for the node's CPU architecture. The runtime finds the tag but no matching platform: no match for platform in manifest.

kubectl get nodes -L kubernetes.io/arch
docker buildx imagetools inspect <image>:<tag>   # the node's architecture missing from Platform: this cause

Build and push the image for every architecture you run: docker buildx build --platform linux/amd64,linux/arm64 -t <image> --push ..

The ConfigMap or Secret does not exist in the Pod's namespace. A Pod can only use its own namespace's objects, so one in another namespace counts as missing: configmap "<name>" not found.

kubectl describe pod <pod>   # the Failed event names it
kubectl get configmap,secret -n <pod-namespace>   # not listed: this cause

Create it in the Pod's namespace, or correct the name the Pod references.

The object exists but not with that key. Keys are case sensitive, so DB_PASSWORD and db-password differ: couldn't find key DB_PASSWORD in Secret.

kubectl describe secret <name>   # key names and sizes, never the values: the key is not there
kubectl get configmap <name> -o yaml   # for a ConfigMap

Use the key exactly as stored, or add it to the Secret or ConfigMap. Mark a reference optional: true only when the app can run without it.

runAsNonRoot is set and the image runs as root. The kubelet refuses to start it: container has runAsNonRoot and image will run as root.

kubectl get pod <pod> -o jsonpath='{.spec.securityContext}{.spec.containers[0].securityContext}'   # runAsNonRoot true, no runAsUser

Set a numeric runAsUser (and fix file ownership in the image), or use an image that runs as a non-root user.

runAsNonRoot is set and the image's USER is a name, not a number. The kubelet cannot tell whether the name is root: image has non-numeric user (nginx), cannot verify user is non-root.

kubectl describe pod <pod>   # non-numeric user in the event

Set runAsUser to that user's number, or change USER in the Dockerfile to a number.

A typo in command or args. The runtime looks for that executable and it does not exist: executable file not found in $PATH.

kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[0].command}'   # the misspelt name

Correct command or args on the Deployment to a binary that exists in the image.

A script without execute permission, or with Windows line endings. The runtime cannot run it: permission denied, or no such file (for the interpreter with a hidden \r).

kubectl describe pod <pod>   # permission denied, or no such file or directory

Make it executable in the image (chmod +x in the Dockerfile) and save it with Unix line endings.

The binary is not in the image, for example a slim or distroless base without a shell.

docker run --rm --entrypoint ls <image> /usr/local/bin   # the binary missing (fails on images with no ls at all)

Use the image's own ENTRYPOINT, or add the binary to the image.

The image was built for another architecture, for example on an Apple Silicon laptop (arm64) and run on amd64 nodes.

kubectl get nodes -L kubernetes.io/arch
docker buildx imagetools inspect <image>   # the node's architecture not under Platform: this cause

Build for the nodes, or for both: docker buildx build --platform linux/amd64,linux/arm64 -t <image> --push ..

A single-architecture build was pushed over a multi-arch tag. Nodes of the other architecture now get an image they cannot run.

docker buildx imagetools inspect <image>   # one platform where there were two

Push the multi-arch build again, and never push a single-architecture build to that tag.

A mixed cluster without a nodeSelector. The Pod lands on a node of an architecture the image lacks.

kubectl get pod <pod> -o wide; kubectl get nodes -L kubernetes.io/arch   # failing Pods only on one architecture

Select the right nodes with a nodeSelector on kubernetes.io/arch, or build for both.

The command is a script with no #! line. The kernel cannot run it on any architecture.

docker run --rm --entrypoint head <image> -n 1 <script>   # platforms match and no #!/bin/sh: this cause

Add #!/bin/sh (or its interpreter) as the first line, or run it through the interpreter in command.

It waits for a Service or DNS name that does not exist. A wait loop such as until nslookup db never ends, so the Pod stays at Init:0/1.

kubectl logs <pod> -c <init-container>   # the same wait message over and over
kubectl get svc -A | grep <name>   # missing, or in another namespace

Create the Service, or correct the name it waits for (<svc>.<namespace> across namespaces).

It waits for a dependency that exists but is down. The Service is there with no Ready endpoints, so the check never succeeds.

kubectl get endpointslices -l kubernetes.io/service-name=<svc>   # no ready endpoints: fix that app first

Fix the dependency first; the init container moves on by itself once it answers.

Its own command fails. It exits with an error and the kubelet restarts it: Init:CrashLoopBackOff or Init:Error.

kubectl logs <pod> -c <init-container> --previous   # an error and a non-zero exit
kubectl describe pod <pod>   # Init Containers: Last State, Exit Code

Fix the command or what it reads; init containers must exit 0 for the Pod to go on.

A ConfigMap or Secret mounted as a volume does not exist. The kubelet retries the mount and never finishes: FailedMount, secret "<name>" not found.

kubectl describe pod <pod>   # FailedMount names the volume and the object
kubectl get secrets,configmaps   # in the Pod's namespace: not listed = this cause

Create the Secret or ConfigMap in the Pod's namespace, or correct its name in the volume.

A ReadWriteOnce volume is still attached to another node. It can attach to one node at a time: FailedAttachVolume, Multi-Attach error.

kubectl describe pod <pod>   # Multi-Attach error or FailedAttachVolume

See Multi-Attach error for volume: the old Pod or node still holds the volume.

The network plugin cannot give the Pod an IP. The kubelet cannot create the Pod sandbox: FailedCreatePodSandBox.

kubectl describe pod <pod>   # FailedCreatePodSandBox, failed to setup network for sandbox

See FailedCreatePodSandBox (CNI): look at the CNI Pod on that node.

A RollingUpdate starts the new Pod before the old one stops. The old Pod keeps the volume on its node.

kubectl get pods -o wide   # old and new Pod on different nodes, both present

For a single-replica app on a ReadWriteOnce volume, use Recreate: kubectl patch deployment <name> -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'.

The old node died, and the volume is not detached yet. The controller waits about six minutes before forcing it.

kubectl get nodes   # the old Pod's node NotReady

Wait, or once the node is shut down for good, kubectl taint node <node> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute (Kubernetes 1.28 and later) so its volumes detach at once.

Several replicas share one ReadWriteOnce claim. Only the replicas on one node can mount it.

kubectl get deploy <name>   # replicas above 1, one claimName in the template

Use a StatefulSet with one claim per replica, or ReadWriteMany storage.

Many nodes behind one NAT address pull at once (new nodes, a scale-out, an upgrade). They share one anonymous limit.

kubectl get events -A | grep -c toomanyrequests   # many at the same time, on many nodes

Pull through a mirror or your own registry: kubectl set image deployment/<name> <container>=<mirror>/library/<image>:<tag>.

A pull Secret that is not used, so pulls stay anonymous. The message says unauthenticated pull rate limit even though you have an account.

kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets}'   # empty: the login never reaches the Pod

Add the docker-registry Secret (for docker.io) to imagePullSecrets, or to the namespace's ServiceAccount.

Images are referenced without a registry, so they come from Docker Hub. Each new node pulls them again.

kubectl get pods -A -o jsonpath='{..image}' | tr ' ' '\n' | sort -u   # names without a registry host

Reference images by your mirror's name. Deleting Pods to retry only spends more of the limit.

imagePullPolicy: Always on Pods that restart often. Each restart asks the registry again.

kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[*].imagePullPolicy}'   # Always

Use IfNotPresent with immutable tags.

The profile was loaded on some nodes only. Pods work where it is and fail where it is not.

kubectl get pods -o wide   # the failing Pods share nodes
ssh <node> sudo aa-status | grep <profile>   # missing on those nodes: this cause

Load it on every node: sudo apparmor_parser -q /etc/apparmor.d/<profile>.

The node rebooted and the profile is not loaded at boot. It was loaded by hand once.

ssh <node> 'ls /etc/apparmor.d/<profile>; sudo aa-status | grep <profile>'   # file present, not loaded

Install it where the AppArmor service loads it at boot, through the node image or a DaemonSet.

A typo in localhostProfile. The name asked for is not the one loaded.

kubectl get pod <pod> -o jsonpath='{.spec.containers[0].securityContext.appArmorProfile}'   # compare with aa-status

Correct localhostProfile to the profile's exact name.

The CNI plugin's own Pod (calico-node, kube-flannel, cilium, aws-node) is not running on that node. Its files are missing, so the plugin call fails.

kubectl -n kube-system get pods -o wide | grep -Ei 'calico|flannel|cilium|aws-node'   # not Running on that node: this cause

Read its logs (kubectl -n kube-system logs <cni-pod>) and delete it so its DaemonSet starts a fresh one.

No CNI installed yet. On a new kubeadm cluster the nodes stay NotReady: cni plugin not initialized.

kubectl describe node <node> | grep -i network   # NetworkPluginNotReady

Install the plugin the cluster was built for, with a Pod CIDR that matches kubeadm's --pod-network-cidr.

Plugin binaries are missing from /opt/cni/bin, or the config in /etc/cni/net.d is missing or broken.

ls /etc/cni/net.d /opt/cni/bin   # on the node: empty or missing the plugin
sudo journalctl -u kubelet | grep -i cni

Restart or reinstall the plugin's DaemonSet, which writes both on each node.

The plugin has run out of Pod IPs for that node.

kubectl describe pod <pod>   # no IP addresses available, or failed to allocate

Free addresses or give the plugin a bigger pool, as its documentation describes.

The app fails at start: a missing setting, a bad config file, a database it cannot reach. It logs an error and exits, usually with code 1 or 2.

kubectl logs <pod> --previous   # an error just before the end: this cause
kubectl describe pod <pod>   # Last State: Terminated, Exit Code 1

Fix the configuration it names on the Deployment (kubectl set env deployment/<name> KEY=value, or the ConfigMap), so a new rollout replaces the Pods.

The command or args are wrong. The process cannot even start, so the logs are empty: StartError with exit code 128, or 127 when a shell cannot find the command.

kubectl describe pod <pod>   # Reason StartError, Exit Code 128, executable file not found
kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[0].command}'

Correct command or args; see StartError, exit code 128, empty logs.

The container ran out of memory. The kernel killed it at its memory limit, so it exits with 137 and Reason OOMKilled.

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState}'   # reason OOMKilled, exitCode 137

Raise the memory limit above the real working set; see OOMKilled.

A liveness probe kills a healthy app. The kubelet restarts the container when the probe fails, so it looks like a crash.

kubectl describe pod <pod>   # Liveness probe failed and Killing events
kubectl logs <pod> --previous   # a clean shutdown, not an error

Fix the probe, not the app; see Liveness probe failed.

The probe uses the wrong port or path. It fails every time, so the container restarts on a steady rhythm.

kubectl get pod <pod> -o yaml | grep -A8 livenessProbe   # compare port and path with what the app serves
kubectl exec <pod> -- wget -qO- -T 2 http://localhost:<port><path>   # fails here too: wrong port or path

Point the probe at the port and path the app really serves, in the Deployment's Pod template.

No startupProbe, so the liveness probe starts before a slow app is up. It is killed during start and never gets there.

kubectl describe pod <pod>   # Killing within the first minute, every run
kubectl logs <pod> --previous   # ends while still starting up

Add a startupProbe with enough failureThreshold x periodSeconds for the slowest start; liveness waits until it passes.

timeoutSeconds or failureThreshold is too strict for a busy app. Under load the answer takes longer than the timeout (1 second by default).

kubectl describe pod <pod>   # context deadline exceeded or Client.Timeout in the Unhealthy event, mostly under load

Raise timeoutSeconds and failureThreshold so a slow answer under load does not count as dead.

The probe checks a dependency (a database, another service) instead of the process. When the dependency is down, every replica is killed at once.

kubectl exec <pod> -- wget -qO- http://localhost:<port><path>   # the response mentions the database or another service

Make liveness check only the process itself; put dependency checks in the readiness probe.

The memory limit is lower than the app's real working set. Each time use reaches the limit, the kernel kills it.

kubectl describe pod <pod>   # Last State OOMKilled, Exit Code 137
kubectl top pod <pod>   # use climbing towards the limit before each restart

Raise the limit above the measured peak: kubectl set resources deployment/<name> --limits=memory=<size> --requests=memory=<size>.

The manifest sets no limit, so the namespace's LimitRange default applies, often a small one. The limit came from the namespace, not you.

kubectl describe limitrange   # a Default Limit for memory
kubectl describe pod <pod>   # an annotation saying LimitRanger set the limit

Set requests and limits explicitly in the manifest.

A runtime heap (JVM, Node.js) is set larger than the container limit. The runtime grows past the limit before it collects garbage.

kubectl exec <pod> -- printenv JAVA_TOOL_OPTIONS JAVA_OPTS NODE_OPTIONS   # -Xmx or --max-old-space-size above the limit: this cause

Cap the heap below the limit, for example -XX:MaxRAMPercentage=75 for the JVM.

A memory leak. Use grows steadily until the limit, whatever size you give it.

kubectl top pod <pod>   # rises without levelling off, run after run

Raise the limit only to buy time; find the leak with the runtime's heap tools.

The node ran out of memory, not the container. When Pods use far more than they request, the kernel can kill a container below its own limit.

kubectl get events -A --field-selector reason=SystemOOM   # node-level OOM: this cause

Set memory requests close to real use, so the scheduler stops packing the node beyond its memory.

The probe's path or port is wrong. The app answers 404, or nothing listens on that port.

kubectl get pod <pod> -o yaml | grep -A8 readinessProbe   # compare with the app's real port and path
kubectl describe pod <pod>   # statuscode: 404 or connection refused

Correct the probe in the Deployment's Pod template (your manifest, or kubectl edit deployment/<name>), so new Pods roll out with it.

The app needs longer to start than the probe allows. It fails until the app is up; that is normal for the first seconds, not for minutes.

kubectl get pod <pod> -w   # becomes 1/1 after a while: this cause
kubectl logs <pod>   # still starting up

Add a startupProbe for slow starts, rather than a long initialDelaySeconds.

A dependency the readiness check needs is down. The app reports itself not ready on purpose (often 503).

kubectl exec <pod> -- wget -qO- http://localhost:<port><path>   # names a database or service
kubectl logs <pod>   # if the app logs requests, the probe's show up here

Fix the dependency. If every replica goes unready together, consider whether readiness should depend on it at all.

The value reaches the app through env or envFrom. Environment variables are fixed when the container starts.

kubectl get pod <pod> -o jsonpath='{.status.startTime}'   # started before the change: this cause
kubectl exec <pod> -- printenv <KEY>   # a ConfigMap value: still the old one

Restart the Pods: kubectl rollout restart deployment/<name>, then kubectl rollout status deployment/<name>.

The file is mounted and updates after the kubelet's sync (often a minute or more), but the app only reads it at start.

kubectl exec <pod> -- cat /etc/<app>/<file>   # a ConfigMap file: new content, old behaviour = the app does not reread

Restart the Pods, or make the app watch the file.

The file is mounted with subPath. subPath mounts never update.

kubectl get deploy <name> -o yaml | grep -B2 subPath

Mount the whole volume as a directory instead, or restart the Pods on each change.

For a Secret, checking the value would print it. Compare lengths instead.

kubectl exec <pod> -- sh -c 'printf %s "$KEY" | wc -c'   # differs from the new value's length: the Pod has the old one

Restart the Pods as above; do not print Secret values to check them.

The value was encoded twice: a base64 value put under stringData:, or passed to --from-literal, which encode it again. The app sends a base64 string as its password.

kubectl get secret <name> -o jsonpath='{.data.<KEY>}' | base64 -d | wc -c   # length only: about 4/3 of the password's, often ending in =, means encoded twice

Recreate it from the plain value: kubectl create secret generic <name> --from-file=KEY=<file> --dry-run=client -o yaml | kubectl apply -f -, with the file written by printf %s (no newline), which also keeps the password out of shell history.

echo without -n added a newline before encoding. The password the app sends ends in a newline.

kubectl get secret <name> -o jsonpath='{.data.<KEY>}' | base64 -d | tail -c 1 | od -c   # \n: a trailing newline

Recreate it as above, and use echo -n or printf %s when encoding by hand.

The Secret was changed but the app read it at start. It still sends the old password.

kubectl exec <pod> -- sh -c 'printf %s "$DB_PASSWORD" | wc -c'   # differs from the stored value's length: the Pod has the old one

Restart the Pods: kubectl rollout restart deployment/<name>.

The Secret is right and the database disagrees. PostgreSQL gives this same message for a wrong password, a user that does not exist, and a user with no SCRAM password where pg_hba.conf asks for scram-sha-256; only its server log says which.

kubectl logs <postgres-pod> | grep -A1 'password authentication failed'   # DETAIL: Password does not match, Role does not exist, or no valid SCRAM secret

Fix the database side the DETAIL line names: create the user, set its password, or match pg_hba.conf's method.

Hardening turned on readOnlyRootFilesystem for an image that writes caches, pid files or /tmp at start.

kubectl logs <pod> --previous   # the path it tried to write
kubectl get deploy <name> -o yaml   # readOnlyRootFilesystem: true, no volume at that path

Keep the root read-only, and mount an emptyDir at each path the app writes (for nginx: /var/cache/nginx, /var/run, /tmp).

A volume is mounted read-only where the app writes.

kubectl get deploy <name> -o jsonpath='{range .spec.template.spec.containers[*].volumeMounts[*]}{.mountPath} {.readOnly}{"\n"}{end}'   # the failing path under a mount marked true: this cause

Remove readOnly from that mount, or write somewhere else. ConfigMap and Secret volumes are always read-only.

The node's disk is nearly full, often with old images under /var/lib/containerd. The kubelet evicts Pods to free ephemeral storage: The node was low on resource: ephemeral-storage.

kubectl describe node <node>   # DiskPressure True, and the disk-pressure taint
ssh <node> sudo df -h   # / or /var near 100%

Free the disk: sudo crictl rmi --prune on the node removes unused images; clear large logs.

The node is low on memory. Pods using more than they request go first, BestEffort Pods first of all.

kubectl describe node <node>   # MemoryPressure True
kubectl get pod <pod> -o jsonpath='{.status.message}'   # low on resource: memory

Set memory requests close to real use, so critical Pods are not BestEffort and nodes are not overpacked.

One Pod used more ephemeral storage than its own limit. Only that Pod is evicted, even with a healthy node.

kubectl get pod <pod> -o jsonpath='{.status.message}'   # ephemeral local storage usage exceeds the total limit

Raise its ephemeral-storage limit, or write the data to a volume instead.

The Pod asks for more CPU or memory than any node has free. The scheduler counts requests, not real use, so a node can look idle and still be full: Insufficient cpu or Insufficient memory.

kubectl describe pod <pod>   # Insufficient memory in FailedScheduling: this cause
kubectl describe nodes | grep -A8 'Allocated resources'   # requests near 100% on every node

Lower the request to what the app really uses (kubectl set resources deployment/<name> --requests=memory=<size>), or add capacity.

A nodeSelector or node affinity names a label no node has. The scheduler then skips every node: didn't match Pod's node affinity/selector.

kubectl get pod <pod> -o jsonpath='{.spec.nodeSelector}'   # the labels it needs
kubectl get nodes --show-labels   # no node with that label: this cause

Correct the label in the Pod template, or label the node meant for it: kubectl label node <node> <key>=<value>.

A node has a taint the Pod does not tolerate. Taints keep Pods off a node unless they opt in: had untolerated taint {key: value}.

kubectl describe node <node> | grep Taints   # the taint named in the event

Add a matching toleration only if the Pod belongs on that node; otherwise remove the selector that points it there.

A PersistentVolumeClaim the Pod uses is not Bound. The scheduler waits for the volume before it places the Pod: unbound immediate PersistentVolumeClaims.

kubectl get pvc   # STATUS Pending: see PersistentVolumeClaim Pending

Get the claim bound first (a StorageClass that exists, a running provisioner); the Pod schedules once it is.

The node already runs as many Pods as its kubelet allows (maxPods, 110 by default). It is full by count, not by CPU: Too many pods.

kubectl get pods -A -o wide --field-selector spec.nodeName=<node> | wc -l   # close to the node's pod capacity
kubectl get node <node> -o jsonpath='{.status.allocatable.pods}'

Add nodes, or raise maxPods in the kubelet config where the node and its Pod network have room for more.

kube-scheduler is down: a broken static Pod manifest, a missing kubeconfig, a bad flag. Nobody picks a node, so nobody writes events.

kubectl -n kube-system get pods   # kube-scheduler-<node> CrashLoopBackOff or missing
kubectl -n kube-system logs kube-scheduler-<node>   # the startup error

Fix /etc/kubernetes/manifests/kube-scheduler.yaml; the kubelet restarts it.

spec.schedulerName names a scheduler that is not running. The default scheduler ignores Pods meant for another one.

kubectl get pod <pod> -o jsonpath='{.spec.schedulerName}'   # not default-scheduler

Run that scheduler, or remove the field.

spec.nodeName is set to a node that does not exist. The Pod skips the scheduler, and no kubelet claims it.

kubectl get pod <pod> -o jsonpath='{.spec.nodeName}'; kubectl get nodes   # a name not in the list

Remove nodeName from the template and use a nodeSelector instead.

More replicas than nodes, with requiredDuringScheduling anti-affinity on kubernetes.io/hostname. One per node is the most it can place.

kubectl get pods -l app=<app> -o wide   # one replica on every schedulable node already
kubectl get deploy <name> -o yaml | grep -A8 podAntiAffinity   # requiredDuringScheduling

Use preferredDuringSchedulingIgnoredDuringExecution so extra replicas may share a node, or topologySpreadConstraints with whenUnsatisfiable: ScheduleAnyway.

A node was drained or removed, so the rule cannot be met any more. Its replica has nowhere to go.

kubectl get nodes   # SchedulingDisabled nodes do not count

Uncordon the node or add one, or keep the hard rule and lower replicas to the number of nodes.

The topologyKey is a zone, with fewer zones than replicas.

kubectl get nodes -L topology.kubernetes.io/zone   # fewer zones than replicas

Spread by zone with topologySpreadConstraints and maxSkew instead of a hard anti-affinity rule.

Critical services and batch jobs both run at the default priority 0. Nothing is lower than the Pending Pod, so nothing may be evicted.

kubectl get priorityclass
kubectl get pods -A -o custom-columns=NAME:.metadata.name,PRIORITY:.spec.priority   # all 0: this cause

Create a PriorityClass for critical workloads (kubectl create priorityclass <name> --value=100000) and set it on the Deployment: kubectl patch deployment <name> -p '{"spec":{"template":{"spec":{"priorityClassName":"<name>"}}}}'.

A priorityClassName that does not exist. The Pod is then not created at all (FailedCreate), so the priority never applies.

kubectl describe rs <replicaset>   # no PriorityClass with name ... was found

Create the PriorityClass, or correct the name.

Requests fill the nodes completely, and everything running really matters. Preemption cannot help.

kubectl describe nodes | grep -A8 'Allocated resources'   # requests near 100%

Add capacity, or lower requests that are larger than real use.

Not this case: Preemption is not helpful for scheduling is printed for nodes that fail on something eviction cannot fix, such as an untolerated taint, a node selector or an unbound PersistentVolumeClaim.

kubectl describe pod <pod>   # Preemption is not helpful: see Pending

See Pending: fix the taint, selector or claim the event names.

The storageClassName does not exist, or the claim names no class and the cluster has no default. Nothing can create a volume: storageclass "fast-ssd" not found, or no storage class is set.

kubectl describe pvc <name>   # ProvisioningFailed or FailedBinding
kubectl get storageclass   # the class missing, or none marked (default)

Use a class that exists, or mark one default: kubectl patch storageclass <name> -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'.

The class uses WaitForFirstConsumer. It binds only once a Pod using the claim is scheduled, so the volume is made in the right zone. This is normal.

kubectl describe pvc <name>   # waiting for first consumer to be created before binding

Nothing to fix: create the Pod. If the Pod is Pending too, fix the Pod's scheduling first.

The class's CSI driver is not installed or not running. The claim keeps waiting for a volume to be created by the external provisioner.

kubectl describe pvc <name>   # Waiting for a volume to be created ... external provisioner
kubectl get csidriver; kubectl get pods -A | grep -i csi   # the class's PROVISIONER not running

Install or repair the CSI driver the StorageClass names, and read its controller Pod's logs.

The provisioner cannot satisfy the size or access mode. Most block storage is ReadWriteOnce only, so a ReadWriteMany claim fails.

kubectl describe pvc <name>   # ProvisioningFailed with the provisioner's reason
kubectl get pvc <name> -o jsonpath='{.spec.accessModes}'

Ask for an access mode and size the provisioner supports, or use storage that offers ReadWriteMany.

The PersistentVolumeClaim is too small for the data. The volume fills up and the next write fails.

kubectl exec <pod> -- df -h /data   # Use% 100%: this cause

Grow the claim if its StorageClass allows it (kubectl get storageclass, ALLOWVOLUMEEXPANSION): kubectl patch pvc <name> -p '{"spec":{"resources":{"requests":{"storage":"<bigger>"}}}}'. Then check kubectl describe pvc <name>: FileSystemResizePending means it grows when the Pod restarts.

Logs or temporary files are written to the volume and never cleaned.

kubectl exec <pod> -- du -sh /data/*   # a log or tmp directory taking most of it

Delete what does not belong there, and rotate or ship the logs.

Out of inodes, not bytes: many small files. df -h shows free space but no new file can be created.

kubectl exec <pod> -- df -i /data   # IUse% 100%: this cause

Remove the small files, or use a file system with more inodes.

The node's own disk is full, for a path that is not on a volume.

kubectl describe node <node>   # DiskPressure True

See Evicted: free the node's disk.

The node has a taint the DaemonSet does not tolerate. Untolerated NoSchedule taints exclude the node, so DESIRED does not count it.

kubectl get ds <name>   # DESIRED below the node count
kubectl describe node <node> | grep Taints   # a taint the DaemonSet does not tolerate

Add a matching toleration to the DaemonSet's Pod template (agents often tolerate every taint: operator: Exists).

A nodeSelector or affinity on the DaemonSet excludes the node. DESIRED does not count it either.

kubectl get ds <name> -o wide   # NODE SELECTOR
kubectl get node <node> --show-labels   # the label missing

Label the node, or widen the selector.

The node is out of resources for the agent. Then DESIRED counts the node, but its Pod stays Pending, so READY is below DESIRED.

kubectl get pods -l app=<name> -o wide   # one Pending on that node

Free room on the node, or give the agent a higher PriorityClass so it can preempt.

<name>-0 cannot start: a missing ConfigMap, an unbound claim, a crash. The rest wait for it.

kubectl describe pod <name>-0   # its own error: fix that first
kubectl logs <name>-0

Fix the first Pod; the others follow once it is Ready.

<name>-0 runs but its readiness probe never passes. Running is not enough for OrderedReady.

kubectl get pod <name>-0   # Running, READY 0/1

Fix the probe so it passes as soon as the Pod can serve.

Its PersistentVolumeClaim is Pending, or the headless Service is missing.

kubectl get pvc -l app=<name>   # Pending
kubectl get svc <serviceName>   # NotFound

Get the claim bound and create the headless Service the StatefulSet names. If replicas do not depend on each other, a new StatefulSet can use podManagementPolicy: Parallel.

The Service's selector and the Pods' labels differ, often by one letter. The selector matches nothing.

kubectl describe svc <svc>   # Selector
kubectl get pods -l <key>=<value>   # the Service's selector exactly: No resources found = this cause

Correct the selector, or the labels in the Deployment's Pod template (not on running Pods).

The Pods match but are not Ready. Only Ready Pods become endpoints.

kubectl get pods -l <key>=<value>   # Pods listed, READY 0/1: this cause

Fix their readiness probe; see Running but 0/1 Ready.

The Service is in a different namespace from the Pods. A selector only matches Pods in its own namespace.

kubectl get svc -A | grep <svc>; kubectl get pods -A -l <key>=<value>   # different namespaces

Create the Service in the Pods' namespace, and call it as <svc>.<namespace> from elsewhere.

The Service's targetPort is not the port the app listens on. kube-proxy forwards to a closed port, so the Pod refuses.

kubectl exec <pod> -- wget -qO- -T 2 http://<pod-ip>:<container-port>   # works by Pod IP but not by Service: this cause
kubectl get svc <svc> -o yaml | grep -A4 ports   # targetPort

Make targetPort the port the container listens on, or name the container port and use the name.

The Service has no endpoints. With nothing to forward to, the connection is refused at once.

kubectl get endpointslices -l kubernetes.io/service-name=<svc>   # no endpoints: this cause

See Service has no endpoints.

The app listens on 127.0.0.1, not 0.0.0.0. It is reachable only from inside its own Pod.

kubectl exec <pod> -- netstat -tln   # 127.0.0.1:<port>, not 0.0.0.0 or :::<port> (or ss -tln)

Make the app listen on 0.0.0.0.

A NetworkPolicy drops the traffic. Dropped packets get no answer, so the client times out instead of being refused.

kubectl get networkpolicy -n <target-namespace>   # a policy selecting the target Pods
kubectl exec <pod> -- wget -qO- -T 2 http://<pod-ip>:<container-port>   # times out by Pod IP too: this cause

Add an allow rule for the client's labels and the port to the policy that selects the target Pods.

Another Deployment (a canary, an old version) carries the same labels. The Service sends part of the traffic to it.

kubectl get pods -l <selector> --show-labels -o wide   # Pods from two Deployments: this cause

Make the selector specific: kubectl set selector service <svc> app=<app>,track=stable.

Some Pods listen on a different port from targetPort. Requests that reach them are refused.

kubectl get endpointslices -l kubernetes.io/service-name=<svc>   # an endpoint on an odd port, such as 8081

Make every Pod behind the Service serve the targetPort, or give the odd ones their own Service.

One node's Pods are unreachable: a NetworkPolicy, or a broken CNI on that node.

kubectl get pods -l <selector> -o wide   # failures line up with one node

Look at that node's CNI Pod and the NetworkPolicies that select those Pods.

The name is wrong, or the Service is in another namespace. A short name only works inside the same namespace, so DNS answers NXDOMAIN.

kubectl get svc -A | grep <svc>   # the namespace it really is in
kubectl exec <pod> -- nslookup <svc>.<namespace>.svc.cluster.local   # resolves: use the longer name

Use the right name; across namespaces it is <svc>.<namespace>.

An egress NetworkPolicy does not allow DNS. Queries to kube-dns on port 53 are dropped, so lookups time out.

kubectl get networkpolicy -n <ns>   # a policy with policyTypes Egress
kubectl exec <pod> -- nslookup kubernetes.default   # times out too: this cause

Allow egress to kube-dns in kube-system on port 53, UDP and TCP, in the default-deny policy.

CoreDNS is not running. Nothing answers the queries, so every lookup times out.

kubectl get pods -n kube-system -l k8s-app=kube-dns   # none Running, or 0 replicas

Bring it back: kubectl -n kube-system scale deployment coredns --replicas=2, and read its logs if it crashes.

404: no rule matches the request's host or path, or the Ingress has an ingressClassName this controller does not serve. The controller falls back to its default backend.

kubectl describe ingress <name>   # Rules: the host and paths
kubectl get ingressclass   # the class the Ingress names exists
curl -i -H 'Host: <host>' http://<ingress-address>/   # 404 with the right Host too: rule or class

Make host, path and ingressClassName match the request and the controller.

404 on sub-paths only: pathType is Exact where Prefix was meant. /menu matches, /menu/today does not.

kubectl get ingress <name> -o yaml   # pathType: Exact

Use pathType: Prefix for a path with pages under it.

Every page shows the home page, or the app's own 404: a rewrite-target drops or keeps the wrong part of the path. The app gets a different path from the one the user asked for.

kubectl get ingress <name> -o yaml | grep rewrite-target
kubectl logs <app-pod>   # the path the app received

Fix the rewrite-target so the app gets the path it expects.

503: the backend Service has no Ready endpoints, or the Ingress names a wrong Service or port. The controller has nowhere to send the request.

kubectl describe ingress <name>   # Backends: <error: services "x" not found> or no endpoints
kubectl get endpointslices -l kubernetes.io/service-name=<backend-svc>

Correct the Service name and port in the Ingress, or give the Service Ready Pods.

The Secret named in spec.tls does not exist, or is in another namespace. The controller can only read TLS Secrets in the Ingress's namespace.

kubectl describe ingress <name>   # TLS: <secret> terminates <host>
kubectl get secret <secret> -n <ingress-namespace>   # NotFound: this cause

Create it in the Ingress's namespace with the name spec.tls uses: kubectl create secret tls <name> --cert=<file.crt> --key=<file.key>.

The Secret is not of type kubernetes.io/tls, or the key does not match the certificate. The controller skips it.

kubectl get secret <secret> -o jsonpath='{.type}'   # not kubernetes.io/tls
kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | grep <secret>   # why it was skipped

Recreate it with kubectl create secret tls, which checks the pair.

The certificate does not name the host. The controller will not serve it for that host.

openssl x509 -in <file.crt> -noout -ext subjectAltName   # the host missing

Issue a certificate that names the host.

The host is not under spec.tls hosts, or the client sends no SNI (it connects by IP). The controller cannot tell which certificate to use, so it answers with its default.

openssl s_client -connect <ingress-address>:443 -servername <host> </dev/null 2>/dev/null | openssl x509 -noout -subject   # right certificate with the name, fake without: this cause

List the host under spec.tls hosts, and connect by name.

parentRefs names a Gateway that does not exist, or one in another namespace without saying so. No Gateway considers the route, so it often has no status at all.

kubectl describe httproute <name>   # no Parents in Status, or NoMatchingParent
kubectl get gateway -A   # the name and namespace it should name

Make parentRefs name the Gateway and its namespace exactly.

The Gateway's listener does not allow routes from this namespace. allowedRoutes defaults to the Gateway's own namespace: NotAllowedByListeners.

kubectl describe httproute <name>   # Reason NotAllowedByListeners
kubectl get gateway <name> -n <ns> -o yaml | grep -A4 allowedRoutes

Allow the route's namespace in the listener's allowedRoutes, or move the route.

The route's hostname matches no listener's hostname: NoMatchingListenerHostname.

kubectl describe httproute <name>   # NoMatchingListenerHostname

Use a hostname the listener accepts, or add a listener for it.

No controller runs for the GatewayClass. Nothing programs the Gateway or writes status.

kubectl get gatewayclass   # ACCEPTED not True
kubectl describe gateway <name>   # Programmed not True

Install the Gateway controller the GatewayClass names.

Accepted, but a backendRef points at a Service in another namespace without a ReferenceGrant. The backend is refused (ResolvedRefs False, RefNotPermitted) and requests get 500, not 404.

kubectl describe httproute <name>   # ResolvedRefs False, RefNotPermitted
kubectl get referencegrant -A

Add a ReferenceGrant in the Service's namespace that allows HTTPRoutes from the route's namespace.

The cluster has nothing that provides load balancers: kind, minikube or bare metal without MetalLB or cloud-provider-kind.

kubectl describe svc <name>   # no events from a service controller: this cause
curl http://<node-ip>:<nodePort>/   # works: only the load balancer is missing

Use the node port, or install a load balancer such as MetalLB or cloud-provider-kind; or change the type: kubectl patch svc <name> -p '{"spec":{"type":"NodePort"}}'.

On a cloud, the cloud controller manager is missing or lacks permissions. It tries and fails.

kubectl describe svc <name>   # SyncLoadBalancerFailed with a permission error

Give the cloud controller the permissions its provider documents for load balancers.

The cloud account hit its load-balancer or IP quota.

kubectl describe svc <name>   # SyncLoadBalancerFailed with a quota or limit error

Raise the quota, or put many Services behind one Ingress or Gateway.

Pod Security admission rejects the Pod. The namespace enforces a level (restricted, baseline) and the template lacks fields it requires: violates PodSecurity.

kubectl describe rs <new-replicaset>   # violates PodSecurity and the fields it wants
kubectl get ns <ns> --show-labels   # pod-security.kubernetes.io/enforce

Add the securityContext fields the message lists to the Pod template.

A ResourceQuota is full. The new Pod would take the namespace over its limit: exceeded quota, with requested, used and limited.

kubectl describe resourcequota   # Used at or near Hard

Lower the requests and limits, free room in the namespace, or raise the quota with its owner.

A quota on limits or requests exists, and the Pod sets none. The quota cannot count a Pod without them: failed quota: <name>: must specify limits.memory.

kubectl describe rs <new-replicaset>   # must specify

Set requests and limits on every container, or add a LimitRange with defaults.

An admission webhook or ValidatingAdmissionPolicy denies it. The message then names the webhook or policy.

kubectl describe rs <new-replicaset>   # admission webhook or ValidatingAdmissionPolicy in the message

See failed calling webhook or FailedCreate: denied by ValidatingAdmissionPolicy.

The ServiceAccount the template names does not exist. Pods cannot be created for it: error looking up service account.

kubectl get serviceaccount <name> -n <ns>   # NotFound: this cause

Create the ServiceAccount, or correct serviceAccountName.

The new image or configuration is broken. New Pods crash or stay unready, so the rollout cannot go on.

kubectl get rs   # the new ReplicaSet has Pods, none Ready
kubectl describe pod <new-pod>   # CrashLoopBackOff, ImagePullBackOff or Readiness probe failed

Undo while you fix it: kubectl rollout undo deployment/<name>, then fix the image or config and roll out again.

The new Pods cannot be created or scheduled: a quota, a taint, no room. They never exist, or stay Pending.

kubectl get pods   # new Pods Pending, or none
kubectl describe rs <new-replicaset>   # FailedCreate

Fix the scheduling or quota problem the events name; see Pending or Deployment 0/N with no Pods.

Someone ran kubectl rollout pause during an incident and never resumed.

kubectl describe deploy <name>   # Progressing Unknown, DeploymentPaused
kubectl get deploy <name> -o jsonpath='{.spec.paused}'   # true

Resume it: kubectl rollout resume deployment/<name>, then kubectl rollout status deployment/<name>.

A tool that batches changes paused it and failed before resuming.

kubectl get deploy <name> -o yaml --show-managed-fields | grep -B12 'f:paused'   # the manager that set it

Resume it as above, and fix the tool so it always resumes.

Deleting Pods does not help. They are recreated from the old ReplicaSet, with the old image.

kubectl get rs   # only the old ReplicaSet

Resume instead of deleting Pods.

Bad values: an image tag that does not exist, a misspelt key. The objects apply, but the new Pods never become Ready, so --wait times out.

helm history <release>   # the failed revision
helm get values <release> --revision <n>   # compare with the last good one
kubectl get pods   # new Pods failing

Roll back (helm rollback <release> <good-revision>), fix the values, then upgrade again with --wait.

An earlier upgrade was interrupted, so the release is stuck in pending-upgrade. Helm refuses a new operation: another operation (install/upgrade/rollback) is in progress.

helm history <release>   # last revision pending-upgrade, and no helm process running

Roll back to the last deployed revision to clear it, then upgrade again.

Someone changed the objects with kubectl. Helm's record and the cluster differ, so the upgrade conflicts.

helm get manifest <release> | kubectl diff -f -   # changes Helm did not make

Put the change in the chart's values, and change Helm-managed objects only through Helm.

The task itself fails. Each Pod exits non-zero, so each one counts as a failure.

kubectl get pods -l job-name=<name>   # several in Error
kubectl logs job/<name>   # the task's error (or kubectl logs <pod>)

Fix the cause, then delete the Job and create it again: a finished Job does not rerun.

Pods are killed from outside: OOMKilled, or evicted. Those count as failures too.

kubectl describe pod <failed-pod>   # Last State OOMKilled, or Reason Evicted

Give the Pod the memory it needs; use podFailurePolicy to ignore disruptions (DisruptionTarget) if evictions are expected.

backoffLimit is too low for a task that fails now and then. A few transient errors use it up.

kubectl get job <name> -o jsonpath='{.spec.backoffLimit}'

Raise backoffLimit, and make the task retry its own transient errors.

Not this case: a Job that runs past activeDeadlineSeconds fails with DeadlineExceeded (Job was active longer than specified deadline), not BackoffLimitExceeded.

kubectl get job <name> -o jsonpath='{.status.conditions[?(@.type=="Failed")].reason}'   # BackoffLimitExceeded or DeadlineExceeded

DeadlineExceeded: raise activeDeadlineSeconds if the task only needs longer.

The CronJob is suspended. The controller skips every run while suspend is true.

kubectl get cronjob <name>   # SUSPEND True

Resume it: kubectl patch cronjob <name> -p '{"spec":{"suspend":false}}'.

concurrencyPolicy is Forbid and a previous run hangs. Each new run is skipped while it is still active: JobAlreadyActive.

kubectl get cronjob <name>   # ACTIVE 1
kubectl get jobs   # one Running for days

Delete the stuck Job, and set activeDeadlineSeconds in the jobTemplate so a run cannot hang again.

The schedule is read in the controller's time zone. Without spec.timeZone it runs at a different hour from the one you meant.

kubectl get cronjob <name>   # TIMEZONE <none>

Set spec.timeZone, for example Europe/London.

Too many starts were missed (over 100), for example after the controller was down. The controller stops scheduling: too many missed start time.

kubectl describe cronjob <name>   # too many missed start

Set startingDeadlineSeconds, so only recent missed starts are counted.

A container in the target has no CPU (or memory) request. Utilisation is a percentage of the request, so without one there is nothing to divide by: missing request for cpu.

kubectl describe hpa <name>   # missing request for cpu in container <name>: this cause
kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[*].resources.requests}'   # one container without cpu

Set a CPU request on every container of the target, sidecars too: kubectl set resources deployment/<name> --requests=cpu=<m>.

metrics-server is not installed or not Ready. The metrics API does not exist, so the HPA has no numbers: the server could not find the requested resource (get pods.metrics.k8s.io).

kubectl get apiservice v1beta1.metrics.k8s.io   # missing, or AVAILABLE False: this cause
kubectl top pods   # error: Metrics API not available

Install metrics-server, and wait until the APIService is Available.

metrics-server runs but cannot scrape the kubelets. The API exists, but has no numbers for these Pods.

kubectl -n kube-system logs deploy/metrics-server   # x509 or timeout errors scraping nodes: this cause

Give metrics-server the kubelet CA, or turn on signed kubelet serving certificates; --kubelet-insecure-tls is for kind and labs only.

The Pods have just started or are not Ready. The HPA skips Pods without fresh metrics for a short while.

kubectl get hpa <name> -w   # a percentage appears within a minute or two: nothing to fix

Wait a minute after a rollout before reading TARGETS.

updateMode is Off (recommend only) or Initial (new Pods only). The VPA is doing exactly what it was told.

kubectl describe vpa <name>   # Update Mode: Off or Initial

Set updateMode: Recreate (or Auto) to let it apply the recommendation, or copy the numbers into the manifest yourself.

The workload has fewer replicas than the updater's minReplicas (2 by default). The updater will not evict the only replica.

kubectl get deploy <name>   # 1 replica

Run at least two replicas, or lower minReplicas in the VPA's updatePolicy if a short outage is acceptable.

The updater or admission controller is not installed or not running. Recommendations are written, but nothing applies them.

kubectl get pods -n kube-system | grep vpa   # recommender only, or one CrashLoopBackOff

Install or repair the missing component.

A mutable tag with imagePullPolicy IfNotPresent. Nodes keep the image they pulled before.

kubectl describe pod <pod>   # Image ID: the digest it runs
docker buildx imagetools inspect <image>   # Digest: what the tag points to now; different = this cause

Deploy an immutable tag or a digest: kubectl set image deployment/<name> <container>=<image>:<version>.

The Deployment did not change, so nothing rolled out. Pushing a tag does not restart anything.

kubectl rollout history deployment/<name>   # no new revision

Change the tag in the Deployment, so a rollout happens.

A registry mirror serves a stale copy of the tag.

docker buildx imagetools inspect <mirror>/<image>   # a different digest from the source registry

Refresh the mirror, or deploy by digest so a stale copy cannot match.

A finalizer is set and no controller removes it, often because its operator was uninstalled. The API server keeps the object until the list is empty.

kubectl get pod <pod> -o jsonpath='{.metadata.finalizers}'   # a finalizer listed: this cause

Bring back the controller that removes it. Only if it is gone for good: kubectl patch pod <pod> -p '{"metadata":{"finalizers":null}}'.

The Pod's node is unreachable. No kubelet can confirm the containers stopped.

kubectl get pod <pod> -o wide; kubectl get nodes   # its node NotReady

Delete the node object if it is gone, or as a last resort kubectl delete pod <pod> --grace-period=0 --force: only when the node is really down, since a StatefulSet could otherwise run two copies.

The app ignores SIGTERM and the grace period is long. The kubelet waits for terminationGracePeriodSeconds before it kills it.

kubectl get pod <pod> -o jsonpath='{.spec.terminationGracePeriodSeconds}'   # a large number

Handle SIGTERM in the app, and keep the grace period as short as a clean shutdown needs.

No kubeconfig. kubectl then tries localhost:8080; under sudo it reads root's kubeconfig, not yours.

echo $KUBECONFIG; ls -l ~/.kube/config   # neither set nor present: this cause
kubectl config current-context   # error: current-context is not set

Point kubectl at one (export KUBECONFIG=<file>, or copy it to ~/.kube/config), and run kubectl as your own user.

The current context points at a cluster that is gone or stopped, such as a kind cluster after Docker restarted.

kubectl config view --minify | grep server   # the address it uses
docker ps --filter name=control-plane   # kind: node container not running

Switch context (kubectl config use-context <name>), or start the kind node again (docker start <cluster>-control-plane), or recreate it with ./nimbus up.

The API server is down on the control-plane node: a broken static Pod manifest, or etcd not running.

sudo crictl ps -a | grep kube-apiserver   # Exited, on a kubeadm control-plane node

Read why: sudo crictl logs <container-id> and sudo journalctl -u kubelet, then fix the manifest or etcd.

A VPN, proxy or firewall between you and a remote cluster drops or refuses the connection.

nc -vz <host> 6443   # fails from your machine, works from inside the network

Connect the VPN, or add the API server to NO_PROXY.

The kubeconfig holds the CA of an earlier cluster at the same address, since recreated (kind delete and create, kubeadm reset and init). The new cluster has a new CA.

kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d | openssl x509 -noout -subject -enddate
openssl s_client -connect <host>:6443 </dev/null 2>/dev/null | openssl x509 -noout -issuer   # issuer differs from your CA's subject: this cause

Take a fresh kubeconfig: kind export kubeconfig --name <cluster>, or copy /etc/kubernetes/admin.conf on a kubeadm control plane.

A proxy or VPN that inspects TLS answers with its own certificate.

openssl s_client -connect <host>:6443 </dev/null 2>/dev/null | openssl x509 -noout -issuer   # your company's proxy as issuer

Bypass the proxy for the API server (add it to NO_PROXY) rather than trusting the proxy's CA.

certificate is valid for ..., not ...: you connect by an IP or name that is not in the API server certificate, such as a public IP or a load balancer added later.

sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -ext subjectAltName   # on a kubeadm control-plane node: the address missing

Connect by a listed name, or add it: list it under apiServer.certSANs, move the old apiserver.crt and apiserver.key aside, run sudo kubeadm init phase certs apiserver --config <file>, and restart the API server.

A merged kubeconfig pairs the context with the wrong cluster entry.

kubectl config view --minify   # the server and cluster entry the context uses

Point the context at the right cluster entry. Do not set insecure-skip-tls-verify: it accepts any server, including the wrong one.

Indentation that does not line up: a key one space off, or a list item not under its parent.

sed -n '15,19p' deploy.yaml   # the reported line and the few before it

Indent each level by the same two spaces. YAML X-ray on this site names each line's path, so a key at the wrong depth stands out.

A tab. YAML indents with spaces only: found character that cannot start any token.

grep -nP '\t' deploy.yaml   # any line: this cause

Replace tabs with spaces.

A missing space after a colon, or an unquoted value that contains : : mapping values are not allowed in this context.

sed -n '<line>p' deploy.yaml   # a second : in the value

Quote the value, such as "*" or "@daily".

A - out of place in a list, or keys of one item at different depths: did not find expected key.

kubectl apply --dry-run=client -f deploy.yaml   # still fails after each fix: look at the next reported line

Line up the keys of each list item under its first key.

A misspelt field or the wrong case. Fields are case sensitive: imagePullPolicy, not imagePullpolicy.

kubectl explain deployment.spec.template.spec.containers   # the fields that exist, with their types

Correct the field the message names (the path after unknown field).

A field at the wrong level, such as replicas under template. It exists, just not there.

kubectl explain <kind>.<path> --recursive | grep <field>

Move it to the level kubectl explain shows.

A value of the wrong type, such as replicas: "3" or a port in quotes: cannot unmarshal string.

kubectl explain deployment.spec.replicas   # FIELD: replicas <integer>

Give it the type shown, for example replicas: 3 without quotes.

A field from a newer Kubernetes version than the cluster runs.

kubectl version   # Server Version older than the docs you followed

Remove the field, or upgrade the cluster. Do not use --validate=false: the field is then dropped silently, not applied.

The apiVersion was removed: Ingress in extensions/v1beta1 (gone in 1.22), CronJob in batch/v1beta1 and PodDisruptionBudget in policy/v1beta1 (gone in 1.25), HorizontalPodAutoscaler in autoscaling/v2beta2 (gone in 1.26).

kubectl api-resources | grep -i <kind>   # the group and version this cluster serves
kubectl version   # Server Version

Use the apiVersion listed, for example networking.k8s.io/v1 for Ingress, and update the fields that changed with it (Ingress v1 needs pathType and service.name with service.port). The kubectl convert plugin rewrites many: kubectl convert -f <file> --output-version <group>/<version>.

A custom resource (ServiceMonitor, Certificate, HTTPRoute) is applied before the CRD that defines it.

kubectl get crd | grep <group>   # missing: this cause

Install its CRDs first, wait with kubectl wait --for condition=established crd/<name>, then apply.

A typo in kind or apiVersion. Kinds are case sensitive.

kubectl explain <kind> | head -5   # the exact kind and version

Correct the spelling.

spec.selector of a Deployment, ReplicaSet, StatefulSet or DaemonSet changed, often by renaming a label. The selector decides which Pods it owns, so it is fixed.

kubectl get deployment <name> -o jsonpath='{.spec.selector}'
kubectl diff -f deploy.yaml   # the selector in the diff: this cause

If not meant, put the old value back. If meant, delete and create it again (kubectl delete deployment <name>, then kubectl apply -f deploy.yaml, or kubectl replace --force -f deploy.yaml): the Pods are replaced, so do it when a short outage is acceptable.

A Job's Pod template changed, such as a new image under the same Job name. A Job runs once, so its template is fixed.

kubectl diff -f job.yaml   # spec.template differs

Delete the Job and create it again, or give the new one its own name.

A Service's clusterIP changed.

kubectl get svc <name> -o jsonpath='{.spec.clusterIP}'   # differs from the manifest

Remove clusterIP from the manifest, or recreate the Service.

For a StatefulSet, anything but its replicas, template, update strategy and a few others changed: updates to statefulset spec for fields other than ... are forbidden, often volumeClaimTemplates.

kubectl diff -f sts.yaml   # volumeClaimTemplates or serviceName in the diff

kubectl delete statefulset <name> --cascade=orphan keeps its Pods and PersistentVolumeClaims; then apply the new spec.

The client certificate in your kubeconfig has expired. kubeadm's admin.conf lasts a year.

kubectl config view --minify --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d | openssl x509 -noout -enddate   # a past date: this cause

sudo kubeadm certs renew all, then copy the new admin.conf to ~/.kube/config (see x509: certificate has expired).

A token expired or was revoked, such as a cloud login that an exec plugin refreshes.

kubectl config view --minify   # user: exec (aws, gke-gcloud-auth-plugin) or token

Log in again with your provider's CLI and fetch fresh credentials, for example aws eks update-kubeconfig --name <cluster> or gcloud container clusters get-credentials <cluster>.

The kubeconfig was written for an older cluster at the same address, since recreated with new credentials.

kubectl config view --minify   # the cluster and user entries

Write a fresh kubeconfig, for kind: kind export kubeconfig --name <cluster>.

The token belongs to a ServiceAccount that was deleted.

kubectl get serviceaccount <name> -n <ns>   # NotFound, from a working login

Recreate it and issue a new token: kubectl create token <name> -n <ns>.

No Role or ClusterRole bound to this identity grants that verb on that resource. RBAC denies anything not allowed.

kubectl auth can-i <verb> <resource> -n <ns> --as=system:serviceaccount:<ns>:<sa>   # no: this cause
kubectl auth can-i --list -n <ns> --as=system:serviceaccount:<ns>:<sa>

Grant only that verb on that resource in a Role, and bind it with a RoleBinding.

The request runs as another identity than the one you bound. The User in the message is the one the API server used, often the namespace's default ServiceAccount.

kubectl get pod <pod> -o jsonpath='{.spec.serviceAccountName}'   # differs from the one in your RoleBinding: this cause
kubectl auth whoami   # for a person: who the API server thinks you are

Set serviceAccountName on the Pod template to the ServiceAccount you bound, or bind the one it uses.

The RoleBinding names the wrong ServiceAccount or namespace. The grant exists, but for someone else.

kubectl describe rolebinding <name> -n <ns>   # Subjects: name and namespace must match the User in the message

Correct the subject's name and namespace in the RoleBinding.

A RoleBinding cannot grant cluster-scoped resources (nodes, namespaces, PersistentVolumes) or other namespaces. The message then has no namespace, or a different one.

kubectl api-resources --namespaced=false | grep <resource>   # listed: it needs a ClusterRoleBinding

Use a ClusterRole with a ClusterRoleBinding for cluster-scoped resources, scoped to the verbs needed.

A sub-resource is missing. Logs and exec are separate resources (pods/log, pods/exec), so get on pods is not enough.

kubectl auth can-i get pods/log --as=<user>   # no, while get pods says yes

Add the sub-resource to the Role: pods/log, pods/exec.

They are in another namespace. kubectl only looks in the context's namespace unless you pass -n or -A.

kubectl get pods -A | grep <name>   # found in another namespace: this cause

Pass -n, or set it once: kubectl config set-context --current --namespace=<ns>.

You are on the wrong cluster. The context points at another cluster that never had them.

kubectl config get-contexts   # the * is not the cluster you meant

Switch: kubectl config use-context <name>.

A NotFound for a named object: a typo in its name or kind.

kubectl get <kind> -A | grep -i <part-of-name>

Use the exact name, and the namespace it is in.

The image is not from an allowed registry. The policy checks each container's image.

kubectl get events --field-selector reason=FailedCreate   # denied request: images must come from ...

Use an image from the allowed registry in the Pod template; change the Pod, not the policy.

A required label, limit or securityContext field is missing. The policy's expression is false for this Pod.

kubectl get validatingadmissionpolicy <name> -o yaml   # the validations and their messages

Add the field the message names to the Pod template.

The policy's expression fails on a field your Pod does not set. A CEL error on a missing field counts as a denial unless failurePolicy is Ignore.

kubectl describe rs -l app=<app>   # expression error, no such key
kubectl get validatingadmissionpolicybinding   # validationActions

Fix the expression (has() before reading optional fields) and test it with validationActions: [Warn, Audit] first.

The webhook's own Pods are down, so its Service has no endpoints: no endpoints available for service. If the webhook also checks its own Pods, they cannot be recreated.

kubectl get validatingwebhookconfiguration <name> -o jsonpath='{.webhooks[*].clientConfig.service}'   # the Service, namespace and port it calls
kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<webhook-svc>   # no endpoints: this cause

Bring its Pods back. If the webhook blocks them, set failurePolicy to Ignore only until it is back: kubectl patch validatingwebhookconfiguration <name> -p '{"webhooks":[{"name":"<hook>","failurePolicy":"Ignore"}]}', then exclude its own namespace with a namespaceSelector.

The webhook's certificate: expired, not signed by the caBundle in the webhook configuration, or not naming <svc>.<ns>.svc. The API server will not trust it: x509 in the message.

kubectl get events -A | grep 'failed calling webhook'   # x509: certificate has expired / signed by unknown authority

Renew the certificate (or let cert-manager do it) and make the caBundle match its CA.

The API server cannot reach the webhook Pods: a NetworkPolicy, or on a managed cluster a firewall from the control plane to the Pod's port. The call times out: context deadline exceeded.

kubectl get events -A | grep 'failed calling webhook'   # context deadline exceeded: this cause
kubectl get networkpolicy -n <ns>

Allow traffic from the control plane to the webhook's Pod port.

The image is built on an old base image. Its OS packages have had fixes since, but the image never got them.

kubectl get deploy -o wide   # IMAGES: the tag running
trivy image <image> --severity HIGH,CRITICAL   # findings in OS packages with a Fixed Version

Rebuild on a current base image, scan it, then roll it out: kubectl set image deployment/<name> <container>=<image>:<tag>.

An application dependency is vulnerable. A newer tag of your own image can still ship the same library version.

trivy image <image> --severity HIGH,CRITICAL   # findings under the language's packages (jar, npm, pip)

Move the dependency to the Fixed Version the scanner lists, rebuild, and scan again.

No scan runs in CI. Nothing stops a vulnerable image before deploy, so it is only found later.

trivy image --exit-code 1 --severity CRITICAL <image>; echo $?   # 1: this would have failed the build

Add that command to CI so critical findings fail the pipeline.

The API server has no --encryption-provider-config. It writes Secrets as they are.

grep encryption-provider-config /etc/kubernetes/manifests/kube-apiserver.yaml   # nothing: this cause

Write an EncryptionConfiguration with aescbc (or a KMS provider) first and identity last, and pass it with --encryption-provider-config.

Encryption is on, but existing Secrets were never rewritten. Only writes after the change are encrypted.

etcdctl get /registry/secrets/<ns>/<name> --print-value-only <certificate flags> | head -c 40 | hexdump -C   # first bytes only: k8s then v1 Secret = still plain text

Rewrite every Secret so it is stored encrypted: kubectl get secrets -A -o json | kubectl replace -f -.

identity is the first provider in the config. The first provider is the one used to write, and identity means no encryption.

sudo grep -A6 'providers:' <encryption-config-file>   # identity: {} listed first

Put aescbc (or KMS) first and identity last, restart the API server, and rewrite the Secrets. Check: the value now starts with k8s:enc:aescbc:v1:.

A person or automation with broad rights deleted it, for example through a shared edit binding or cluster-admin. The audit entry names that identity.

grep '"verb":"delete"' /var/log/kubernetes/audit/audit.log | grep <name>   # the file --audit-log-path names; read user.username and the time
kubectl auth can-i --list --as=<user>   # what that identity may do

Restore the object from your manifests or backup, and replace the broad binding with a narrow Role.

A controller or the garbage collector deleted it, because its owner was deleted. The audit user is then a system account.

grep <name> /var/log/kubernetes/audit/audit.log | grep delete   # user system:serviceaccount:kube-system:generic-garbage-collector

Find why the owner went; restore the owner and its children together.

Audit logging is off, or the policy level is None for that resource. Nothing was recorded.

grep audit /etc/kubernetes/manifests/kube-apiserver.yaml   # no --audit-policy-file: this cause

Turn on audit logging now, with at least Metadata for Secrets and RBAC objects, so the next one is recorded.

The kubelet or the container runtime stopped. Nothing reports the node's status, so it goes to Unknown.

kubectl get node <node> -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}'   # Kubelet stopped posting node status: this cause
ssh <node> 'sudo systemctl status kubelet containerd; sudo journalctl -u kubelet -n 50'

Start what stopped (sudo systemctl restart containerd kubelet) after reading why it stopped.

The kubelet runs and reports itself not ready, for example container runtime network not ready (no CNI) or PLEG is not healthy (the runtime is stuck). The condition is False with that reason.

kubectl describe node <node>   # Ready False, and the message names the reason

Fix what the message names: the CNI Pod on that node, or restart the runtime.

Disk, memory or PID pressure. The kubelet protects the node, and may stop working well if it cannot write.

kubectl describe node <node>   # MemoryPressure, DiskPressure or PIDPressure True

Free the disk or memory the condition names; see Evicted.

The network between the node and the API server is down. The kubelet runs but its reports do not arrive.

ssh <node> 'curl -sk https://<api-server>:6443/healthz'   # times out: this cause

Restore the route or firewall between the node and the control plane.

The kubelet's client certificate expired. The API server rejects its reports.

ssh <node> 'sudo journalctl -u kubelet -n 50 | grep -iE "x509|Unauthorized"'   # certificate errors: this cause

Renew it (kubeadm certs on the control plane, or rejoin the node), and keep rotateCertificates on.

minAvailable equals the replica count, or is 1 on a one-replica app. No Pod may ever go.

kubectl get pdb -A   # ALLOWED DISRUPTIONS 0
kubectl describe pdb <name>   # Min available equal to the replicas

Scale up first (kubectl scale deployment/<name> --replicas=<more>) and drain again, or set minAvailable below the replica count, or maxUnavailable: 1.

Some replicas are not Ready. They already use up the budget, so none of the healthy ones may go.

kubectl get pods -l <selector>   # some not Ready
kubectl describe pdb <name>   # Current below Desired

Fix the unready Pods first; the budget frees up when they are Ready.

kubeadm's certificates expired: the cluster has not been upgraded in a year, and an upgrade is what renews them.

sudo kubeadm certs check-expiration   # EXPIRES in the past, RESIDUAL TIME invalid
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate

On a control-plane node: sudo kubeadm certs renew all, then restart the control-plane static Pods (move their manifests out of /etc/kubernetes/manifests and back).

The certificates were renewed, but the static Pods still run with the old ones loaded.

sudo kubeadm certs check-expiration   # renewed dates
echo | openssl s_client -connect <host>:6443 2>/dev/null | openssl x509 -noout -enddate   # still the old date: this cause

Restart the control-plane static Pods so they load the new files.

Your kubeconfig embeds an expired client certificate. The API server rejects it: You must be logged in to the server.

kubectl config view --minify --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d | openssl x509 -noout -enddate   # a past date

Copy the renewed admin.conf: sudo cp /etc/kubernetes/admin.conf ~/.kube/config && sudo chown $(id -u):$(id -g) ~/.kube/config.

A clock is wrong on the client or a node. A valid certificate looks expired, or not yet valid (is before).

date -u; timedatectl   # far from the real time, or NTP not synchronized

Fix time sync (NTP) rather than renewing certificates that are fine.

The kubeadm package is older than the target. kubeadm cannot install a version newer than itself: is higher than the kubeadm version.

kubeadm version -o short   # older than the target

Upgrade kubeadm first: sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=<1.x.y>-* && sudo apt-mark hold kubeadm.

The apt source on pkgs.k8s.io is per minor version and still points at the old one. apt cannot see the new packages.

cat /etc/apt/sources.list.d/kubernetes.list   # /v1.33/ while upgrading to 1.34
apt-cache madison kubeadm   # no 1.34 versions

Point it at the new minor version (pkgs.k8s.io/core:/stable:/v1.34/deb/), then sudo apt-get update.

The packages are held. apt refuses to change them: Held packages were changed.

apt-mark showhold   # kubeadm, kubelet, kubectl

Unhold, install the exact version, hold again (as above), or pass --allow-change-held-packages for that one install.

The target skips a minor version. kubeadm upgrades one minor at a time.

sudo kubeadm upgrade plan   # the versions it will go to

Upgrade through each minor version in turn: sudo kubeadm upgrade apply v<1.x.y>.

systemctl daemon-reload was skipped after the package changed the unit. systemd still runs the old unit definition.

sudo systemctl status kubelet   # The unit file ... changed on disk

sudo systemctl daemon-reload && sudo systemctl restart kubelet.

The kubelet is newer than the API server. A kubelet may not be newer than the API server, so it does not register.

kubectl version   # Server Version older than the kubelet's
kubectl get nodes   # VERSION column

Upgrade the control plane first, then the kubelets.

A flag the new version removed, such as --container-runtime (gone in 1.27). The kubelet exits at start: unknown flag.

sudo journalctl -u kubelet -n 50 | grep -i 'unknown flag'   # present: this cause
cat /var/lib/kubelet/kubeadm-flags.env   # flags kubeadm passes to the kubelet

Remove the flag from /var/lib/kubelet/kubeadm-flags.env (or /etc/default/kubelet), moving the setting into the kubelet config file if it has one, then restart the kubelet.

The wrong endpoint, or no client certificate. etcd ignores or drops the connection, and etcdctl gives up: context deadline exceeded.

kubectl -n kube-system describe pod etcd-<node>   # --listen-client-urls and the cert paths
ls /etc/kubernetes/pki/etcd   # ca.crt, server.crt, server.key

Pass what etcd expects: --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key.

--cacert is not the etcd CA (often the cluster CA, /etc/kubernetes/pki/ca.crt). etcd has its own CA: x509 unknown authority.

grep trusted-ca-file /etc/kubernetes/manifests/etcd.yaml   # the CA etcd uses

Use /etc/kubernetes/pki/etcd/ca.crt on kubeadm clusters.

The output directory does not exist. etcdctl cannot create the file.

ls -d <dir>   # No such file or directory

Create the directory first, then check the result: etcdutl snapshot status <file> --write-out=table.

Restoring over /var/lib/etcd. etcdutl refuses a directory that exists, and writing over a running etcd would corrupt it.

ls /var/lib/etcd   # not empty: restore elsewhere

Restore into a new, empty directory: etcdutl snapshot restore <file> --data-dir /var/lib/etcd-restore.

Restored into a new directory, but the etcd manifest still points at the old one. etcd starts on the old data, so nothing changes.

grep -A2 hostPath /etc/kubernetes/manifests/etcd.yaml   # path still /var/lib/etcd

Point the etcd-data hostPath at it: sudo sed -i 's#path: /var/lib/etcd#path: /var/lib/etcd-restore#' /etc/kubernetes/manifests/etcd.yaml; the kubelet restarts etcd.

--data-dir changed but not the hostPath volume, or the reverse. etcd looks inside the container for a path that is not mounted, and does not start.

sudo crictl ps -a | grep etcd   # Exited
sudo crictl logs <etcd-container>   # the data dir error

Change only the hostPath, and leave --data-dir and the mountPath as they were.

Custom resources keep finalizers that no controller removes, because their operator was uninstalled first: Some content in the namespace has finalizers remaining.

kubectl describe ns <ns>   # NamespaceFinalizersRemaining and the finalizer name
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n <ns>   # what is left

Bring back the operator if you can. Only if it is gone for good: kubectl patch <kind> <name> -n <ns> --type=merge -p '{"metadata":{"finalizers":null}}'.

An APIService (such as metrics) is unavailable, so the namespace controller cannot list everything to delete: NamespaceDeletionDiscoveryFailure.

kubectl get apiservice | grep False   # a broken APIService

Repair or delete that APIService; the namespace then finishes.

A PersistentVolumeClaim or Pod with its own finalizer waits for something else, such as a Pod still using the claim.

kubectl get pvc,pods -n <ns>   # Terminating ones

Fix what each one waits for. Do not clear the namespace's own finalizer through its finalize API: objects left in it stay in etcd and come back if it is created again.