Production Kubernetes Debugging Handbook: Pods, DNS

Back to Kubernetes Guide
Kubernetes is powerful. It is also one of the most complex systems a DevOps engineer will operate in production Kubernetes clusters.
When something breaks — and it will break — the failure is rarely obvious. A pod disappears. A service stops responding. A node silently goes NotReady at 2 AM. The symptom you see on the surface is almost never where the actual problem lives.
Unlike debugging a single server, Kubernetes failures span multiple layers. A single application outage can involve container runtime issues, scheduler decisions, network policy conflicts, resource exhaustion, and control plane delays. Each layer has its own signals, tools, and failure modes.
This is what makes Kubernetes debugging hard. Not the complexity of any single component, but the sheer number of components that interact — and the fact that in production, you are debugging under pressure, with real users affected and stakeholders watching.
This handbook exists to give you a systematic approach, linking concepts across pod, node, networking, storage, and control plane troubleshooting.
After managing AKS clusters running 500+ cores for Fortune 500 clients, the most important lesson is this: a solid production Kubernetes debugging workflow is not about memorizing error messages. It is about knowing where to look, in what order, and what each signal means. That requires Kubernetes cluster observability — logs, events, metrics, and state — working together.
This handbook covers every major Kubernetes pod failure troubleshooting scenario and cluster-level issue you will encounter in real production environments:
DevOps engineers, SREs, and platform engineers operating Kubernetes in production. It assumes you know what a pod is — not that you have seen every failure mode.
Tip: If you do not have stern and k9s installed, start there. They cut debugging time significantly compared to running repeated kubectl logs commands manually.
Each section follows the same pattern: what the failure means, how to diagnose it step by step, and a concrete fix checklist. Use the links below to jump directly to the section you need:
This handbook is a living reference. Use it alongside the Kubernetes Guide for full topic coverage.
Pod failures are the most common class of Kubernetes issues you will encounter in production. The challenge is that the same symptom — a pod not running — can have a dozen different root causes. This section gives you a systematic approach to diagnose and resolve each one.
Before diving into specific failure types, always start with these four commands. They give you 80% of the information you need in under five minutes.
Run these in order, every time. Do not skip steps. Events in describe often reveal the root cause before you even check logs.
CrashLoopBackOff is not an error itself — it is Kubernetes telling you that your container keeps starting and immediately crashing. Kubernetes applies an exponential backoff between each restart attempt (10s, 20s, 40s, up to 5 minutes), which is where the name comes from.
What to look for in describe output
A container that exits within 2 seconds almost always means the application failed to start — check your startup logs and environment variables first.
Kubernetes cannot pull the container image from the registry. ErrImagePull is the first attempt failure. ImagePullBackOff is Kubernetes backing off after repeated failures.
The Linux kernel killed your container because it exceeded its memory limit. Exit code will be 137.
Important: always set requests and limits together. A container without a memory limit can consume all node memory and cause node-level failures affecting other workloads.
The pod has been accepted by the API server but the scheduler cannot place it on any node.
⬆ Back to Table of Contents
Node failures are more severe than pod failures. When a node goes NotReady, every workload running on it is at risk. In a production cluster, a single NotReady node can trigger a cascade — pods get evicted, rescheduled onto already-strained nodes, and suddenly you have a cluster-wide resource pressure event instead of a single node issue.
The key is catching it early and diagnosing the root cause before it spreads.
Run these in order before doing anything else. The describe node output will almost always tell you which condition is failing.
When you run kubectl describe node, look for the Conditions section. Each condition tells you a specific story:
Warning: A Ready: Unknown status means the node controller has not received a heartbeat from the kubelet in the last 40 seconds (default node-monitor-grace-period). This usually means the node is completely unreachable — network failure, VM crash, or kubelet process died.
The kubelet process on the node has stopped reporting to the control plane. This is the most common cause of NotReady.
PLEG (Pod Lifecycle Event Generator) errors are particularly important. If you see PLEG is not healthy, it means the kubelet cannot communicate with the container runtime (containerd or Docker). This often points to a container runtime crash.
The node is running out of available memory. Kubernetes will stop scheduling new pods onto this node and may begin evicting existing pods.
Common causes in production
Tip: Always set memory requests and limits on every container in production. A single pod without limits can bring down an entire node.
The node is running low on disk space. Kubernetes will evict pods and stop scheduling new workloads onto the node.
Most common culprits
The CNI (Container Network Interface) plugin has not configured networking correctly on the node. New pods cannot get IP addresses.
Warning: kubectl drain will evict all pods from the node. Make sure your workloads have enough replicas running on other nodes before draining, or you will cause downtime.
⬆ Back to Table of Contents
Networking failures in Kubernetes are among the hardest to debug. The system has multiple layers — pod networking,