Sep 04, 2026 Kubernetes

Debugging DNS Resolution Failures in Kubernetes

A service that resolves fine from your laptop times out from inside the cluster. Or it works most of the time, then randomly adds five seconds to every outbound call. Or every pod on one node can't resolve anything while the rest of the cluster is fine. These are all "DNS is broken," and they are three different problems with three different fixes.

DNS failures are disproportionately painful in Kubernetes because almost everything depends on service discovery — an app can't reach its database, an ingress controller can't resolve an upstream, a sidecar can't call the control plane, and none of it looks like a DNS problem at first. It looks like "the service is down." Engineers spend an hour checking the wrong service before anyone runs nslookup.

This is the diagnostic process I actually use, broken down by failure mode, because "restart CoreDNS and hope" is not a strategy.

Short answer

Start here, in order:

  1. Are the CoreDNS pods healthy? kubectl -n kube-system get pods -l k8s-app=kube-dns — if they're crash-looping or not Running, that's your answer.
  2. Can a test pod resolve anything at all? Run a debug pod and try kubernetes.default.svc.cluster.local. If that fails but the CoreDNS pods look healthy, it's networking (NetworkPolicy, kube-proxy, or the CoreDNS Service), not CoreDNS itself.
  3. Is it slow, not failing? That's almost always ndots:5 amplifying lookups for external domains, not a broken resolver.
  4. Is it intermittent, roughly one request in three? That's the UDP conntrack race, not application flakiness.

Each of those has a distinct fix below. Guessing which one you have and applying a random fix (bumping CoreDNS replicas, restarting kube-proxy, adding retries in application code) wastes time and often just masks the symptom.

How Kubernetes DNS actually works

Two pieces matter for debugging, both documented in the Kubernetes DNS reference:

  • CoreDNS runs as a Deployment in kube-system, fronted by a Service (usually named kube-dns for historical reasons — CoreDNS replaced kube-dns as the default back in 1.13, and nobody renamed the Service). Every pod's /etc/resolv.conf points at that Service's ClusterIP.
  • Pod DNS config: unless overridden, pods get dnsPolicy: ClusterFirst, which means CoreDNS handles cluster-internal names and forwards everything else upstream. The pod's search domains (<namespace>.svc.cluster.local, svc.cluster.local, cluster.local, plus whatever the node adds) and ndots:5 come from the same config and are the source of most of the "DNS is slow" reports, covered below.

If either of those two things is misconfigured or unhealthy, resolution breaks in a specific, recognizable way — which is the point of going through each failure mode deliberately instead of poking at it.

Failure mode 1: CoreDNS itself is unhealthy

Symptom: total resolution failure, cluster-wide, for both internal and external names.

kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system get deploy coredns

If pods are CrashLoopBackOff, check why:

kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100
kubectl -n kube-system describe pod -l k8s-app=kube-dns

Three things show up here repeatedly:

  • OOMKilled. CoreDNS's default resource limits (often copied from a Helm chart's defaults years ago) don't scale with cluster size or query volume. describe pod will show Last State: Terminated, Reason: OOMKilled. Fix: raise memory limits and check actual usage with kubectl top pod -n kube-system -l k8s-app=kube-dns before guessing a number.
  • plugin/loop: Loop detected in the logs. CoreDNS refuses to run when it detects it would forward a query back to itself — usually because the node's own /etc/resolv.conf points at a local resolver (like systemd-resolved) that in turn forwards to CoreDNS, or because of a misconfigured forward directive in the Corefile. This is CoreDNS protecting you from an infinite loop, not a bug to work around by removing the loop plugin. Fix the upstream resolver configuration on the node, or the forward . target in the Corefile — check with kubectl -n kube-system get configmap coredns -o yaml.
  • Too few replicas for the node count. Two CoreDNS replicas is the default in most installers and is fine for a small cluster. It is not fine for 200 nodes doing heavy service-to-service traffic. There's no universal number — check kubectl top pod for CPU saturation on the existing replicas and scale from there, and consider the cluster-proportional-autoscaler so it scales with node count automatically instead of being hand-tuned once and forgotten.

Failure mode 2: CoreDNS is healthy, but nothing can reach it

Symptom: total resolution failure again, but CoreDNS pods are Running and logs are clean.

This is a networking problem between the client pod and the CoreDNS Service, and it has become far more common since default-deny NetworkPolicies became a standard hardening step (it's item #2 on the cluster audit checklist for good reason). Enabling default-deny egress without an explicit allow rule for port 53 is the single most common self-inflicted DNS outage I see.

Isolate it with a throwaway debug pod — this is the pod the Kubernetes docs themselves use for DNS debugging, and it's disposable (--rm), not something you need to clean up after:

kubectl run dns-debug --image=busybox:1.28 --restart=Never --rm -it -- \
  nslookup kubernetes.default.svc.cluster.local

Illustrative output when it's working:

Server:    10.96.0.10
Address:   10.96.0.10:53

Name:      kubernetes.default.svc.cluster.local
Address:   10.96.0.1

If this times out entirely (no Server: line, just a timeout), work through these in order:

  1. Check the CoreDNS Service has endpoints. kubectl -n kube-system get endpoints kube-dns — if ENDPOINTS is empty, the Service's selector doesn't match any Ready pod, which happens after a label change on the CoreDNS Deployment or a readiness probe failure.
  2. Check NetworkPolicies in the querying pod's namespace. kubectl get networkpolicy -n <namespace> -o yaml. A default-deny egress policy needs an explicit rule allowing UDP/TCP 53 to the kube-system namespace (or to the CoreDNS pod selector directly, which is more precise):
# Illustrative — allow DNS egress alongside your existing default-deny policy.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: checkout
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
  1. Check kube-proxy is actually programming the Service. On the node running the querying pod: iptables-save | grep <coredns-clusterip> (iptables mode) or the equivalent IPVS check. A missing rule points at a kube-proxy problem, not a DNS one — a different debugging path entirely.

Failure mode 3: Resolution works, but external lookups are slow

Symptom: internal service names resolve fast; external domains (a third-party API, a managed database endpoint) take 3-5+ seconds intermittently.

This is ndots:5, and it's not a bug — it's the default pod DNS config working as documented, just interacting badly with how most applications resolve names. With ndots:5, any name with fewer than 5 dots gets the search domains appended and tried first. Looking up api.stripe.com (2 dots) inside a pod actually attempts, in order:

api.stripe.com.checkout.svc.cluster.local
api.stripe.com.svc.cluster.local
api.stripe.com.cluster.local
api.stripe.com.<node-search-domain>   (if present)
api.stripe.com                         (finally, the one that works)

Every one of those NXDOMAIN responses costs a round trip. For a busy service making a lot of external calls, that's a meaningful and confusing tail-latency source that won't show up as an error — just as slowness that's hard to pin on DNS unless you're specifically looking for it.

Confirm it by capturing CoreDNS query logs for a few seconds (enable the log plugin temporarily, or check if it's already on) and looking for a burst of NXDOMAIN responses for the same external domain right before the real answer. Fix it one of two ways:

  • Set ndots lower for pods that make heavy external calls, via dnsConfig:
# Illustrative — reduces search-domain amplification for external lookups.
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
  • Or fully-qualify the external domain in application config with a trailing dot (api.stripe.com.), which skips the search list entirely. This is more surgical but requires touching application configuration rather than the pod spec.

Don't set ndots: 1 cluster-wide as a blanket fix — it also breaks short-form internal lookups like redis instead of redis.checkout.svc.cluster.local, which most in-cluster tooling and Helm charts assume works.

Failure mode 4: Intermittent failures, roughly one in three

Symptom: resolution fails randomly, often reported as "flaky," with failures clustering suspiciously close to ~33% of requests. Retrying immediately usually succeeds.

This is the well-documented UDP conntrack race: when two DNS queries from the same pod hit the conntrack table at nearly the same time (which happens constantly, because glibc and musl both issue A and AAAA lookups in parallel by default), Linux can insert two conntrack entries for what should be one connection, and one of the two response packets gets dropped in the kernel before your application ever sees it. It's a kernel-level race condition, not an application bug or a CoreDNS bug, and it's exactly why NodeLocal DNSCache exists — it runs a caching DNS agent on each node so most queries never cross the network (and hit conntrack) at all.

You can't fix the kernel race per-application. What actually works in production:

  • Deploy NodeLocal DNSCache. It's the standard mitigation, and most managed Kubernetes offerings (EKS, GKE) support it as an add-on rather than something you build from scratch.
  • Disable AAAA (IPv6) lookups on IPv4-only clusters if you don't need them — this halves the parallel-query volume that triggers the race in the first place. This is a per-application/runtime setting (e.g., disabling IPv6 resolution in the app's HTTP client), not a cluster-wide DNS setting.

Common mistakes

  • Restarting CoreDNS as a first move. It resets the symptom for a minute (fresh pods, empty conntrack table) and destroys the evidence you needed to diagnose the actual cause. Capture logs and describe output first.
  • Blaming the application for "flaky networking" when the failure rate and pattern (intermittent, fails fast, immediate retry succeeds) are the conntrack race's signature, not application flakiness.
  • Adding NetworkPolicy default-deny and not testing DNS afterward. This is the most common cause of "the cluster went down right after the security audit," and it's entirely avoidable with the explicit port-53 allow rule above.
  • Tuning ndots cluster-wide instead of scoping it to the workloads that actually make heavy external calls, then being surprised when internal short-name lookups break somewhere else.
  • Confusing a Service-level problem with a DNS problem. If nslookup resolves the name to the correct ClusterIP but the connection still fails, that's kube-proxy, a Service selector, or an endpoint readiness issue — not DNS. Resolution succeeding is proof DNS isn't the problem; don't keep debugging DNS past that point.

Production considerations

  • Monitor CoreDNS itself, not just application-level symptoms. The CoreDNS /metrics endpoint exposes coredns_dns_responses_total (watch for a rising SERVFAIL/REFUSED rate) and coredns_dns_request_duration_seconds (watch p99 latency, not just averages — the ndots amplification and conntrack race both show up as tail latency long before averages move). This is exactly the kind of operational signal worth wiring into Kubernetes observability rather than discovering it during an incident.
  • Set a PodDisruptionBudget for CoreDNS. Two replicas with no PDB means a node drain during a cluster upgrade can take both down simultaneously.
  • Test DNS after every NetworkPolicy change, not just after the initial rollout — a policy that's correct today can silently break DNS after a namespace label change or a Kubernetes upgrade that adjusts default labels.
  • Version-specific behavior: dnsPolicy and dnsConfig behavior is stable across supported Kubernetes versions, but CoreDNS's bundled plugin defaults and NodeLocal DNSCache's exact deployment manifest vary by cluster distribution (EKS, GKE, AKS, kubeadm) — check your provider's current documentation before copying a manifest from a blog post, this one included.

Checklist

  • [ ] CoreDNS pods are Running, not crash-looping, with clean logs (plugin/loop errors resolved)
  • [ ] CoreDNS resource limits are based on actual kubectl top usage, not defaults from years ago
  • [ ] kube-dns Service has non-empty Endpoints
  • [ ] Default-deny NetworkPolicies include an explicit egress rule for port 53 to kube-system
  • [ ] ndots is scoped per-workload for services with heavy external API traffic, not blanket-set
  • [ ] NodeLocal DNSCache is deployed (or a documented reason it isn't)
  • [ ] A PodDisruptionBudget exists for the CoreDNS Deployment
  • [ ] CoreDNS response codes and p99 latency are in your monitoring, with alerting on the trend — not discovered during an incident

Conclusion

Most "Kubernetes DNS is broken" reports collapse into one of the four failure modes above, and each one has a specific signature that tells you which it is before you touch anything: total failure with unhealthy CoreDNS pods, total failure with healthy pods (networking), consistent slowness on external names only (ndots), or intermittent failures at a suspicious ~33% rate (conntrack). Diagnose which one you actually have before changing anything — it's faster than guessing, and it's the difference between fixing the problem and just resetting the clock on it.

If DNS reliability is one symptom of a broader pattern — NetworkPolicies that don't quite work as intended, resource limits nobody's revisited, monitoring that doesn't cover the signals that actually predict an incident — that's usually worth a wider look rather than a one-off fix. That's exactly the kind of networking and reliability work covered under Kubernetes Consulting, or if you want a prioritized, written assessment of where a specific cluster actually stands first, the Kubernetes Cluster Audit covers DNS and networking as part of a broader production-readiness review — see the sample audit report for what that looks like.

#kubernetes #networking #dns #coredns #troubleshooting
← Back to Blog