Kubernetes High Availability: What Actually Keeps Workloads Running

replicas: 3 in a Deployment spec is not a high-availability guarantee. It's a starting point that becomes one only if the three replicas actually land on different failure domains, a PodDisruptionBudget doesn't accidentally block the maintenance that's supposed to be safe, and the probes deciding when a pod is "ready" or "should be restarted" are configured for what they actually measure. Most Kubernetes outages I see in this category aren't caused by missing HA features — they're caused by HA features present, but configured in a way that looks correct and isn't.

Short answer

Three replicas on the same node is not HA — it's three copies of a single point of failure. The mechanisms that actually matter, roughly in order of how often they're the reason an "HA" setup wasn't:

1. Topology spread constraints or pod anti-affinity to force replicas onto different nodes/zones — without this, the scheduler is free to pack them together.
2. A PodDisruptionBudget that permits at least one voluntary disruption — minAvailable set equal to replica count blocks node drains and cluster upgrades entirely, which is the opposite of the intended effect.
3. Readiness probes that reflect real dependency health, so a rolling deployment doesn't send traffic to a pod that isn't actually ready to serve it.
4. Liveness probes that are conservative, because an aggressive one restarts healthy-but-slow pods, which is self-inflicted unavailability.

None of this replaces a deliberate decision about how much availability you're paying for — HA is a cost and complexity trade-off, not a checkbox to tick.

Replica count and failure domains

A Deployment with replicas: 3 tells the scheduler to run three copies — it says nothing about where. Left unconstrained, the scheduler can (and under low cluster utilization, often will) place all three on the same node, because bin-packing efficiency is its default objective, not spreading for resilience. When that node fails, all three replicas go down together, at the same moment as whatever else was on that node.

Topology spread constraints fix this directly:

# Illustrative — spreads replicas evenly across nodes, tolerating a 1-pod skew.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 3
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: checkout-api
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: checkout-api

Two constraints doing two different jobs here: the node-level one with DoNotSchedule is a hard requirement (better to fail scheduling than violate node spread), while the zone-level one with ScheduleAnyway is a best-effort preference — a reasonable split when you have more nodes than zones and want zone spread to yield to actually getting scheduled if a zone is momentarily full. Pod anti-affinity (podAntiAffinity with requiredDuringSchedulingIgnoredDuringExecution) achieves a similar node-spread result and predates topology spread constraints; the newer API is generally easier to reason about for multi-dimensional spreading (node and zone at once), which is why it's the current recommendation.

Zone spread specifically matters because node failure and availability-zone failure are different failure domains with different blast radii — three replicas correctly spread across nodes but all in one AZ survive a node failure and not a zone-level event (a real, if infrequent, occurrence on every major cloud provider). Whether that's a risk worth engineering around depends on the workload's actual availability requirement, not a default you apply everywhere.

PodDisruptionBudgets: the setting that silently blocks maintenance

A PodDisruptionBudget tells Kubernetes the minimum availability to preserve during voluntary disruptions — node drains for maintenance, cluster autoscaler scale-downs, kubectl drain before an upgrade. It does nothing for involuntary disruptions (a node crashing outright), which is a distinction worth being explicit about since PDBs are sometimes assumed to be a general reliability mechanism.

# Illustrative
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: checkout-api

With replicas: 3 and minAvailable: 2, a drain can evict one pod at a time, wait for its replacement to become Ready, and continue — normal, expected behavior. The mistake that comes up constantly: setting minAvailable equal to the replica count (or maxUnavailable: 0), intending "always fully available," which actually means the PDB permits zero voluntary evictions ever. A node drain during a routine cluster upgrade then hangs indefinitely on that PDB (or gets forcibly overridden, if whoever's draining the node has --disable-eviction, which defeats the entire purpose of having a PDB). If you actually need zero-disruption guarantees, that's a signal you need more replicas, not a stricter PDB on the replica count you already have.

Readiness probes vs. liveness probes: different mechanisms, frequently conflated

These solve two unrelated problems and get treated as interchangeable constantly:

• Readiness probes control whether a pod receives traffic through its Service. A pod that fails readiness stays running, just gets pulled from Service endpoints until it passes again. This is the mechanism that makes rolling deployments safe — new pods don't receive traffic until they report ready, so a bad rollout doesn't get real requests routed to broken instances.
• Liveness probes control whether the kubelet restarts the container. A pod that fails liveness gets killed and restarted, full stop — no matter what state it's in or how long it's been slow to respond for a legitimate reason.

# Illustrative — deliberately different thresholds for two different jobs.
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /healthz/live
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 5

The dangerous version of conflating them: an aggressive liveness probe (short initialDelaySeconds, low failureThreshold) on a service with a genuinely slow startup — a JVM app warming up, a service that runs migrations on boot — gets killed and restarted mid-startup, repeatedly, because the liveness probe interprets "not ready yet" as "dead." That's CrashLoopBackOff self-inflicted by probe configuration, not an application problem, and it's worth ruling out early when diagnosing CrashLoopBackOff in Kubernetes. The fix is either a generous initialDelaySeconds/failureThreshold on the liveness probe, or better, a startup probe that suppresses liveness checks entirely until the application reports it's actually started.

Readiness and liveness endpoints should also check different things. /healthz/live should answer "is the process fundamentally functional" — not "are my downstream dependencies up," because a downstream outage restarting every dependent pod simultaneously turns one outage into a cascading one. /healthz/ready, by contrast, is the right place to check real dependencies (database connection pool, cache, required upstream services), since removing an individual pod from Service endpoints when it can't actually serve traffic is exactly the intended behavior.

Rolling deployment strategy: maxUnavailable and maxSurge

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 1
    maxSurge: 1

maxUnavailable caps how many pods can be down during the rollout; maxSurge caps how many extra pods can be created above the desired replica count while the rollout progresses. maxUnavailable: 0, maxSurge: 1 gives you a rollout with no capacity reduction at any point (strictly additive, then old pods removed after new ones are Ready), at the cost of briefly running above your normal replica count — usually the right default when you have the headroom, and depends entirely on readiness probes being accurate, since "new pod exists" and "new pod is actually ready to serve" are different states and the rollout only respects the second one if the probe is configured to detect it correctly.

Common mistakes

• Treating replica count as sufficient without topology spread or anti-affinity — three replicas that can all land on one node is a common finding, not a hypothetical.
• Setting minAvailable to the full replica count on a PDB, blocking every voluntary disruption including the ones that keep the cluster patched and upgraded.
• Using one health endpoint for both readiness and liveness probes, which means a downstream dependency outage restarts pods that were otherwise fine — the opposite of what you want during an incident.
• No startup probe on slow-starting applications, leaving the liveness probe to either restart pods mid-boot (too aggressive) or take unacceptably long to detect a genuinely stuck container (too lenient, tuned to accommodate slow startup).
• Assuming HA within a region covers a regional failure. It doesn't, and conflating the two is a common and expensive mistake — see the distinction with disaster recovery below.

Production considerations

• HA and DR are different problems. High availability, as covered here, is about surviving the loss of a pod, node, or zone while the cluster and your data remain intact. Disaster recovery is about what happens when you lose the cluster, the region, or the underlying data itself — a different set of mechanisms (backups, restore testing, cross-region replication) entirely. A cluster with excellent HA and no tested backup/restore process is not disaster-resilient; see Kubernetes Backup and Disaster Recovery for where that line is and what's actually required on the other side of it.
• Resource pressure undermines HA mechanisms you've already configured — a node under memory pressure evicts pods regardless of your PDB (node-pressure eviction is involuntary, not covered by PDBs), which is one more reason resource requests and limits being accurate matters beyond just performance.
• Control-plane availability is a separate concern from workload availability. On managed Kubernetes (EKS, GKE, AKS) the control plane's HA is the provider's responsibility and typically already multi-AZ; on self-managed clusters, control-plane HA (etcd quorum, multiple API server instances) is your responsibility and a materially different engineering problem than workload HA, worth confirming explicitly rather than assuming it's covered by the same effort that made your Deployments resilient.
• Test the failure, don't just configure for it. Cordon and drain a node in a non-production environment and watch what actually happens — that's the only way to confirm the PDB, topology spread, and probes are working together the way the YAML suggests they should.

Conclusion

High availability in Kubernetes is the sum of several independently-configured mechanisms, each of which fails in a specific, findable way when misconfigured: replicas without spread constraints share a failure domain, a PDB set too strictly blocks the maintenance it wasn't meant to block, and probes that don't distinguish "starting up" from "genuinely broken" cause the exact restarts they're supposed to prevent. None of it is exotic — it's a handful of settings that need to agree with each other and with what the workload actually needs, which is different for every service and worth deciding deliberately rather than copying defaults.

If reliability engineering like this is part of a broader gap — resource sizing nobody's revisited, no tested disaster recovery path, availability assumptions nobody's actually tested with a real node drain — that's exactly the kind of production-readiness work covered under Kubernetes Consulting. For a structured look at where a specific cluster stands across all of this, see the Kubernetes production readiness guide or the Kubernetes Cluster Audit.