Most teams don't have too little Kubernetes monitoring — they have the wrong coverage. Dashboards full of CPU and memory graphs, and no signal for the things that actually cause outages: pods stuck Pending for ten minutes, an HPA that stopped scaling three deploys ago, a PVC that's been at 94% for a week. Nobody notices until something falls over, because nothing was watching the part that actually broke.
This is about what to instrument before you get to alerting on it. Alert design — how to turn signals into pages without burying your team in noise — is a separate problem, covered in building an observability stack that doesn't page you at 3AM. This article is upstream of that one: you can't design good alerts on signals you don't have.
Short answer
If you're starting from close to nothing, prioritize in this order:
- Container restarts and OOMKills — the fastest-moving signal for workload instability.
- Pod scheduling failures — Pending pods that have been unschedulable for more than a few minutes.
- Node conditions — MemoryPressure, DiskPressure, and Ready flapping.
- HPA behavior — is it actually scaling, or stuck at min/max replicas while load says otherwise.
- PVC capacity — approaching full is a predictable, preventable outage.
- API server latency and error rate — the control plane degrading is often the earliest sign of a cluster-wide problem.
- Kubernetes Events, shipped somewhere durable — the most underused signal source in most clusters.
Everything below explains why each of these matters and how to actually get at it.
Metrics, logs, and traces — briefly
The three pillars apply to Kubernetes the same way they apply anywhere else, but the value isn't evenly distributed. Metrics (via Prometheus) are where almost all of the Kubernetes-specific signal lives — the scheduler, kubelet, and control plane expose state as metrics, not logs. Logs matter for application-level debugging once you know something is wrong. Traces matter once you have enough services that "which hop introduced the latency" stops being answerable by looking at one dashboard. For a team instrumenting Kubernetes for the first time, metrics coverage is where the return on effort is highest — that's the focus here.
The signals, and where they come from
Two exporters do most of the work: kube-state-metrics exposes the state of Kubernetes objects (pod phase, deployment replica counts, PVC status) as Prometheus metrics — it's not resource usage, it's object state, and it's the piece most default Helm-chart monitoring setups under-use. node-exporter covers host-level resource metrics. Both are standard, both ship with kube-prometheus-stack, and if you only have one of the two, you're missing half the picture — usage without object state, or object state without knowing if a node is actually the cause.
Container restarts and OOMKills
A restart count on its own is a lagging indicator — by the time it's incremented, the container already died. What's useful is the rate, and specifically catching it climbing before it becomes a full CrashLoopBackOff:
sum(rate(kube_pod_container_status_restarts_total[15m])) by (namespace, pod)
Pair this with kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} — a nonzero value here means a container was killed for exceeding its memory limit, which is a resourcing problem, not a code crash, and gets misdiagnosed as one constantly. If you haven't sized requests and limits deliberately, OOMKills are often the first evidence that they're wrong — see the practical side of that in Kubernetes resource requests and limits.
Pod scheduling failures
A pod stuck Pending is silent until something (usually a person) notices a deployment never finished. kube_pod_status_phase{phase="Pending"} combined with pod age tells you a pod has been unschedulable for longer than a normal scheduling delay — which almost always means insufficient node capacity, a taint/toleration mismatch, or a PVC that can't bind. This is worth watching with a duration threshold (Pending for >5 minutes, not any Pending state, which happens normally during every rollout).
Node conditions
kube_node_status_condition{condition="MemoryPressure",status="true"} and the DiskPressure equivalent tell you a node is about to start evicting pods — before the eviction happens, not after. kube_node_status_condition{condition="Ready",status="unknown"} catches nodes flapping in and out of readiness, which is a different failure mode than a node cleanly going NotReady, and often points at kubelet or network issues rather than resource exhaustion.
HPA behavior
The HPA existing doesn't mean it's working. kube_horizontalpodautoscaler_status_current_replicas compared against kube_horizontalpodautoscaler_status_desired_replicas over time will show you an HPA that's stuck — hasn't moved in days despite load changing, or is pinned at maxReplicas and quietly under-provisioning. A stuck HPA is worse than no HPA, because the team believes autoscaling is handling load that it isn't.
PVC capacity
kubelet_volume_stats_available_bytes versus kubelet_volume_stats_capacity_bytes gives you real fill percentage per volume. A PVC filling up is one of the most preventable production incidents in Kubernetes — the data is there well in advance, it's just usually not watched until the volume is full and the application starts failing writes.
API server latency and error rate
apiserver_request_duration_seconds and the 5xx rate on apiserver_request_total are control-plane health signals, and they matter because API server degradation cascades — controllers stop reconciling, the scheduler stalls, kubectl commands hang, and none of it looks like an API server problem from the application layer. On managed Kubernetes (EKS, GKE, AKS) you may have limited or no direct scrape access to these; check your provider's control-plane metrics offering (e.g., EKS control plane logging to CloudWatch, GKE's built-in control-plane metrics) rather than assuming you can scrape the API server directly.
etcd health, where you can see it
If you manage your own control plane, etcd_server_has_leader and etcd_disk_wal_fsync_duration_seconds (fsync latency, the classic etcd early-warning signal) matter — etcd degrading is a cluster-wide failure mode, not a component-level one. On managed Kubernetes you generally don't have etcd access at all; this is one of several control-plane internals that a Kubernetes API-only audit can't observe either — worth knowing as a scope boundary, not something to fake instrumentation for.
Kubernetes Events — the underused one
kubectl get events -A --sort-by=.lastTimestamp
Events carry information that never becomes a metric — a specific scheduling failure reason, an image pull error, a volume mount failure — but they're stored in etcd with a short retention window (an hour by default in most distributions), so if nobody's watching in real time, they're gone before anyone looks. Shipping events somewhere durable (a log aggregator, or a tool like kube-eventer/an events exporter feeding Loki or Elasticsearch) turns a signal that currently gets lost into one you can actually query after the fact — "what happened to this pod at 3am" shouldn't depend on someone having had a terminal open at the time.
A small set of illustrative PromQL queries
# Restart rate by pod, last 15 minutes
sum(rate(kube_pod_container_status_restarts_total[15m])) by (namespace, pod)
# Nodes currently under memory pressure
kube_node_status_condition{condition="MemoryPressure", status="true"}
# PVCs above 90% utilized
(1 - kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes) > 0.9
# HPA at max replicas — possible under-provisioning
kube_horizontalpodautoscaler_status_current_replicas
== kube_horizontalpodautoscaler_spec_max_replicas
These are illustrative starting points, not tuned production rules — thresholds and durations depend on your workload's actual behavior, which is exactly the kind of thing alert design (not signal selection) needs to account for.
Common mistakes
- Instrumenting resource usage and stopping there. CPU/memory graphs tell you a node is busy; they don't tell you a pod can't schedule, an HPA is stuck, or a PVC is about to fill. Object-state metrics (kube-state-metrics) are usually the bigger gap, not usage metrics.
- Treating restart count as the signal instead of restart rate. A count climbs forever and tells you nothing about whether the situation is active or historical; rate over a window tells you if it's happening right now.
- Leaving Kubernetes Events unshipped. They expire in etcd within the hour and are the most detailed signal available for scheduling and volume failures — losing them by default is a choice, even if an unintentional one.
- Assuming API server/etcd metrics are available on managed Kubernetes. They often aren't, directly — check your provider's specific control-plane observability offering instead of assuming a standard scrape target exists.
- Building dashboards before deciding what actually needs an alert. Coverage and alerting are separate steps; conflating them is how you end up with 40 panels and 3 alerts that matter, in either order.
Production considerations
- Retention matters as much as collection — a signal that exists for 15 days but the incident happened 20 days ago is functionally not there. Set retention deliberately, not on Prometheus's default.
- Cardinality is a real cost with kube-state-metrics at scale — labeling everything by pod name in a cluster with high pod churn (CI runners, batch jobs) can blow up your time-series count. Know what you're paying for before you add another label.
- SLOs are the mechanism that turns "we have a lot of metrics" into "we know if the service is actually healthy" — a small number of signals tied to what users actually experience (successful request rate, latency at the percentile that matters) beats broad instrumentation with no prioritization. This is also the natural transition into alert design, not something to solve in the same pass as picking metrics.
Conclusion
The Kubernetes-specific gap in most monitoring setups isn't resource usage — that part's usually covered. It's object state: pods stuck Pending, HPAs that stopped moving, PVCs quietly filling, events expiring unread. Those are the signals that predict an incident instead of just describing one after it's already visible everywhere else. Get those in place, then move to alert design — building an observability stack that doesn't page you at 3AM covers turning this into alerts that don't create fatigue. And if observability coverage is one gap among several — alongside RBAC, resource limits, or availability patterns nobody's revisited — that's the kind of thing a Kubernetes production readiness review is built to surface in one pass.
If you want this built out properly rather than assembled ad hoc, Kubernetes Observability & Monitoring covers exactly this — the right signals, wired to alerts your team will actually act on.