The most common request I get isn't "set up monitoring" — it's "we have monitoring, but the alerts are useless and everyone's ignoring them." That's not a Prometheus problem. It's an alert-design problem.
Alert on symptoms, not causes
Don't alert on "CPU is at 85%." Alert on "the API is returning 5xx to real users" or "the queue is growing faster than it's draining." CPU at 85% might be completely fine. A growing error rate never is.
Every alert needs an owner and a runbook link
If an alert fires and the on-call engineer doesn't know who owns the service or what to do next, that alert is training people to ignore pages. Every alert rule should link straight to a runbook.
Separate "wake someone up" from "check it in the morning"
Not everything is a page. Route low-urgency signals to a Slack channel, and reserve PagerDuty/phone calls for things that are actually customer-facing and actually urgent right now.
Delete alerts nobody has acted on in 90 days
If an alert has fired repeatedly and nobody has ever changed behavior because of it, it's noise. Either fix the underlying issue so it stops firing, or delete the alert.
Getting this right usually cuts on-call noise by 70-80% within the first month — without missing anything that actually mattered.
If your team is drowning in alerts, Kubernetes Observability & Monitoring or the Prometheus + Grafana Monitoring package can rebuild this from the ground up. For a broader platform review, book a free call to talk through it.