Alert fatigue is a design problem, not a tuning problem

Every on-call engineer knows the feeling: a 3am page for something that resolves itself in ninety seconds. Do it enough times and the instinct kicks in — glance at the phone, see it's "probably nothing," go back to sleep. That instinct is exactly how the alert that matters gets missed.

The usual fix is tuning: raise the threshold, add a longer evaluation window, require three consecutive failures instead of one. That helps, but it's treating the symptom. The actual problem is upstream — most alerting setups don't distinguish between "something changed" and "something is broken."

Two different questions

"Is this metric outside its normal range?" is a monitoring question. "Does a human need to act on this right now?" is an alerting question. They're not the same question, and treating every anomaly as page-worthy is what causes fatigue in the first place.

A CPU spike at 2am might be a real problem or a scheduled batch job. A single failed health check might be a real outage or a network blip. The alert that matters is the one correlated with actual user impact — requests failing, latency crossing a threshold users would notice, a dependency actually being unreachable.

What we optimize for

This is the design question behind Reliable Uptime's alerting: not "how sensitive can we make the check," but "how do we only wake someone up when it's real." Retries before declaring something down, checks from multiple regions before trusting a single failed probe, and alert routing that matches severity to urgency instead of treating every check the same way.

None of this is a novel idea — it's standard SRE practice. It's just surprising how often monitoring tools skip it in favor of a simpler "threshold crossed, send alert" model that looks fine in a demo and generates fatigue within a month of real production traffic.