[{"data":1,"prerenderedAt":55},["ShallowReactive",2],{"blog-\u002Fblog\u002Falert-fatigue-is-a-design-problem":3},{"id":4,"title":5,"body":6,"date":43,"description":44,"extension":45,"meta":46,"navigation":47,"path":48,"seo":49,"stem":50,"tags":51,"__hash__":54},"blog\u002Fblog\u002Falert-fatigue-is-a-design-problem.md","Alert fatigue is a design problem, not a tuning problem",{"type":7,"value":8,"toc":37},"minimark",[9,13,16,21,24,27,31,34],[10,11,12],"p",{},"Every on-call engineer knows the feeling: a 3am page for something that resolves itself in ninety seconds. Do it enough times and the instinct kicks in — glance at the phone, see it's \"probably nothing,\" go back to sleep. That instinct is exactly how the alert that matters gets missed.",[10,14,15],{},"The usual fix is tuning: raise the threshold, add a longer evaluation window, require three consecutive failures instead of one. That helps, but it's treating the symptom. The actual problem is upstream — most alerting setups don't distinguish between \"something changed\" and \"something is broken.\"",[17,18,20],"h2",{"id":19},"two-different-questions","Two different questions",[10,22,23],{},"\"Is this metric outside its normal range?\" is a monitoring question. \"Does a human need to act on this right now?\" is an alerting question. They're not the same question, and treating every anomaly as page-worthy is what causes fatigue in the first place.",[10,25,26],{},"A CPU spike at 2am might be a real problem or a scheduled batch job. A single failed health check might be a real outage or a network blip. The alert that matters is the one correlated with actual user impact — requests failing, latency crossing a threshold users would notice, a dependency actually being unreachable.",[17,28,30],{"id":29},"what-we-optimize-for","What we optimize for",[10,32,33],{},"This is the design question behind Reliable Uptime's alerting: not \"how sensitive can we make the check,\" but \"how do we only wake someone up when it's real.\" Retries before declaring something down, checks from multiple regions before trusting a single failed probe, and alert routing that matches severity to urgency instead of treating every check the same way.",[10,35,36],{},"None of this is a novel idea — it's standard SRE practice. It's just surprising how often monitoring tools skip it in favor of a simpler \"threshold crossed, send alert\" model that looks fine in a demo and generates fatigue within a month of real production traffic.",{"title":38,"searchDepth":39,"depth":39,"links":40},"",2,[41,42],{"id":19,"depth":39,"text":20},{"id":29,"depth":39,"text":30},"2026-07-08","Most teams respond to alert fatigue by tuning thresholds. That treats the symptom. The real fix is deciding what actually deserves a page.","md",{},true,"\u002Fblog\u002Falert-fatigue-is-a-design-problem",{"title":5,"description":44},"blog\u002Falert-fatigue-is-a-design-problem",[52,53],"monitoring","engineering","QEK_02Ledenjejm6TjdGzXbz_bHD9bLGo2X6fQkkIHo",1783640617125]