Too many false positive alerts

Symptoms

Alerts fire frequently, but investigations show no customer-impacting issue—classic false positives.

Likely causes

• Single-region checks hitting transient packet loss
• Aggressive timeouts during deploys or autoscaling events
• WAF/bot mitigation blocking probes
• DNS flapping or third-party dependency glitches unrelated to core user flows

Remediation playbook

1. Confirm with multi-region failure correlation before paging.
2. Tune timeouts and required consecutive failures.
3. Stabilize tests—avoid brittle content assertions on dynamic pages.
4. Schedule maintenance windows for expected noisy changes.
5. Route low-severity alerts to tickets instead of pages.

Culture note

If on-call dreads alerts, they will miss real incidents. Treat noisy monitors as broken monitors.