Escalation ladder

A single alert is easy to miss — someone's asleep, the Slack channel is muted, the on-call handoff slipped. The escalation ladder makes sure a critical problem keeps getting louder until a person actually responds.

Escalation applies only to critical alerts, and only while they're open and unacknowledged.

Re-notify

Set a re-notify interval on a monitor, and while a critical alert stays open and unacknowledged, CronWarden re-sends it to its critical channels every that-many minutes. The reminders stop the moment someone acknowledges the alert or the monitor recovers.

Fallback

For the case where the primary channel gets no response at all, set a fallback: a delay in minutes and a fallback integration. If the critical alert is still open and unacknowledged after that delay, CronWarden notifies the fallback integration once, then includes it in every following reminder. This is how an unanswered page rolls over to a secondary person or a louder channel.

Both parts of the fallback — the delay and the integration — must be set for it to fire.

Acknowledging

Acknowledging an alert tells CronWarden a human is on it: the re-notifications and fallback stop, but the monitor keeps showing its real state until the underlying problem clears. When the monitor recovers, a Recovered notice goes to the same channels, so everyone who was pulled in knows it's over.

Setting it up

Re-notify and fallback are configured per monitor in the dashboard, so you can escalate hard on the jobs that matter and leave the rest to a single notification.