Set up alert notifications

Alert rules are defined as code, but delivery is configured in the app and applies across the whole organization. Everything below lives under Alerts in Everr: the default destination and channels on the Notifications page, silences on the Silences page.

The delivery model

Every firing and resolved alert delivers to the organization's default destination: a set of channels you pick once. There is no routing tree to configure.

1. If the alert's rule declares its own channels (spec.notifications.channels), those channels receive it and the default destination is skipped.
2. Otherwise the alert goes to the default destination: one channel list for all alerts, or one list per severity when you split.
3. Silences hold delivery without stopping evaluation. The alert keeps firing.

Your first channel automatically becomes the default destination, so creating one channel is a complete setup.

Channels

A channel is a named delivery endpoint, created on the Notifications page with New channel and edited or deleted from its row. Each row shows what the channel receives, how many notifications reached it in the selected time range, how many never got through, and when it last sent. Channel types: Slack (an Incoming Webhook URL), Discord (a channel webhook URL), webhook (any HTTPS endpoint), and Telegram (a bot token and chat IDs, see Telegram's BotFather guide).

Webhook URLs (generic, Slack, and Discord) and Telegram bot tokens are write-only: after saving, Everr shows them redacted (***). To rotate a redacted secret, edit the channel and enter the new value; the secret fields start blank on edit precisely because the stored values are never shown again. Names are editable labels.

Channels can always be deleted. A deleted channel drops out of the default destination. A rule that named it directly does not fall back to the default destination: it stays pointed at the name, so its alerts are not delivered until the name exists again, and each one is recorded as undelivered. The rule's as-code spec keeps the name, so re-creating the channel (or the next everr apply) links it up again. Only a channel with notifications still sending is briefly refused; retry once they settle.

The default destination

The Default targets list on the Notifications page shows where alerts go. By default it is one row: All alerts and its channels. Each channel's row also says what it receives, and the Not delivered list names every tier or rule whose alerts had no channel in the selected time range, with the one action that closes the gap.

Open Edit delivery and Split by severity to send different severities different places: one channel list each for critical, warning, and info. severity is set on every alert rule (see the AlertRule spec). A tier with no channels is a delivery gap: those alerts are not delivered, and the Triage board says so. A rule that names a channel of its own is treated as delivered there, so the board does not catch a rule pointing at a name that no longer exists.

Rule overrides

A rule that names spec.notifications.channels in its YAML delivers to exactly those channels and skips the default destination. The Rule overrides list shows these rules read-only; overrides are changed in the rule's YAML and applied as code. See the AlertRule spec.

Grouping and timing

Grouping decides which alert instances are batched into the same Slack message or webhook payload. It is fixed, with no knobs:

• Alerts group by rule and severity, so one rule firing on 20 pods produces one notification with 20 instances, and unrelated rules never share a message.
• A new group waits 10 seconds before its first send, so a burst becomes one notification.
• A group that already notified sends updates at most every 5 minutes.
• Notifications go out when the group's contents change: new instances, and resolutions. There are no repeat notifications for an unchanged group.

Silences

To stop notifications for a known issue or maintenance window without touching the rule, create a silence on the Silences page: matchers select the alerts (label comparisons such as severity = critical, using equals or not-equals operators), a time window bounds the suppression, and a comment records why. Silences can be scheduled ahead of a maintenance window. The rule keeps evaluating the whole time, and the alert keeps firing: a silence stops the message, never the alerting. A notification held by a silence is not dropped. It waits for the window to end and then goes out, unless the alert stopped firing first, in which case nothing is sent. Both outcomes are recorded in the event stream, so a held notification is visible while it waits.

For when notifications fire, see How alerts work.