How alerts work

Every row the alert query returns is a firing instance. An empty result means resolved. There is no separate threshold field, no pending duration, no state machine to configure — the SQL is the whole condition.

The evaluation loop

Everr runs spec.query every evaluationInterval. Notifications fire on the edges — when an instance newly starts firing and when it resolves — not on every evaluation while it stays firing. A service whose event loop has been blocked for an hour produces one "firing" notification, not one per minute.

kind: AlertRule
metadata:
  name: eventloop-delay
spec:
  evaluationInterval: 1m
  query: |
    SELECT ServiceName, round(avg(Value) * 1000, 1) AS p99_delay_ms
    FROM metrics_gauge
    WHERE MetricName = 'nodejs.eventloop.delay.p99'
      AND TimeUnix >= now() - INTERVAL 5 MINUTE
    GROUP BY ServiceName
    HAVING p99_delay_ms > 100

The threshold lives in the SQL — a WHERE filter, a HAVING clause, whatever comparison defines "something is wrong." A healthy service sits around 10ms, so HAVING p99_delay_ms > 100 returns rows only when the loop is genuinely blocked.

Instance identity

By default an instance's identity is the set of string columns the query returns; numeric columns are measurements, not identity. The rule above returns ServiceName (a string) and p99_delay_ms (a number), so it produces one instance per service automatically — a single rule scales across every service, firing independently for checkout, payments, and anything else. Because the numeric column is not part of identity, a service stays the same instance as its delay rises and falls. To carry an extra string column in the message without churning identity, override which columns count. See alert queries.

Flapping and windows

The query window is the smoothing mechanism: a 5-minute lookback evaluated every minute averages a single momentary blip into the window instead of notifying on it. Align the window to the interval to control flapping.

Notifications

Each evaluation with changes sends one notification summarizing every instance that newly fired or resolved. Channels are org-wide app configuration, not part of the rule's code. See set up notifications.