Alert query patterns

The query returns useful data for both alerting and visualization. Every row must contain a numeric column named value; spec.condition decides which rows are firing instances.

Writing an alert? Start from Writing good alerts, then link it to a runbook. This page is the SQL reference behind both.

Unlike panel queries, alert SQL does not support ${...} template tokens; those are only for message templates.

Express a threshold

Use WHERE to scope the data you measure, but keep healthy rows in the result. Alias the measured signal to value, then configure the threshold separately:

SELECT ServiceName, round(avg(Value) * 1000, 1) AS value
FROM metrics_gauge
WHERE MetricName = 'nodejs.eventloop.delay.p99'
  AND TimeUnix >= now() - INTERVAL 5 MINUTE
GROUP BY ServiceName

condition: { operator: gt, threshold: 100 }

When no service crosses the threshold, the query still returns values for a panel while the alert is resolved.

For percentage alerts, guard against tiny samples. One failed request out of one total request is technically a 100% error rate, but it usually should not notify anyone. Add a minimum-volume condition alongside the rate:

SELECT
  ServiceName,
  count() AS total,
  countIf(SeverityNumber >= 17) AS errors,
  round(errors / total, 4) AS error_rate,
  if(total >= 100, error_rate, 0) AS value
FROM logs
WHERE Timestamp >= now() - INTERVAL 15 MINUTE
GROUP BY ServiceName
ORDER BY error_rate DESC
LIMIT 50

condition: { operator: gt, threshold: 0.05 }

Filter known-benign data in SQL. For example, exclude test services, development environments, or maintenance jobs when those rows do not represent a real problem.

Instance identity

By default, an instance's identity is the set of string columns the query returns. The query above returns ServiceName (a string) and value (a number), so it produces one instance per service automatically, a single rule scales across every service instead of needing one rule each. Because the numeric value column is not part of identity, a service stays the same instance as its delay rises and falls.

The common reason to override is a string column you want in the notification but not in the identity. Say you surface the worst-affected host so the message can name it:

spec:
  query: |
    SELECT
      ServiceName,
      argMax(ResourceAttributes['host.name'], Value) AS worst_host,
      round(avg(Value) * 1000, 1) AS value
    FROM metrics_gauge
    WHERE MetricName = 'nodejs.eventloop.delay.p99'
      AND TimeUnix >= now() - INTERVAL 5 MINUTE
    GROUP BY ServiceName
  condition: { operator: gt, threshold: 100 }

The query returns one row per service, but worst_host is also a string, so by default the identity is ServiceName and worst_host. Host names like app-f9867cfd6-7p8qf change on every redeploy and as load moves around, so the identity would churn: the old instance resolves and a new one fires each time the worst host changes, spamming notifications for a service that never actually recovered.

Set spec.instanceLabels to pin the identity to just the service, keeping worst_host available for the message:

spec:
  instanceLabels: [ServiceName]   # identity is the service, regardless of host

The columns you list become the entire identity; any other string column is carried along only for the message. Make sure the query returns a single row per identity, if two rows share the same identity, they collapse into one instance.

Choose an evaluation interval

spec.evaluationInterval is an <integer><s|m|h|d> duration with a minimum of 1m (for example 1m, 5m, 1h). Align the query's time window to the interval: a 5-minute lookback evaluated every minute smooths transient spikes, because a single momentary blip is averaged into the window rather than notifying on its own.

Everr notifies on the firing and resolved edges as soon as the rule evaluates. Set spec.for to require a matching condition for a duration before firing, and spec.resolveAfter to tolerate isolated non-matching evaluations.

Keep results small

The evaluator reads every row before applying the condition. Emit one row per instance, avoid SELECT * on large sets, aggregate early, and add a LIMIT for safety on broad queries. Do not put the alert threshold in WHERE or HAVING, because that hides healthy values from panels.

Return the columns needed for identity, message templates, and the measured values that explain why the alert fired. Extra high-cardinality string columns can create extra instances unless you override spec.instanceLabels.