Writing good alerts

Good alerts start as queries: this rule fires one instance per service whose error rate exceeds 5% over 15 minutes, and stays quiet on tiny samples:

kind: AlertRule
metadata:
  name: error-spike
spec:
  evaluationInterval: 1m
  notificationMessage:
    title: "${ServiceName} error rate is high"
    description: "Error rate is ${error_pct}% over the last 15 minutes."
  query: |
    SELECT
      ServiceName,
      count() AS total,
      countIf(SeverityNumber >= 17) AS errors,
      round(errors / total, 4) AS error_rate,
      round(errors / total * 100, 2) AS error_pct
    FROM logs
    WHERE Timestamp >= now() - INTERVAL 15 MINUTE
    GROUP BY ServiceName
    HAVING total >= 100
      AND error_rate > 0.05
    ORDER BY error_rate DESC
    LIMIT 50

Every row is a firing instance; empty result = resolved (how alerts work).

Choosing the threshold and window

• Guard percentage alerts with a minimum-volume condition — one failure out of one request is a 100% error rate that should notify no one; the HAVING total >= 100 clause keeps tiny samples quiet.
• Align the query window to the evaluation interval — a 15-minute lookback evaluated every 1m smooths a momentary blip into the window instead of firing on it.
• Exclude known-benign services and environments in SQL — filter out test services, development environments, or maintenance jobs when those rows don't represent a real problem.

What makes it worth alerting on

Before shipping the rule, check it against Google's SRE practice:

• Alert on symptoms, not causes — notify on what users feel (elevated errors), not on every underlying cause.
• Make every alert actionable — if no one would act when it fires, it shouldn't fire.
• Tune for signal, not noise — pick a threshold and window that stay quiet under normal conditions.

Next: link the alert to a runbook so responders land on live panels. SQL patterns (thresholds, identity, templating): Alert query patterns.