Writing good alerts

Good alerts start as queries: this rule fires one instance per service whose error rate exceeds 5% over 15 minutes, and stays quiet on tiny samples:

kind: AlertRule
metadata:
  name: error-spike
spec:
  evaluationInterval: 1m
  notificationMessage:
    title: "${ServiceName} error rate is high"
    description: "Error rate is ${error_pct}% over the last 15 minutes."
  query: |
    SELECT
      ServiceName,
      count() AS total,
      countIf(SeverityNumber >= 17) AS errors,
      round(errors / total, 4) AS error_rate,
      round(errors / total * 100, 2) AS error_pct,
      if(total >= 100, error_rate, 0) AS value
    FROM logs
    WHERE Timestamp >= now() - INTERVAL 15 MINUTE
    GROUP BY ServiceName
    ORDER BY error_rate DESC
    LIMIT 50
  condition:
    operator: gt
    threshold: 0.05

The query keeps both healthy and breaching services available for visualization. The condition turns only rows with a guarded error rate above 5% into firing instances (how alerts work).

Choosing the threshold and window

• Guard percentage alerts with a minimum-volume signal. One failure out of one request is a 100% error rate that should notify no one; if(total >= 100, error_rate, 0) AS value keeps tiny samples below the configured threshold.
• Align the query window to the evaluation interval: a 15-minute lookback evaluated every 1m smooths a momentary blip into the window instead of firing on it.
• Exclude known-benign services and environments in SQL: filter out test services, development environments, or maintenance jobs when those rows don't represent a real problem.

What makes it worth alerting on

Before shipping the rule, check it against Google's SRE practice:

• Alert on symptoms, not causes: notify on what users feel (elevated errors), not on every underlying cause.
• Make every alert actionable: if no one would act when it fires, it shouldn't fire.
• Tune for signal, not noise: pick a threshold and window that stay quiet under normal conditions.

Next: link the alert to a runbook so responders land on live panels. SQL patterns (thresholds, identity, templating): Alert query patterns.