Add a runbook
Give your alert a response procedure with live telemetry beside the steps.
A runbook helps you respond when an alert fires: what it means, how to investigate, and which actions are appropriate. Keep the evidence beside the instructions with panels that query live telemetry.
This guide links a runbook to the latency alert you just created. Its YAML and Markdown files belong in the same everr/ directory as the dashboard and alert.
Use your coding agent
From your project, ask:
/everr-setup-resources Add a runbook for the alert we just created. Explain when to act, embed panels for the signal and supporting evidence, and write investigation and recovery steps grounded in this app. Link it to the alert and preview both changes.The agent reads the rule and telemetry, writes the procedure, and stages the runbook and alert update in a preview. Review the suggested actions against how your app is deployed and operated.
Write it manually
A runbook has a YAML declaration and a Markdown document. Save this as everr/api-latency.runbook.yaml:
kind: Runbook
metadata:
name: api-latency
spec:
display:
name: Slow API requests runbook
description: Investigate sustained high request latency.
markdown:
file: ./api-latency.runbook.mdThe Markdown path is relative to the YAML file. Save the procedure below as everr/api-latency.runbook.md, replacing my-service in both queries with the service from your alert:
# Slow API requests
Use this procedure when the `api-latency` alert fires. It tracks
p99 latency for successful server requests in production, with a
minimum of 100 requests over the last 5 minutes.
## Confirm the scope
Select a time range around the alert. Compare latency before and
after it began. The chart uses the same request filters as the rule;
its buckets follow the time picker rather than the rule's 5-minute window.
```panel
kind: Panel
height: 280
spec:
display: { name: p99 request latency (ms) }
plugin:
kind: TimeSeriesChart
spec: { showLegend: true, valueFormat: { unit: ms } }
queries:
- kind: ClickHouseSQL
spec:
plugin:
kind: ClickHouseSQL
spec:
query: |
SELECT toStartOfInterval(Timestamp, INTERVAL {step:UInt32} SECOND) AS ts,
quantile(0.99)(Duration) / 1e6 AS p99_ms
FROM traces
WHERE ServiceName = 'my-service'
AND Timestamp >= {from:String} AND Timestamp <= {to:String}
AND SpanKind = 'Server'
AND StatusCode != 'Error'
AND toString(ResourceAttributes.`deployment.environment.name`) = 'production'
GROUP BY ts
ORDER BY ts
```
## Find slow requests
Use the examples below to identify repeated operations and trace IDs
for investigation. This table shows successful requests, matching the
latency signal; investigate errors separately if failures are also rising.
```panel
kind: Panel
height: 350
spec:
display: { name: Slowest successful requests }
plugin:
kind: Table
spec:
columns:
duration_ms:
valueFormat: { unit: ms }
queries:
- kind: ClickHouseSQL
spec:
plugin:
kind: ClickHouseSQL
spec:
query: |
SELECT Timestamp, SpanName, Duration / 1e6 AS duration_ms, TraceId
FROM traces
WHERE ServiceName = 'my-service'
AND Timestamp >= {from:String} AND Timestamp <= {to:String}
AND SpanKind = 'Server'
AND StatusCode != 'Error'
AND toString(ResourceAttributes.`deployment.environment.name`) = 'production'
ORDER BY Duration DESC
LIMIT 20
```
1. Identify which operations slowed down and when the change began.
2. Investigate a representative trace to locate slow dependencies or work.
3. Compare the timing with recent releases and traffic changes.
4. If evidence points to a recent release, follow your team's rollback
procedure. Otherwise, use the mitigation for the affected dependency
or operation and involve its owner.
5. After mitigation, confirm that latency returns to its normal range
and the alert resolves. Record the cause and update this procedure.Replace the general recovery steps with your app's actual procedures before publishing. A rising latency chart alone is not enough evidence to choose a rollback.
A panel block uses the same panel schema as a dashboard. Panels run against the runbook's selected time range. See the runbook spec for variables, shared panels, and multi-page procedures.
Link it to the alert
In the existing everr/api-latency.alert.yaml, add runbook under spec, keeping the rest of the rule:
spec:
runbook: api-latency
# Keep the display, query, condition, and other settings from your rule.A bare slug refers to a runbook in the same project as the alert. Both examples use the default project. The runbook must already exist or be included in the same apply directory.
Preview and publish
Stage both changes:
everr apply ./everr --previewOpen the printed link. In Runbooks, check the procedure and panels. In Alerts, check the link from the latency rule to the runbook.
When you're ready, publish the whole directory:
everr apply ./everrReview the plan and confirm. The runbook is available at /runbooks/default/api-latency and from the alert's detail page. Edit its source files whenever your investigation or recovery process changes, then preview and apply again.
Keep using your telemetry
You now have a path from instrumentation to a dashboard, an alert, and a response procedure. Use the same data for everyday debugging with everr-use-telemetry, and refine your resources as the app changes.
- Writing good alerts: choose signals and thresholds that need a response.
- Publishing resources: publish reviewed changes from your repository.
- Connect a repository: bring GitHub Actions data into Everr.