SRE Runbooks

SRE Runbooks

ɳSelf ships 7 pre-written runbooks for the most common failure scenarios. When a Grafana alert fires, Alertmanager can automatically invoke the matching runbook via the claw-ops agent.

Built-in runbooks

ID	Trigger Alert	Severity	Auto-executable
high-error-rate	HighErrorRate	critical	Yes
postgres-connection-exhaustion	PostgresConnectionsHigh	critical	No (requires_confirmation)
redis-oom	RedisMemoryHigh	critical	No
webhook-dlq-overflow	WebhookDLQHigh	warning	No
loki-disk-full	LokiDiskFull	warning	No
certificate-expiry	CertificateExpiringSoon	warning	Yes
plugin-crash-loop	PluginCrashLoop	critical	Yes

CLI commands

# List all installed runbooks
nself runbook list

# Execute a runbook (production)
nself runbook run high-error-rate

# Dry run - prints steps without executing
nself runbook test high-error-rate

# Pass alert context variables
nself runbook run plugin-crash-loop --var service=ai --var plugin_port=8100

Runbook format

Runbooks are Markdown files with a YAML front matter header. Custom runbooks go in monitoring/runbooks/ in your ɳSelf project:

---
id: my-custom-runbook
title: My Custom Runbook
trigger:
  alert: MyCustomAlert
  severity: warning
steps:
  - action: check_logs
    params:
      service: my-service
      last: 10m
    requires_confirmation: false
  - action: restart_service
    params:
      service: my-service
    requires_confirmation: true
---

# My Custom Runbook

Describe what this runbook does...

Available actions

Action	Parameters	Auto-executes
check_logs	service, last, level	Always
restart_service	service	Unless requires_confirmation: true
notify_slack	channel, message	Always (requires SLACK_WEBHOOK_URL)
run_command	command	Always
run_query	query (SQL)	Always
escalate	message	Posts to Slack, requires_confirmation=true

claw-ops auto-execution

When Alertmanager fires an alert, it sends a webhook to the claw-ops endpoint. The claw-ops agent matches the alert name to a runbook and executes all steps not marked requires_confirmation: true.

Enable by setting nself in .env.prod:

NSELF_CLAW_OPS_WEBHOOK_URL=http://127.0.0.1:8001/runbook/trigger

Steps with requires_confirmation: true always escalate to Slack for human review.