Log Aggregation with Loki

Log Aggregation with Loki

ɳSelf ships Loki as part of the monitoring plugin bundle. Logs from every plugin service flow through Promtail into Loki, viewable in Grafana Explore.

What’s included

• 30-day log retention (auto-purged via compactor)
• Structured JSON log format with level, trace_id, and plugin labels
• Trace ID linking: clicking a trace_id in Loki Explore jumps to Tempo
• Automatic LogQL alert rule for error rate spikes

Enabling the monitoring bundle

nself plugin install monitoring
nself build && nself start

Logs are visible in Grafana at http://localhost:3000 under Explore → Loki within 60 seconds.

Structured logging in plugins

All ɳSelf plugins use log/slog with JSON output. The standard fields are:

Field	Description
ts	Timestamp (RFC3339Nano)
level	Log level: debug, info, warn, error
service	Plugin name, e.g. ai, mux
trace_id	OpenTelemetry trace ID (auto-injected)
plugin	Same as service, used as Loki label
caller	File:line for quick navigation

Log level configuration

Set LOG_LEVEL in your .env.prod:

LOG_LEVEL=info   # default: info in prod, debug in dev
LOG_FORMAT=json  # always json in production

Querying logs in Grafana

Open Grafana → Explore → select Loki datasource.

# All errors from the AI plugin in the last hour
{nself_plugin="ai", nself_level="error"} | json

# Filter by trace ID (link from Tempo)
{job="containerlogs"} |= "trace_id=abc123"

# Error rate over 5 minutes
sum(rate({nself_level="error"}[5m])) by (nself_plugin)

Retention configuration

Default retention is 30 days. To change it, edit monitoring/loki/local-config.yaml:

limits_config:
  retention_period: 720h  # 30 days, change to e.g. 360h for 15 days

Then redeploy: nself build && nself start.

Alerts

The monitoring bundle ships a LogQL alert rule that fires when any plugin logs more than 10 errors per minute. Routed to Alertmanager → Slack (if configured).

See Runbooks for automated response.