Distributed Tracing

Distributed Tracing

ɳSelf ships distributed tracing via OpenTelemetry. Every plugin service emits spans to Grafana Tempo with zero manual configuration. A single request from the Admin UI through Hasura to a plugin service produces one connected trace, visible in Grafana within 60 seconds of starting the monitoring stack.

Supabase and Nhost offer logs only. ɳSelf gives you production-grade tracing out of the box.

Prerequisites

The monitoring plugin must be installed:

nself plugin install monitoring
nself build && nself restart

This starts Tempo (port 3200) with an OTLP/gRPC receiver on port 4317. Grafana already has Tempo configured as a datasource.

How it works

Admin UI / client
    | traceparent header (W3C Trace Context)
    v
Nginx, logs trace ID to access log
    |
    v Hasura, forwards traceparent to Remote Schema services
    |
    v Plugin service (Go)
    |  OTel SDK: InitTracing("ai", "1.2.0")
    |  Exporter: OTLP gRPC → Tempo (localhost:4317)
    |  HTTP server middleware: extracts + propagates traceparent
    |
    v Tempo, stores spans
    |
    v Grafana: TraceQL queries, trace-to-log correlation

Hasura forwards the traceparent header to Remote Schema services when HASURA_GRAPHQL_ENABLE_TRACING=true (set by default in the nself env template). The plugin HTTP server reads this header via the OTel middleware, so every plugin span joins the originating trace automatically.

Plugin SDK integration

All Go plugin services call InitTracing in main():

import tracing "github.com/nself-org/nself-shared-tracing"

func main() {
    shutdown, err := tracing.InitTracing("ai", "1.2.0")
    if err != nil {
        log.Fatal(err)
    }
    defer shutdown()

    // ... start HTTP server
}

The plugin SDK server (plugin-sdk-go/server) automatically mounts W3C trace context middleware on every route, so incoming traceparent headers are extracted and downstream calls propagate the trace ID without any additional code.

For outbound HTTP calls (for example, the ai plugin calling an LLM API), wrap the client transport:

import tracing "github.com/nself-org/nself-shared-tracing"

client := &http.Client{
    Transport: tracing.HTTPClientTransport(nil),
}

Env vars

All env vars are optional. The defaults work with the monitoring plugin out of the box.

Variable	Default	Notes
OTEL_EXPORTER_OTLP_ENDPOINT	http://localhost:4317	Tempo OTLP/gRPC endpoint
OTEL_SERVICE_NAME	auto (plugin name)	Override when running multiple instances
OTEL_TRACES_SAMPLER	parentbased_traceidratio	Respects incoming sampling decisions
OTEL_TRACES_SAMPLER_ARG	1.0	1.0 = 100% (dev), 0.1 = 10% (prod)
OTEL_LOG_LEVEL	warn	OTel SDK internal log verbosity

To reduce Tempo ingest volume in production without code changes:

# In .env.prod or .env.secrets
OTEL_TRACES_SAMPLER_ARG=0.1

Restart the affected plugin services. No recompile needed.

Viewing traces in Grafana

1. Open Grafana (default: http://localhost:3000).
2. Go to Explore.
3. Select the Tempo datasource.
4. Use TraceQL to query: {resource.service.name="ai"} or search by trace ID.

For trace-to-log correlation, Loki is already linked to Tempo in the monitoring stack. Click any span to jump directly to the associated log lines.

Sampling

The default sampler is parentbased_traceidratio:

• If an incoming request already carries a traceparent with the sampled flag set, the span is always recorded.
• For new root traces (no incoming traceparent), the sampler uses OTEL_TRACES_SAMPLER_ARG as the probability (default 1.0 = all requests).

At 0.1, request volume to Tempo drops approximately 10x. Latency-sensitive outliers are still captured because sampled requests propagate their sampling decision downstream.

Troubleshooting

No traces appearing in Grafana:

1. Verify Tempo is running: nself status should show tempo: running.
2. Check the plugin service log for otel tracer initialised at startup.
3. Confirm OTEL_EXPORTER_OTLP_ENDPOINT resolves. Default is localhost:4317: correct when the plugin runs on the same host as the monitoring stack.

Spans not connected across services:

The traceparent header must flow through every hop. Check that HASURA_GRAPHQL_ENABLE_TRACING=true is set and that any custom HTTP clients use tracing.HTTPClientTransport.

High Tempo disk usage:

Lower the sampling ratio: OTEL_TRACES_SAMPLER_ARG=0.01 (1%). The monitoring plugin sets block_retention: 48h by default; reduce to 24h in monitoring/configs/tempo/tempo.yaml if disk is constrained.