Troubleshooting

Diagnose and fix common issues. Start with nself doctor — it catches most problems automatically.

# 1. Run the doctor — diagnoses most issues automatically
nself doctor

# 2. Check service health
nself health

# 3. See recent logs (last 5 minutes)
nself logs --since 5m

# 4. Fix detected issues automatically
nself doctor --fix

# 5. If all else fails — nuclear reset (dev only, destroys data)
nself stop && nself db reset && nself start

Diagnostic Commands

nself doctor

The doctor command checks everything and explains what's wrong. Run it first, every time.

# Full diagnostic scan
nself doctor

# Output (example):
#   Checking Docker...                  OK
#   Checking DNS resolution...          OK (*.local.nself.org → 127.0.0.1)
#   Checking SSL certificates...        WARN (cert expires in 3 days)
#   Checking service health...          FAIL (hasura: not running)
#   Checking database connectivity...   FAIL (cannot connect)
#   Checking port availability...       OK
#   Checking disk space...              OK (42 GB free)
#   Checking env config...              OK
#
#   2 errors found. Run: nself doctor --fix

# Auto-fix detected issues
nself doctor --fix

# Deep security + config check
nself doctor --deep

# Fix a specific category
nself doctor --fix ports
nself doctor --fix ssl
nself doctor --fix dns

Service health and logs

# Health summary for all services
nself health

# Verbose — shows container status, uptime, restart count
nself health --verbose

# Specific service
nself health hasura

# Tail all service logs
nself logs --follow

# Logs for one service
nself logs hasura --follow
nself logs postgres --follow
nself logs nginx --follow

# Last N lines
nself logs --tail 200

# Filter by time window
nself logs --since 30m
nself logs --since "2026-05-07 14:00"

# Filter by log level
nself logs --level error
nself logs --level warn

# Open log viewer (Grafana Loki — dev only)
nself logs --ui

Service status and inspection

# Status of all services
nself status

# Show resource usage (CPU, RAM, disk)
nself status --resources

# Show which ports each service uses
nself status --ports

# Check Nginx routing table
nself service exec nginx nginx -T | grep server_name

# Inspect a service's running config
nself service exec hasura env | grep HASURA

# Run a command inside a service container
nself service exec postgres psql -U postgres -c "SELECT version();"

# Check which cert Nginx is serving
openssl s_client -connect api.local.nself.org:443 -servername api.local.nself.org 2>/dev/null   | openssl x509 -noout -subject -dates

Services Won't Start

One service keeps crashing

All services have restart: unless-stopped. If a service keeps crashing, the logs will tell you why.

# See why it crashed
nself logs hasura --tail 100

# Try restarting it manually
nself service restart hasura

# Check for common causes:

# PostgreSQL — disk full
df -h                           # check disk
nself service exec postgres psql -U postgres -c "SELECT pg_size_pretty(pg_database_size('postgres'));"
nself db vacuum                 # run VACUUM to reclaim space

# Hasura — metadata inconsistency
nself db hasura reload          # reload metadata without restart
nself service restart hasura    # or hard restart

# Auth — JWT config changed
# (auth caches the JWT key at startup — must restart on key rotation)
nself service restart auth

# Redis — out of memory
nself service exec redis redis-cli info memory | grep used_memory_human
# If near limit, increase REDIS_MAX_MEMORY in .env, then:
nself build && nself service restart redis

All services fail to start

# Check Docker is running
docker info

# Check for port conflicts
nself doctor --fix ports

# Check .env for syntax errors
nself env validate

# Rebuild the Docker Compose config and try again
nself build
nself start

# If still failing, check for stale containers/volumes
docker ps -a | grep nself
docker volume ls | grep nself

Docker Issues

Docker not running

# macOS
open -a Docker                  # launch Docker Desktop
# Wait ~30 seconds for it to start, then:
nself start

# Linux
sudo systemctl start docker
sudo systemctl enable docker    # start on boot

# Verify Docker is up
docker info

Port conflicts

# Auto-detect and fix port conflicts
nself doctor --fix ports

# Manual: find what's using a port
lsof -i :5432                   # PostgreSQL default
lsof -i :8080                   # Hasura default
lsof -i :443                    # Nginx HTTPS

# Kill a blocking process
kill -9 $(lsof -t -i :5432)

# Check which ports nself is using
nself status --ports

Out of disk space

# Check disk usage
df -h
docker system df

# Clean up Docker build cache and dangling images
docker system prune

# More aggressive: remove all unused images (not just dangling)
docker system prune -a

# nSelf never deletes volumes automatically — your data is safe
# Check volume sizes
docker volume ls | grep nself
docker system df -v | grep nself

Database Issues

PostgreSQL won't start or crashes

# Check logs
nself logs postgres --tail 50

# Common errors and fixes:

# "data directory has wrong ownership"
# → volume was created by a different user
docker volume rm nself_postgres_data   # ⚠️ destroys data — backup first!
nself start

# "database system identifier differs"
# → volume from a different Postgres version
nself db shell                         # try connecting
nself logs postgres | grep "FATAL"

# "out of disk space"
nself db vacuum                        # reclaim dead tuple space

# "too many connections"
# → PgBouncer is needed
# In .environments/prod/.env:
#   PGBOUNCER_ENABLED=true
#   PGBOUNCER_MAX_CLIENT_CONN=200
nself build && nself start

Hasura metadata errors

# Check Hasura logs
nself logs hasura --tail 50 --level error

# Metadata inconsistency — most common after manual DB changes
nself db hasura reload

# Force metadata drop and re-apply from tracked files
nself db hasura apply

# Open Hasura console to inspect
nself db console

# Check migration status
nself db migration status

# Apply pending migrations
nself db migrate

Migration failures

# Check migration status
nself db migration status

# Show which migration failed
nself db migration status --verbose

# Roll back last migration
nself db migrate rollback

# Apply a specific migration file
nself db migrate --file migrations/1234567890_add_users.sql

# Reset database completely (dev only — destroys all data)
nself db reset

SSL / Certificate Issues

Browser shows certificate warning (dev)

# Re-run trust setup
nself trust remove
nself trust

# Verify trust status
nself trust status

# On macOS — if Chrome still shows warning after nself trust
# Chrome has its own trust store; restart Chrome after running nself trust
# Or open: chrome://settings/certificates → Authorities → import ~/.local/share/mkcert/rootCA.pem

# On Linux
sudo update-ca-certificates

Let's Encrypt certificate not issuing (production)

# Check SSL status
nself ssl status --domain yourdomain.com

# Common causes:

# 1. Port 80 not open to the internet (HTTP-01 challenge needs it)
curl http://yourdomain.com/.well-known/acme-challenge/test
# If this fails → open port 80 in your firewall/Hetzner firewall rules

# 2. DNS not propagated yet
dig A yourdomain.com +short      # should return your VPS IP
nself prod check --item dns --domain yourdomain.com

# 3. Rate limit hit (5 certs per domain per week)
# Wait a week or use the staging ACME endpoint for testing
nself ssl request --domain yourdomain.com --staging

# 4. Wildcard cert — needs DNS-01 challenge
nself ssl request --domain "*.yourdomain.com" --challenge dns --dns-provider cloudflare

Certificate expired

# Check expiry
nself ssl status
openssl s_client -connect yourdomain.com:443 -servername yourdomain.com 2>/dev/null   | openssl x509 -noout -dates

# Force renewal
nself ssl renew --domain yourdomain.com

# Check that auto-renewal cron is running (runs daily at 03:00)
nself ssl status --renewal-cron

DNS Issues

Domain not resolving

# Check DNS propagation
nself prod check --item dns --domain yourdomain.com

# Manual checks
dig A yourdomain.com +short
dig A api.yourdomain.com +short
dig A auth.yourdomain.com +short

# Should all return your VPS IP
# If not — add the missing A records in your DNS provider

# Check from multiple global locations
curl https://dnschecker.org/api/dns/yourdomain.com

# For local dev — verify wildcard DNS
dig A api.local.nself.org +short    # should return 127.0.0.1

Custom local domain not working

# Using custom local domain (instead of *.local.nself.org)
# You must add entries to /etc/hosts manually

sudo nano /etc/hosts
# Add:
127.0.0.1  local.mycompany.com
127.0.0.1  api.local.mycompany.com
127.0.0.1  auth.local.mycompany.com
127.0.0.1  storage.local.mycompany.com

# Or use dnsmasq for wildcard routing:
# /etc/dnsmasq.conf:
#   address=/local.mycompany.com/127.0.0.1

Authentication Issues

Users can't log in

# Check auth service logs
nself logs auth --tail 50 --level error

# Test auth endpoint directly
curl -X POST https://auth.yourdomain.com/v1/user/email/signin   -H "Content-Type: application/json"   -d '{"email":"test@example.com","password":"password"}'

# Common causes:

# 1. JWT secret mismatch (auth and Hasura must share the same key)
nself service exec auth env | grep JWT_SECRET
nself service exec hasura env | grep JWT_SECRET
# If different — update to match and restart both services

# 2. CORS blocking (frontend domain not allowlisted)
# In .env:
#   AUTH_CLIENT_URL=https://yourapp.com

# 3. Rate limit triggered (10 req/min per IP on auth endpoints)
nself logs nginx | grep "429" | grep auth

JWT errors in Hasura

# Check JWT config
nself logs hasura | grep -i jwt

# Verify the Hasura JWT secret matches auth
nself service exec hasura env | grep HASURA_GRAPHQL_JWT_SECRET

# After rotating JWT keys — restart auth first, then Hasura
nself service restart auth
nself service restart hasura

Nginx / Routing Issues

502 Bad Gateway

# 502 = Nginx reached the service but it's not responding

# Check which service is down
nself health

# Check Nginx logs to see which upstream is failing
nself logs nginx | grep "502|upstream"

# Check if the service is actually running
nself status

# Restart the failing service
nself service restart hasura   # or whichever service is down

404 on API requests

# Check Nginx routing table
nself service exec nginx nginx -T | grep "location|server_name"

# Verify the subdomain points to your server
dig A api.yourdomain.com +short

# Test the endpoint directly on the container
nself service exec hasura curl http://localhost:8080/v1/version

# Check if Nginx config is up to date
# (it gets regenerated by nself build)
nself build
nself service reload nginx

WebSocket connections failing (subscriptions broken)

# Hasura subscriptions need a direct connection — Cloudflare proxy breaks them
# Make sure api.yourdomain.com is set to "DNS only" (grey cloud), not proxied

# Test WebSocket
wscat -c wss://api.yourdomain.com/v1/graphql

# Check Nginx WebSocket upgrade headers
nself service exec nginx nginx -T | grep -A 5 "upgrade"

Performance Degradation

Slow GraphQL queries

# Find slow queries (top 10 by total time)
nself db slow-queries --top 10

# Explain a specific query
nself db explain "SELECT * FROM np_items WHERE user_id = $1 ORDER BY created_at DESC LIMIT 20"

# Check if PgBouncer is enabled (needed above ~50 concurrent users)
nself service exec postgres psql -U postgres -c "SHOW max_connections;"

# Check connection pool usage
nself db pool-stats

High memory or CPU

# Live resource usage
nself perf status

# Identify which service is consuming resources
nself perf watch   # refreshes every 2 seconds

# Check Redis memory
nself service exec redis redis-cli info memory | grep used_memory_human

# Run a quick benchmark to establish baseline
nself perf bench --requests 100 --concurrency 5   --endpoint https://api.yourdomain.com/v1/graphql

# If Hasura is the bottleneck — scale horizontally
# In .env:
#   HASURA_REPLICAS=2
nself build && nself start

Backup and Restore Issues

Backup failing

# Check backup agent status
nself backup status

# Run a manual backup and watch for errors
nself backup create

# Common causes:

# 1. Disk full
df -h
nself backup prune              # remove old backups

# 2. Remote storage credentials invalid
# Check in .env:
#   BACKUP_S3_ACCESS_KEY, BACKUP_S3_SECRET_KEY
nself backup create             # re-run after fixing creds

# 3. Backup agent not running
nself service restart backup-agent

Restore not working

# List available backups
nself backup list

# Verify a backup before restoring
nself backup verify <backup-id>

# Restore from remote if local copy is missing
nself backup restore <backup-id> --from-remote

# Check backup encryption key is set
# BACKUP_ENCRYPT_KEY must be the same key that was used when the backup was created

Plugin Issues

Plugin fails to install

# Check license key is set
nself license status

# Set or update license key
nself license set nself_pro_xxx...

# Try installing again
nself plugin install ai

# Check plugin service logs
nself logs ai --tail 50

# Verify plugin is listed
nself plugin list

Plugin installed but not working

# Check plugin health
nself health ai

# Check plugin logs
nself logs ai --follow

# Rebuild to regenerate plugin config
nself build
nself service restart ai

# Check plugin-specific env vars are set
# Each plugin has required env vars — see plugin docs
nself env list | grep AI_

Email Not Sending

# Check mail service logs
nself logs mail --tail 50

# Test SMTP connection
nself service exec mail curl -v smtp://smtp.yourmailprovider.com:587

# Check mail env vars
nself env list | grep SMTP
nself env list | grep MAIL

# In dev — check Mailpit (catches all outgoing mail)
# Open browser: https://mail.local.nself.org
# Or:
nself service exec mail mailpit status

# Common causes:
# - Missing SMTP_HOST / SMTP_USER / SMTP_PASS in .env
# - Port 25 blocked by VPS provider (common — use 587 or 465)
# - From address not verified with email provider

Common Error Messages

Collecting Debug Information

When opening a GitHub issue or asking for help, include the output of these commands:

# Full system info + service status
nself doctor > debug-report.txt
nself health --verbose >> debug-report.txt
nself status --resources >> debug-report.txt

# Recent error logs
nself logs --since 30m --level error >> debug-report.txt

# Version info
nself version >> debug-report.txt

# Attach debug-report.txt to your GitHub issue

Best Practices

• Run nself doctor before debugging manually — it finds 90% of issues instantly.
• Always check nself logs <service> --tail 50 for the failing service before restarting.
• In production, create a backup before attempting any fix: nself backup create --label pre-fix.
• Use nself service restart <service> instead of restarting everything — other services stay up.
• Never hand-edit generated files (docker-compose.yml, nginx/sites/) — run nself build instead.
• Add Grafana alerts so issues surface before users report them: nself monitor open grafana.
• Enable WAL archiving for Postgres to reduce data loss risk: BACKUP_WAL_ENABLED=true.

Related

• Production deployment guide
• Disaster recovery runbook
• Backup and restore
• SSL/TLS configuration
• Performance tuning
• Local development