Production Setup

Updated for nself v0.4.8

Deploy nself v0.4.8 to production with the new environment management system, nself prod commands, SSH-based deployment, security hardening, and monitoring.

New in v0.4.8: Production Commands

  • * nself prod init: Initialize production configuration
  • * nself prod check: Run comprehensive security audit
  • * nself prod secrets: Generate and manage production secrets
  • * nself prod ssl: SSL certificate management
  • * nself prod harden: Apply all security hardening measures
  • * nself deploy: SSH-based deployment with health checks

Production Preparation

Initialize Production Environment

# Initialize production for your domain
nself prod init yourdomain.com --email admin@yourdomain.com

# This sets up:
# - ENV=production
# - Disables debug mode
# - Enables SSL with Let's Encrypt
# - Disables Hasura dev mode and console
# - Creates docker-compose.prod.yml

Generate Production Secrets

# Generate all production secrets
nself prod secrets generate

# Creates .environments/prod/.env.secrets with:
# - POSTGRES_PASSWORD (32 chars)
# - HASURA_GRAPHQL_ADMIN_SECRET (32 chars)
# - JWT_SECRET (64 chars)
# - COOKIE_SECRET (32 chars)
# - MINIO_ROOT_PASSWORD (32 chars)
# - REDIS_PASSWORD (32 chars)
# - GRAFANA_ADMIN_PASSWORD (32 chars)

# Validate secrets
nself prod secrets validate

Environment Directory Structure

.environments/
├── dev/
│   └── .env              # Development config
├── staging/
│   ├── .env              # Staging config
│   ├── .env.secrets      # Staging secrets (git-ignored)
│   └── server.json       # SSH connection details
└── prod/
    ├── .env              # Production config
    ├── .env.secrets      # Production secrets (git-ignored)
    └── server.json       # SSH connection details

Review Production Configuration

Edit .environments/prod/.env with your production settings:

# Environment
ENV=prod
BASE_DOMAIN=yourdomain.com

# Security
DEBUG=false
HASURA_GRAPHQL_DEV_MODE=false
HASURA_GRAPHQL_ENABLE_CONSOLE=false
NSELF_ADMIN_ENABLED=false  # DISABLED in production

# Performance
POSTGRES_MAX_CONNECTIONS=100
REDIS_MAXMEMORY=512mb
HASURA_GRAPHQL_POOL_SIZE=10

# Monitoring
LOG_LEVEL=info
MONITORING_ENABLED=true

Server Requirements

Minimum Production Requirements

Recommended Production Specs

Server Setup

Initial Server Configuration

# Update system
sudo apt update && sudo apt upgrade -y

# Install Docker and Docker Compose
curl -fsSL https://get.docker.com | bash
sudo usermod -aG docker $USER

# Install nself
curl -sSL https://install.nself.org | bash

# Configure firewall
sudo ufw allow 22/tcp   # SSH
sudo ufw allow 80/tcp   # HTTP
sudo ufw allow 443/tcp  # HTTPS
sudo ufw --force enable

# Set up swap (if not already configured)
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Security Hardening

# Disable root login
sudo sed -i 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sudo systemctl restart sshd

# Configure automatic security updates
sudo apt install unattended-upgrades
sudo dpkg-reconfigure unattended-upgrades

# Install fail2ban
sudo apt install fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

# Configure log rotation
sudo logrotate -d /etc/logrotate.conf

SSL/TLS Configuration v0.4.8

nself v0.4.8 provides comprehensive SSL management via nself prod ssl commands.

SSL Certificate Management

# Check current SSL certificate status
nself prod ssl status
# Shows subject, issuer, validity dates, and days until expiry

# Request a Let's Encrypt certificate
nself prod ssl request yourdomain.com --email admin@yourdomain.com

# Use staging for testing (avoids rate limits)
nself prod ssl request yourdomain.com --staging

# Renew SSL certificates
nself prod ssl renew
nself prod ssl renew --force

# Generate self-signed certificate (for development)
nself prod ssl self-signed yourdomain.com

# Verify certificate chain matches private key
nself prod ssl verify

Custom SSL Certificates

# If using custom certificates:
SSL_ENABLED=true
SSL_PROVIDER=custom

# Place certificates in ssl/ directory
mkdir -p ssl/
cp your-cert.crt ssl/cert.pem
cp your-key.key ssl/key.pem

# Verify certificate installation
nself prod ssl verify

Database Setup

Managed Database (Recommended)

Use a managed PostgreSQL service for production:

AWS RDS Configuration

# Update .env.prod-template:
POSTGRES_HOST=your-rds-endpoint.amazonaws.com
POSTGRES_PORT=5432
POSTGRES_DB=yourapp_production
POSTGRES_USER=yourapp_user
POSTGRES_PASSWORD=secure_password
POSTGRES_SSL_MODE=require

# Connection pooling
POSTGRES_MAX_CONNECTIONS=100
POSTGRES_POOL_SIZE=10

Google Cloud SQL Configuration

POSTGRES_HOST=your-cloud-sql-ip
POSTGRES_PORT=5432
POSTGRES_DB=yourapp_production
POSTGRES_USER=yourapp_user
POSTGRES_PASSWORD=secure_password
POSTGRES_SSL_MODE=require

# Cloud SQL Proxy (if using private IP)
POSTGRES_PROXY_ENABLED=true

Self-Hosted Database

If self-hosting PostgreSQL, use these production settings:

# PostgreSQL production configuration
POSTGRES_SHARED_BUFFERS=2GB
POSTGRES_EFFECTIVE_CACHE_SIZE=6GB
POSTGRES_WORK_MEM=64MB
POSTGRES_MAINTENANCE_WORK_MEM=512MB
POSTGRES_MAX_CONNECTIONS=200
POSTGRES_WAL_BUFFERS=16MB
POSTGRES_CHECKPOINT_COMPLETION_TARGET=0.9

# Enable connection pooling
PGBOUNCER_ENABLED=true
PGBOUNCER_POOL_SIZE=25

Deployment Process

Configure Server Connection

# Create .environments/prod/server.json
{
  "name": "production",
  "type": "prod",
  "host": "your-server.example.com",
  "port": 22,
  "user": "deploy",
  "key": "~/.ssh/nself_production",
  "deploy_path": "/opt/nself"
}

SSH-Based Deployment

# Check SSH access to all environments
nself deploy check-access

# Preview deployment without executing
nself deploy prod --dry-run

# Deploy to production (backend only by default)
nself deploy prod

# Skip health checks after deployment
nself deploy prod --skip-health

# Force deployment without confirmation
nself deploy prod --force

# Check deployment health
nself deploy health prod

Deployment Workflow

# What happens during nself deploy prod:
# 1. Local Build - Runs nself build
# 2. Create Directories - Creates remote directory structure
# 3. Sync Project Files - Transfers docker-compose.yml, nginx/, postgres/, services/
# 4. Sync Environment - Transfers .env and .env.secrets
# 5. Pull Images - Runs docker compose pull on server
# 6. Start Services - Runs docker compose up -d --force-recreate
# 7. Health Checks - Verifies services are running

Deployment Verification

# Check deployment status
nself deploy status

# View deployment logs
nself deploy logs

# Check all services are running
nself status

# Test connectivity
curl -k https://yourdomain.com/healthz
curl -k https://yourdomain.com/v1/graphql

# Check SSL certificate
openssl s_client -connect yourdomain.com:443 -servername yourdomain.com

Monitoring and Observability

Enable Production Monitoring

# In .env production:
MONITORING_ENABLED=true
METRICS_ENABLED=true
LOG_LEVEL=info

# Optional monitoring services
GRAFANA_ENABLED=true
PROMETHEUS_ENABLED=true
ALERTMANAGER_ENABLED=true

# External monitoring integrations
DATADOG_API_KEY=your-datadog-key
NEWRELIC_LICENSE_KEY=your-newrelic-key

Health Checks and Alerts

# Set up health check cron job
crontab -e

# Add this line:
*/5 * * * * /usr/local/bin/nself doctor > /tmp/nself-health.log 2>&1 || echo "nself health check failed" | mail -s "Production Alert" admin@yourdomain.com

# Configure Grafana alerts
# CPU usage > 80%
# Memory usage > 85%  
# Disk usage > 90%
# Database connections > 80% of max
# API error rate > 5%

Backup Strategy

Automated Database Backups

# Daily backup script
#!/bin/bash
# /usr/local/bin/backup-database.sh

BACKUP_DIR="/backup/database"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=30

# Create backup directory
mkdir -p $BACKUP_DIR

# Create database backup
nself db backup --name "daily_$DATE"

# Compress and move to backup location
tar -czf "$BACKUP_DIR/backup_$DATE.tar.gz" bin/dbsyncs/

# Clean old backups
find $BACKUP_DIR -name "backup_*.tar.gz" -mtime +$RETENTION_DAYS -delete

# Log backup completion
echo "$(date): Database backup completed - backup_$DATE.tar.gz" >> /var/log/nself-backup.log

Backup Cron Jobs

# Add to crontab (crontab -e):

# Daily database backup at 2 AM
0 2 * * * /usr/local/bin/backup-database.sh

# Weekly full system backup at 3 AM Sunday
0 3 * * 0 /usr/local/bin/backup-full-system.sh

# Monthly backup verification
0 4 1 * * /usr/local/bin/verify-backups.sh

Security Configuration

Security Audit

# Run comprehensive security audit
nself prod check
nself prod audit --verbose

# Security checks performed:
# - Environment settings (DEBUG, LOG_LEVEL, etc.)
# - Secrets strength and configuration
# - SSL certificate validity and expiry
# - Docker security settings
# - Network security (HSTS, XSS protection, etc.)
# - File permissions

Firewall Configuration

# Check firewall status
nself prod firewall status

# Configure recommended firewall rules
nself prod firewall configure --dry-run
nself prod firewall configure

# Default rules applied:
# - Deny incoming traffic by default
# - Allow outgoing traffic
# - Allow SSH (port 22) with rate limiting
# - Allow HTTP (port 80)
# - Allow HTTPS (port 443)

# Allow additional ports
nself prod firewall allow 8080
nself prod firewall allow 53 udp

# Show security recommendations
nself prod firewall recommendations

Apply All Security Hardening

# Apply all security hardening measures
nself prod harden

# What it does:
# 1. Generate secrets (if missing)
# 2. Apply production environment settings
# 3. Fix SSL key permissions
# 4. Check firewall status
# 5. Fix sensitive file permissions

# Preview changes without applying
nself prod harden --dry-run

# Skip firewall configuration
nself prod harden --skip-firewall

Application Security

# Security headers and CORS
CORS_ORIGINS=https://yourdomain.com,https://www.yourdomain.com
SECURITY_HEADERS_ENABLED=true
HSTS_ENABLED=true
XSS_PROTECTION_ENABLED=true

# Rate limiting
RATE_LIMITING_ENABLED=true
RATE_LIMIT_MAX_REQUESTS=100
RATE_LIMIT_WINDOW=900  # 15 minutes

# DDoS protection
DDOS_PROTECTION_ENABLED=true
DDOS_MAX_CONNECTIONS=1000

Performance Optimization

Resource Allocation

# Optimize service resource limits
POSTGRES_MEMORY_LIMIT=4GB
HASURA_MEMORY_LIMIT=2GB
REDIS_MEMORY_LIMIT=1GB
NGINX_MEMORY_LIMIT=512MB

# CPU limits
POSTGRES_CPU_LIMIT=2.0
HASURA_CPU_LIMIT=1.0
REDIS_CPU_LIMIT=0.5

Caching Configuration

# Redis cache settings
REDIS_MAXMEMORY=1gb
REDIS_MAXMEMORY_POLICY=allkeys-lru
REDIS_CACHE_TTL=3600

# HTTP caching
NGINX_CACHE_ENABLED=true
NGINX_CACHE_SIZE=1g
NGINX_CACHE_INACTIVE=24h

High Availability

Load Balancer Configuration

# Multiple server setup
SERVER_1=server1.yourdomain.com
SERVER_2=server2.yourdomain.com
LOAD_BALANCER=lb.yourdomain.com

# Health check endpoints
HEALTH_CHECK_PATH=/healthz
HEALTH_CHECK_INTERVAL=30s
HEALTH_CHECK_TIMEOUT=10s

Database Replication

# Master-slave PostgreSQL setup
POSTGRES_REPLICATION_ENABLED=true
POSTGRES_MASTER_HOST=db-master.yourdomain.com
POSTGRES_REPLICA_HOST=db-replica.yourdomain.com

# Read-only replica for queries
POSTGRES_READ_REPLICA_ENABLED=true

Maintenance and Updates

Update Strategy

# Scheduled maintenance window
# 1. Backup current state
nself backup --name "pre-update-$(date +%Y%m%d)"

# 2. Update nself CLI
nself update

# 3. Apply any new migrations
nself db migrate:up

# 4. Restart services with new configuration
nself restart

# 5. Verify everything is working
nself doctor
nself status --verbose

Zero-Downtime Updates

# For critical production systems
# 1. Set up blue-green deployment
# 2. Update standby environment
# 3. Switch traffic to updated environment  
# 4. Keep old environment as fallback

# Rolling updates for stateless services
nself update --rolling
nself restart --rolling hasura

Disaster Recovery

Recovery Procedures

# Complete system recovery
# 1. Provision new server
# 2. Install nself and dependencies  
# 3. Restore configuration files
# 4. Restore database from backup
nself db restore /path/to/backup.sql
# 5. Start services
nself up
# 6. Update DNS if needed

Recovery Testing

Regularly test your disaster recovery procedures:

Next Steps