Architecture Overview

Updated for nself v1.0.12

ɳSelf provides a modern backend architecture that combines the best of cloud-native technologies. The Plugin System (available since v1.0.0), environment management via .environments/, and production deployment commands give you a scalable, production-ready backend infrastructure with a single CLI.

Environment Management Architecture

.environments/ Directory Structure

ɳSelf uses a structured approach to environment management:

.environments/
├── dev/
│   └── .env              # Development configuration
├── staging/
│   ├── .env              # Staging configuration
│   ├── .env.secrets      # Staging secrets (git-ignored)
│   └── server.json       # SSH connection details
└── prod/
    ├── .env              # Production configuration
    ├── .env.secrets      # Production secrets (git-ignored)
    └── server.json       # SSH connection details

Environment Commands

Manage environments with dedicated commands:

# List all environments
nself env list

# Switch environments
nself env switch dev
nself env switch staging
nself env switch prod

# Compare environments
nself env diff staging prod

Deployment Architecture

Staging vs Production Deployment

Different deployment strategies for staging and production:

Service Categories

Five Distinct Service Types

ɳSelf organizes services into five categories for clear deployment management:

• Core Services (4): PostgreSQL, Hasura, Auth, Nginx - Always deployed
• Optional Services (6): nself-admin, MinIO, Redis, Functions, Mailpit, Meilisearch - Based on *_ENABLED vars (MLflow is now a plugin)
• Monitoring Bundle (10): Prometheus, Grafana, Loki, Tempo, etc. - All or nothing
• Custom Services (CS_N): User-defined backend applications
• Frontend Apps (FRONTEND_APP_N): External apps for Nginx routing

Core Architecture Principles

Microservices Architecture

ɳSelf is built on microservices principles, where each service has a specific responsibility:

• Single Responsibility: Each service handles one domain or function
• Independent Deployment: Services can be deployed and scaled independently
• Technology Diversity: Use the best technology for each service
• Fault Isolation: Failure in one service doesn't bring down the entire system

Container-First Design

Everything in nself runs in Docker containers:

• Consistent Environments: Same container runs in dev, staging, and production
• Resource Isolation: Each service has its own resource allocation
• Easy Scaling: Scale services by adding more container instances
• Cloud Native: Ready for Kubernetes and cloud deployment

System Components

Core Infrastructure Stack

Data Flow Architecture

Request Flow

1. Client Request: Frontend app or external client makes API call
2. Nginx Proxy: Routes request to appropriate service
3. Authentication: JWT token validation and user context
4. API Processing: GraphQL query or REST endpoint processing
5. Database Query: PostgreSQL data retrieval or modification
6. Response: JSON response back to client

Background Processing Flow

1. Job Creation: API endpoint or webhook creates background job
2. Queue Storage: Job stored in Redis queue
3. Worker Processing: BullMQ worker picks up and processes job
4. External Services: Worker may call external APIs or services
5. Result Storage: Job results stored in database or cache
6. Notifications: Webhooks or real-time updates sent to clients

Service Communication Patterns

Synchronous Communication

• HTTP/REST: Direct service-to-service API calls
• GraphQL: Unified API layer through Hasura
• gRPC: High-performance RPC for Go services

Asynchronous Communication

• Message Queues: BullMQ for job processing
• Event Streams: Redis pub/sub for real-time events
• Webhooks: HTTP callbacks for external integrations

Scalability Architecture

Horizontal Scaling

Scale by adding more instances:

# Scale specific services
NESTJS_API_REPLICAS=3
BULLMQ_EMAIL_REPLICAS=2
PYTHON_ML_API_REPLICAS=4

# Auto-scaling based on metrics
CPU_SCALE_THRESHOLD=80
MEMORY_SCALE_THRESHOLD=85

Vertical Scaling

Scale by increasing resource allocation:

# Resource limits per service
POSTGRES_MEMORY_LIMIT=2048MB
POSTGRES_CPU_LIMIT=2.0

HASURA_MEMORY_LIMIT=1024MB
HASURA_CPU_LIMIT=1.0

Security Architecture

Network Security

• Container Isolation: Each service runs in isolated container
• Internal Networks: Services communicate over private Docker networks
• Reverse Proxy: Nginx handles SSL termination and routing
• Firewall Rules: Only necessary ports exposed externally

Authentication & Authorization

• JWT Tokens: Stateless authentication across services
• Role-Based Access: User roles and permissions
• API Keys: Service-to-service authentication
• Row-Level Security: Database-level access control

Development vs Production Architecture

Development Environment

• All-in-One: Single machine deployment with 100% service reliability
• Hot Reloading: Code changes trigger automatic rebuilds
• HTTPS Development: All services accessible via HTTPS with trusted certificates
• Smart Port Management: Organized port allocation prevents conflicts
• Debug Tools: MailPit (port 3101), database adminer (port 3100), log viewers
• Enhanced Security: SSL by default while maintaining easy debugging

Production Environment

• Distributed: Services can run on multiple machines with smart port allocation
• High Availability: Multiple replicas and enhanced health checks
• 100% Uptime: Proven service reliability with automatic recovery
• SSL/TLS Everywhere: Complete HTTPS with Let's Encrypt or custom certificates
• Monitoring: complete logging and metrics
• Security Hardening: Enhanced SSL, secrets management, network isolation

Database Architecture

Schema-First Design

• DBML Schema: Human-readable database design
• Migration System: Version-controlled schema changes
• Automatic GraphQL: Hasura generates API from schema
• Type Safety: Shared types across services

Data Access Patterns

• GraphQL First: Primary API through Hasura
• Direct SQL: Custom services can query directly
• Connection Pooling: Efficient database connections
• Read Replicas: Scale read operations

Deployment Architecture

Single Machine Deployment

# Deploy to a single server
nself build        # Generates docker-compose.yml from .env
nself start        # Boots all services in dependency order

# Or for production
nself deploy prod

Multi-Node Deployment

# Kubernetes deployment example
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nself-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nself-api
  template:
    metadata:
      labels:
        app: nself-api
    spec:
      containers:
      - name: api
        image: my-registry/nself-api:latest
        resources:
          limits:
            memory: "512Mi"
            cpu: "500m"

Monitoring Architecture

Health Monitoring

• Service Health Checks: HTTP endpoints for each service
• Database Monitoring: Connection counts, query performance
• Resource Monitoring: CPU, memory, disk usage
• Application Metrics: Custom business metrics

Logging Architecture

• Centralized Logging: All logs aggregated in one place
• Structured Logs: JSON format for easy parsing
• Log Rotation: Automatic cleanup of old logs
• Search & Analytics: Full-text search capabilities

Disaster Recovery Architecture

Backup Strategy

• Database Backups: Automated PostgreSQL dumps
• File Backups: MinIO storage backup
• Configuration Backups: Environment and config files
• Off-site Storage: Cloud storage for backups

Recovery Procedures

• Point-in-Time Recovery: Restore to specific timestamp
• Rolling Updates: Zero-downtime deployments
• Rollback Capability: Quick revert to previous version
• Multi-Region Support: Geographic distribution

Performance Architecture

Caching Strategy

• Redis Cache: Application-level caching
• GraphQL Caching: Hasura query result caching
• CDN Integration: Static asset caching
• Database Query Cache: PostgreSQL query optimization

Performance Optimization

• Connection Pooling: Efficient database connections
• Lazy Loading: Load data only when needed
• Async Processing: Non-blocking operations
• Resource Limits: Prevent resource exhaustion

Plugin Architecture

Plugin System

ɳSelf ships 112 plugins (25 free + 87 paid). Install, update, and manage them from the central registry:

# Browse available plugins
nself plugin list

# Install plugins
nself plugin install ai
nself plugin install notify
nself plugin install cron

# Check status
nself plugin status

# Remove a plugin
nself plugin uninstall ai

Plugin Directory Structure

~/.nself/plugins/
├── _shared/              # Shared utilities
├── ai/
│   ├── plugin.json       # Plugin manifest
│   ├── schema/tables.sql # Database tables
│   └── go/               # Go service source
└── notify/
    └── ...

Plugin Bundles

• nClaw Bundle ($0.99/mo): ai, claw, mux, voice, browser, google, notify, cron, and more
• nTV Bundle ($0.99/mo): media-processing, streaming, epg, recording, subtitle-manager, and more
• nChat Bundle ($0.99/mo): chat, livekit, recording, moderation, bots, realtime
• ɳSelf+ ($3.99/mo): All 5 bundles + support

Integration Points

• Webhooks: Standardized endpoints with HMAC-SHA256 verification
• API Gateways: Custom routing and transformation
• Message Queues: Event-driven architecture
• Custom Functions: Serverless compute

Best Practices

Service Design

• Stateless Services: Store state in databases, not containers
• Idempotent Operations: Safe to retry operations
• Circuit Breakers: Fail fast when dependencies are down
• Graceful Degradation: Continue operating with reduced functionality

Data Management

• Database Per Service: Each service owns its data
• Event Sourcing: Track all changes as events
• CQRS Pattern: Separate read and write models
• Data Consistency: Use transactions and constraints

Next Steps

Now that you understand the architecture:

• PostgreSQL Database - Learn about the database layer
• Hasura GraphQL - Understand the API layer
• Production Deployment - Deploy to production
• Monitoring & Logs - Set up observability

Understanding the architecture helps you make informed decisions about scaling, security, and extending your nself deployment.