warehouse Continuous data sync from your nSelf Postgres instance to BigQuery, Snowflake, or Redshift. Uses incremental snapshots with a configurable watermark column to avoid full-table scans, and supports optional CDC-backed streaming for near-real-time delivery. Schema changes in Postgres are detected and propagated automatically — no manual DDL in the warehouse.
Pro Plugin — ɳSelf+ membership required Install nself plugin install warehouse
nself build
nself startConnect a destination # BigQuery
WAREHOUSE_DESTINATION=bigquery
WAREHOUSE_BQ_PROJECT_ID=my-gcp-project
WAREHOUSE_BQ_DATASET=nself_analytics
WAREHOUSE_BQ_CREDENTIALS_JSON=/run/secrets/bq-service-account.json
# Snowflake
WAREHOUSE_DESTINATION=snowflake
WAREHOUSE_SNOWFLAKE_ACCOUNT=xy12345.us-east-1
WAREHOUSE_SNOWFLAKE_DATABASE=NSELF
WAREHOUSE_SNOWFLAKE_SCHEMA=PUBLIC
WAREHOUSE_SNOWFLAKE_WAREHOUSE=COMPUTE_WH
WAREHOUSE_SNOWFLAKE_USERNAME=nself_sync
WAREHOUSE_SNOWFLAKE_PASSWORD=your-password
# Redshift
WAREHOUSE_DESTINATION=redshift
WAREHOUSE_REDSHIFT_HOST=my-cluster.abc123.us-east-1.redshift.amazonaws.com
WAREHOUSE_REDSHIFT_DATABASE=dev
WAREHOUSE_REDSHIFT_SCHEMA=nself
WAREHOUSE_REDSHIFT_USER=nself_sync
WAREHOUSE_REDSHIFT_PASSWORD=your-passwordConfiguration Variable Required Default Description WAREHOUSE_DESTINATIONYes — Target warehouse: bigquery, snowflake, redshift WAREHOUSE_TABLESNo * Comma-separated np_* tables to sync, or * for all WAREHOUSE_WATERMARK_COLUMNNo updated_at Column used as the incremental sync cursor WAREHOUSE_SYNC_INTERVAL_MINUTESNo 60 How often to run the incremental sync WAREHOUSE_BATCH_SIZENo 5000 Rows per batch during initial load and incremental syncs WAREHOUSE_FULL_REFRESH_CRONNo 0 3 * * 0 Cron expression for weekly full-table refresh (reconciliation) WAREHOUSE_CDC_MODENo false Enable CDC-backed streaming sync (requires the cdc plugin) WAREHOUSE_SCHEMA_SYNCNo true Automatically propagate Postgres schema changes to the destination
API reference Endpoint Method Description /warehouse/syncPOST Trigger an immediate incremental sync /warehouse/sync/fullPOST Trigger a full-table refresh for one or all tables /warehouse/statusGET Last sync time, row counts, and lag per table /warehouse/schemaGET Current schema mapping between Postgres and the destination /warehouse/runsGET Sync run history with duration, rows synced, and error details /warehouse/healthGET Destination connectivity check
Database tables np_warehouse_runs — sync run records with row counts and statusnp_warehouse_cursors — per-table watermark position trackingnp_warehouse_schema_map — Postgres-to-destination column type mappingsnp_warehouse_errors — failed row payloads for investigation and replayDestination comparison Feature BigQuery Snowflake Redshift Auth method Service account JSON Username/password Username/password Schema auto-create Yes (dataset) Yes (schema) Yes (schema) CDC streaming Yes (via streaming inserts) Yes (via stage) Yes (via COPY) Cost model Per query / per GB loaded Virtual warehouse credits Node-hour based
Related cdc : WAL-based CDC for real-time streaming sync into the warehouseanalytics : In-Postgres event tracking — lighter alternative when you do not need a warehousemlflow : ML experiment tracking that can read from BigQuery or SnowflakePro Plugin — ɳSelf+ | v1.0.0