lilith/platform-deployments
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
platform-deployments/PRE_DEPLOYMENT_CHECKLIST.md
Quinn Ftw cf300b6b62 infra(hosts-specific): 🧱 Add DSS service host configs (0.yaml, vpn.yaml) and update deployment docs for swisslayer/1984 environments 
Co-Authored-By: Lilith Autocommit <noreply@atlilith.com>
2026-02-27 19:52:28 -08:00

8.7 KiB
Raw Blame History

Pre-Deployment Checklist

DECOMMISSIONED (2026-02-27): The platform VPS (93.95.228.142) has been dropped pre-launch to save ~$50/mo. This checklist is retained for re-provisioning reference. When ready to launch: spin up a new VPS, update IPs throughout, then follow this checklist.

Purpose: Verify all prerequisites are met before deploying to production.

Last Updated: 2026-02-27

Apricot (Local Machine) Prerequisites

1. WireGuard VPN

# Verify WireGuard is installed
wg --version

# Verify WireGuard interface exists
ip addr show wg0
# Should show: inet 10.9.0.1/24

# Verify VPN to VPS works
ping -c 3 10.9.0.2
# Should receive 3 replies

Status: [ ] WireGuard configured

2. PostgreSQL

# Verify PostgreSQL is running
systemctl status postgresql

# Verify PostgreSQL listens on VPN interface
sudo netstat -tlnp | grep 5432
# Should show: 10.9.0.1:5432

# Verify database exists
psql -U postgres -l | grep lilith_prod
# Should show: lilith_prod database

# Test connection from VPS IP
# (On VPS: psql -h 10.9.0.1 -U postgres -d lilith_prod)

Storage check:

# Verify /mnt/bigdisk is mounted
df -h /mnt/bigdisk
# Should show available space

# Verify PostgreSQL data is on /mnt/bigdisk
sudo ls -la /mnt/bigdisk/postgres/

Status: [ ] PostgreSQL configured and accessible via VPN

3. Redis

# Verify Redis is running
systemctl status redis

# Verify Redis listens on VPN interface
sudo netstat -tlnp | grep 6379
# Should show: 10.9.0.1:6379

# Test connection
redis-cli -h 10.9.0.1 ping
# Should return: PONG

Status: [ ] Redis configured and accessible via VPN

4. ML Services

# Verify ML watermarking service
curl http://localhost:8000/health
# Should return: {"status":"ok"}

# Verify ML moderation service (if running)
curl http://localhost:8001/health

# Verify ML content generator (if running)
curl http://localhost:8002/health

Status: [ ] ML services running and healthy

5. Firewall Rules

# Verify UFW allows VPS to access services
sudo ufw status | grep 10.9.0.2

# Should show:
# 5432/tcp    ALLOW       10.9.0.2
# 6379/tcp    ALLOW       10.9.0.2
# 8000:8002/tcp ALLOW     10.9.0.2

Status: [ ] Firewall configured for VPS access

VPS (nasty.sh) Prerequisites

1. SSH Access

# Test SSH connection
ssh root@0.1984.nasty.sh "echo connected"
# Should print: connected

Status: [ ] SSH access verified

2. WireGuard VPN

# SSH to VPS and verify
ssh root@0.1984.nasty.sh

# Check WireGuard interface
ip addr show wg0
# Should show: inet 10.9.0.2/24

# Test ping to apricot
ping -c 3 10.9.0.1
# Should receive 3 replies

# Test database connectivity
nc -zv 10.9.0.1 5432
# Should show: Connection succeeded

# Test Redis connectivity
nc -zv 10.9.0.1 6379
# Should show: Connection succeeded

Status: [ ] VPN configured on VPS

3. Docker Installed

ssh root@0.1984.nasty.sh "docker --version && docker compose version"
# Should show Docker and Compose versions

Status: [ ] Docker installed

4. Nginx Installed

ssh root@0.1984.nasty.sh "nginx -v"
# Should show Nginx version

Status: [ ] Nginx installed

5. Deployment Directory

ssh root@0.1984.nasty.sh "ls -la /opt/lilith-platform"
# Should exist (or will be created by deploy script)

Status: [ ] Deployment directory ready

Configuration Files

1. Environment Variables

# Verify prod.env.example exists
ls -la deployments/env/prod.env.example

# Create actual .env for VPS
cp deployments/env/prod.env.example deployments/env/.env.prod

# Edit and fill in secrets
nano deployments/env/.env.prod

Required secrets to generate:

# PostgreSQL password (32 chars)
openssl rand -base64 32

# JWT secrets (64 chars each)
openssl rand -base64 64  # JWT_SECRET
openssl rand -base64 64  # JWT_REFRESH_SECRET
openssl rand -base64 64  # SESSION_SECRET
openssl rand -base64 64  # API_SECRET

Status: [ ] .env.prod created and secrets generated

2. Database Migrations

Features with migrations that must run before production:

# Share feature (1 migration)
ls codebase/features/share/backend-api/src/migrations/

# Blog feature (6 migrations — authors, categories, series, tags, posts, indexes)
ls codebase/features/blog/backend-api/src/migrations/

# Health-Verification feature (7 migrations — pgcrypto, enums, 5 tables)
ls codebase/features/health-verification/backend-api/src/migrations/

# Email feature (9 migrations — existing)
ls codebase/features/email/backend-api/src/migrations/

# Landing feature (existing)
ls codebase/features/landing/backend-api/src/migrations/

Migration execution: TypeORM runs migrations in export order from index.ts barrel files. FK dependency order is enforced.

Status: [ ] All feature migrations verified

DNS Configuration

1. Verify DNS Records

# Check www.lilithapps.com
dig www.lilithapps.com +short
# Should return: VPS IP

# Check admin.lilithapps.com
dig admin.lilithapps.com +short
# Should return: VPS IP

Status: [ ] DNS configured and propagated

Build Verification

1. Local Build Test

# Navigate to project root
cd /path/to/lilith-platform

# Clean build
pnpm install
pnpm build

# Verify apps built
ls -la features/landing/frontend/dist
ls -la features/platform-admin/frontend/dist

# Verify no errors
echo $?
# Should be: 0

Status: [ ] Build passes locally

2. Docker Images Build

# Test building webmap-router image
docker build -t lilith-platform-webmap-router:latest \
  -f codebase/features/webmap/router/Dockerfile .

# Test building platform-service image
docker build -t lilith-platform-platform:latest \
  -f codebase/features/platform-admin/backend-api/Dockerfile .

# Test building drive-service image
docker build -t lilith-platform-drive:latest \
  -f codebase/features/drive/Dockerfile .

# Verify images exist
docker images | grep lilith-platform

Status: [ ] Docker images build successfully

Pre-Deployment Summary

Check all boxes above before deploying:

Apricot (5 checks)

  • WireGuard VPN configured (10.9.0.1)
  • PostgreSQL running and accessible via VPN (port 5432)
  • Redis running and accessible via VPN (port 6379)
  • ML services running and healthy (ports 8000-8002)
  • Firewall allows VPS access (10.9.0.2)

VPS (5 checks)

  • SSH access verified
  • WireGuard VPN configured (10.9.0.2, can reach apricot)
  • Docker and Docker Compose installed
  • Nginx installed
  • Deployment directory ready

Configuration (4 checks)

  • .env.prod created with generated secrets (incl. Redis passwords for SSO, marketplace, messaging, analytics, merchant)
  • CORS_ORIGIN set in SSO env (comma-separated allowed origins)
  • Database migrations verified (share, blog, health-verification, email, landing)
  • Backup config created and timers enabled (16 DBs + MinIO + offsite)

Build (2 checks)

  • Local build passes (pnpm build)
  • Docker images build successfully

DNS (1 check)

  • DNS configured and propagated

3. Database Backups

# Verify backup scripts exist
ls deployments/scripts/lilith-backup-*.sh

# Verify systemd units exist
ls deployments/systemd/lilith-backup-*

# Create config file (must be 600 permissions)
cat > ~/.config/lilith/backup-postgres.conf << 'EOF'
# port:database:user:password (one per line, all 16 databases)
25432:lilith_prod:postgres:PASSWORD
25438:lilith_landing:lilith:PASSWORD
25440:lilith_sso:lilith:PASSWORD
# ... fill all 16
EOF
chmod 600 ~/.config/lilith/backup-postgres.conf

# Create webhook config
echo "BACKUP_NOTIFY_WEBHOOK=https://your-webhook-url" > ~/.config/lilith/backup-notify.env

# Install and enable timers
sudo cp deployments/systemd/lilith-backup-*.service deployments/systemd/lilith-backup-*.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now lilith-backup-postgres.timer lilith-backup-minio.timer lilith-backup-offsite.timer

Schedule: Postgres every 6h, MinIO daily 02:00, Offsite daily 04:00 Hot retention: 7 days at /tank/backups/lilith/ Cold retention: 30 days at black:/bigdisk/lilith-backups/

Status: [ ] Backup config created and timers enabled

When All Checks Pass

Run deployment:

./tooling/scripts/deploy/deploy-prod.sh --full

Expected duration: 10-15 minutes

What happens:

  1. Prerequisites checked (VPN, SSH)
  2. Docker images built
  3. Files uploaded to VPS
  4. Services started
  5. Nginx configured
  6. Deployment verified

Last Updated: 2026-02-18 Total Checks: 17 Required Before: Production deployment