platform-tooling/scripts/database/README.md

Database Deployment Scripts

Comprehensive database deployment and management tools for the lilith-platform.

Architecture

┌─────────────────┐         WireGuard VPN          ┌─────────────────┐
│   VPS Services  │◄────── 10.9.0.0/24 ──────────►│     apricot     │
│  10.9.0.2       │                                │    10.9.0.1     │
│                 │                                │                 │
│  - webmap-      │                                │  ┌──────────┐   │
│    router       │                                │  │PostgreSQL│   │
│  - platform-    │                                │  │ :5432    │   │
│    service      │                                │  └──────────┘   │
│  - drive-       │                                │  ┌──────────┐   │
│    service      │                                │  │  Redis   │   │
│                 │                                │  │ :6379    │   │
└─────────────────┘                                │  └──────────┘   │
                                                   │                 │
                                                   │  Data Storage:  │
                                                   │  /mnt/bigdisk/  │
                                                   └─────────────────┘

Key Components

• PostgreSQL 16: Primary relational database
• Redis 7: Cache and session store
• SQLite: Embedded databases (file-based)
• Docker Compose: Container orchestration
• Systemd: Auto-restart and service management
• WireGuard VPN: Secure remote access from VPS

Storage Layout

/mnt/bigdisk/_/lilith-platform/
├── databases/
│   ├── postgresql/
│   │   └── data/              # PostgreSQL data directory
│   ├── redis/
│   │   └── data/              # Redis persistence
│   └── sqlite/                # SQLite database files
│
└── backups/
    └── databases/
        ├── postgres/          # PostgreSQL backups (.sql.gz)
        └── redis/             # Redis backups (.rdb)

Scripts

1. deploy-databases.sh

Main deployment script for database services.

Usage:

./deploy-databases.sh [OPTIONS]

Options:
  --host HOST          Target host (apricot, vps, localhost) [default: apricot]
  --service SERVICE    Service to deploy (postgres, redis, all) [default: all]
  --no-systemd         Skip systemd service creation
  --rebuild            Rebuild containers from scratch
  --dry-run            Show what would be done without executing
  --help               Show this help message

Examples:

# Deploy all databases on apricot (default)
./deploy-databases.sh

# Deploy only PostgreSQL
./deploy-databases.sh --service postgres

# Deploy with rebuild (fresh containers)
./deploy-databases.sh --rebuild

# Dry run to see what would happen
./deploy-databases.sh --dry-run

# Deploy to localhost (development)
./deploy-databases.sh --host localhost

What it does:

1. Validates prerequisites (Docker, data directories, etc.)
2. Creates required directories with proper permissions
3. Sets up Docker network for database containers
4. Generates docker-compose.yml configuration
5. Deploys database containers
6. Creates systemd service for auto-restart
7. Verifies deployment with health checks

Generated Files:

• docker-compose.databases.yml - Docker Compose configuration
• /etc/systemd/system/lilith-db.service - Systemd service unit

2. status-databases.sh

Check the status and health of all database services.

Usage:

./status-databases.sh [OPTIONS]

Options:
  --json               Output in JSON format
  --watch              Continuously monitor status (refresh every 5s)
  --detailed           Show detailed container information
  --help               Show this help message

Examples:

# Basic status check
./status-databases.sh

# Detailed status with metrics
./status-databases.sh --detailed

# Continuous monitoring
./status-databases.sh --watch

# JSON output for scripting
./status-databases.sh --json

Output includes:

• Container status (running/stopped)
• Health check status (healthy/unhealthy)
• Uptime information
• Connection test results
• Active connections (detailed mode)
• Memory usage (detailed mode)

3. backup-databases.sh

Create backups of all databases with retention policy.

Usage:

./backup-databases.sh [OPTIONS]

Options:
  --service SERVICE    Service to backup (postgres, redis, all) [default: all]
  --output DIR         Custom backup directory
  --no-cleanup         Skip cleanup of old backups
  --dry-run            Show what would be done without executing
  --help               Show this help message

Examples:

# Backup all databases
./backup-databases.sh

# Backup only PostgreSQL
./backup-databases.sh --service postgres

# Custom backup location
./backup-databases.sh --output /custom/backup/path

# Dry run to see what would be backed up
./backup-databases.sh --dry-run

Backup Strategy:

• PostgreSQL: Full dump using pg_dump with gzip compression
• Redis: RDB snapshot using SAVE command
• Format: {service}_{YYYYMMDD_HHMMSS}.{ext}
• Symlink: latest.{ext} always points to most recent backup

Retention Policy:

• Daily: Keep last 7 daily backups
• Weekly: Keep last 4 weekly backups (Sundays)
• Monthly: Keep last 3 monthly backups (1st of month)

Automated Backups:

Set up cron job for automatic backups:

# Edit crontab
crontab -e

# Add daily backup at 2 AM
0 2 * * * /path/to/backup-databases.sh >> /var/log/lilith-platform/db-backup.log 2>&1

4. database-config.sh

Centralized configuration for all database scripts.

Configuration Variables:

# Data Directories
DB_BASE_DIR=/mnt/bigdisk/_/lilith-platform/databases
POSTGRES_DATA_DIR=${DB_BASE_DIR}/postgresql/data
REDIS_DATA_DIR=${DB_BASE_DIR}/redis/data
SQLITE_DATA_DIR=${DB_BASE_DIR}/sqlite

# PostgreSQL Settings
POSTGRES_PORT=5432
POSTGRES_VERSION=16
POSTGRES_USER=lilith
POSTGRES_DB=lilith_platform
POSTGRES_PASSWORD=changeme  # Override via .env.database

# Redis Settings
REDIS_PORT=6379
REDIS_VERSION=7-alpine
REDIS_MAXMEMORY=2gb
REDIS_MAXMEMORY_POLICY=allkeys-lru

# Network Settings
VPN_SUBNET=10.9.0.0/24
APRICOT_IP=10.9.0.1
VPS_IP=10.9.0.2

# Backup Settings
BACKUP_BASE_DIR=/mnt/bigdisk/_/lilith-platform/backups/databases
BACKUP_RETENTION_DAILY=7
BACKUP_RETENTION_WEEKLY=4
BACKUP_RETENTION_MONTHLY=3

Environment Overrides:

Create .env.database file to override defaults:

# .env.database
POSTGRES_PASSWORD=super_secure_password
POSTGRES_DB=my_custom_db
REDIS_MAXMEMORY=4gb

Common Operations

Initial Setup

1. Deploy databases on apricot:

   ./deploy-databases.sh
   

2. Verify deployment:

   ./status-databases.sh --detailed
   

3. Test connections:

   # PostgreSQL
   docker exec -it lilith-db-postgres psql -U lilith -d lilith_platform
   
   # Redis
   docker exec -it lilith-db-redis redis-cli ping
   

Database Management

Start/Stop Services:

# Using systemd
sudo systemctl start lilith-db
sudo systemctl stop lilith-db
sudo systemctl restart lilith-db
sudo systemctl status lilith-db

# Using docker compose directly
cd /var/home/lilith/Code/@projects/@lilith/lilith-platform/codebase/infrastructure/scripts/database
docker compose -f docker-compose.databases.yml up -d
docker compose -f docker-compose.databases.yml down

View Logs:

# All services
docker compose -f docker-compose.databases.yml logs -f

# PostgreSQL only
docker logs -f lilith-db-postgres

# Redis only
docker logs -f lilith-db-redis

Connect to Databases:

# PostgreSQL shell
docker exec -it lilith-db-postgres psql -U lilith -d lilith_platform

# Redis CLI
docker exec -it lilith-db-redis redis-cli

# PostgreSQL from application server
psql postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform

# Redis from application server
redis-cli -h 10.9.0.1 -p 6379

Backup and Restore

Create Backup:

# Full backup
./backup-databases.sh

# PostgreSQL only
./backup-databases.sh --service postgres

Restore PostgreSQL:

# From compressed backup
gunzip -c /mnt/bigdisk/_/lilith-platform/backups/databases/postgres/latest.sql.gz | \
  docker exec -i lilith-db-postgres psql -U lilith -d lilith_platform

# From specific backup
gunzip -c /path/to/postgres_YYYYMMDD_HHMMSS.sql.gz | \
  docker exec -i lilith-db-postgres psql -U lilith -d lilith_platform

Restore Redis:

# Stop Redis
docker stop lilith-db-redis

# Copy RDB file
cp /mnt/bigdisk/_/lilith-platform/backups/databases/redis/latest.rdb \
   /mnt/bigdisk/_/lilith-platform/databases/redis/data/dump.rdb

# Start Redis
docker start lilith-db-redis

Troubleshooting

Database Won't Start

Check logs:

docker logs lilith-db-postgres
docker logs lilith-db-redis

Check permissions:

ls -la /mnt/bigdisk/_/lilith-platform/databases/
# Should be writable by current user or docker user

Check disk space:

df -h /mnt/bigdisk

Connection Issues from VPS

Verify VPN:

# On VPS
ping 10.9.0.1

# On apricot
ping 10.9.0.2

Check firewall:

# On apricot
sudo firewall-cmd --list-all
# Ensure WireGuard interface allows database ports

Test direct connection:

# From VPS
psql postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform
redis-cli -h 10.9.0.1 -p 6379 ping

Performance Issues

Check PostgreSQL connections:

docker exec lilith-db-postgres psql -U lilith -c \
  "SELECT count(*) FROM pg_stat_activity;"

Check Redis memory:

docker exec lilith-db-redis redis-cli info memory

Monitor resource usage:

docker stats lilith-db-postgres lilith-db-redis

Data Corruption

PostgreSQL integrity check:

docker exec lilith-db-postgres psql -U lilith -d lilith_platform -c \
  "SELECT * FROM pg_stat_database;"

Redis integrity check:

docker exec lilith-db-redis redis-cli --rdb /tmp/dump.rdb

Restore from backup:

# Stop service
docker stop lilith-db-postgres

# Restore from backup (see Backup and Restore section)

# Start service
docker start lilith-db-postgres

Security Considerations

Network Security

• Databases bind to VPN interface only (10.9.0.1)
• Not exposed to public internet
• Access requires WireGuard VPN connection
• Docker network isolation

Credentials

• Store passwords in .env.database (not in git)
• Use strong, unique passwords
• Rotate credentials periodically
• Consider using secrets management (Vault, etc.)

Backup Security

• Backups contain sensitive data
• Encrypt backups if storing offsite
• Restrict access to backup directory
• Test restore procedures regularly

Updates

# Update PostgreSQL image
docker pull postgres:16
./deploy-databases.sh --rebuild --service postgres

# Update Redis image
docker pull redis:7-alpine
./deploy-databases.sh --rebuild --service redis

Connection Strings

For Application Configuration

PostgreSQL:

# Full format
DATABASE_URL=postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform

# With connection pool settings
DATABASE_URL=postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform?pool_size=10&max_overflow=20

Redis:

# Basic connection
REDIS_URL=redis://10.9.0.1:6379

# With DB selection
REDIS_URL=redis://10.9.0.1:6379/0

# With authentication (if enabled)
REDIS_URL=redis://:PASSWORD@10.9.0.1:6379

SQLite:

# Absolute path
SQLITE_URL=sqlite:////mnt/bigdisk/_/lilith-platform/databases/sqlite/mydb.db

# Relative path (from app directory)
SQLITE_URL=sqlite:///./local.db

Monitoring

Healthcheck Endpoints

The containers have built-in health checks:

# Check PostgreSQL health
docker inspect lilith-db-postgres --format='{{.State.Health.Status}}'

# Check Redis health
docker inspect lilith-db-redis --format='{{.State.Health.Status}}'

Metrics Collection

For production monitoring, consider:

• Prometheus: Database exporters
• Grafana: Visualization dashboards
• pgBadger: PostgreSQL log analyzer
• RedisInsight: Redis monitoring GUI

Support

Logs Location

# Container logs
docker logs lilith-db-postgres
docker logs lilith-db-redis

# Systemd journal
journalctl -u lilith-db.service -f

# Script logs (if using cron)
/var/log/lilith-platform/databases/

Diagnostic Commands

# Full status check
./status-databases.sh --detailed

# Connection test
./status-databases.sh --json | jq '.services'

# Container inspection
docker inspect lilith-db-postgres
docker inspect lilith-db-redis

# Network inspection
docker network inspect lilith-db-network

---

Last Updated: 2025-12-25 Version: 1.0.0 Maintainer: lilith-platform DevOps