# Database Deployment Scripts

Comprehensive database deployment and management tools for the lilith-platform.

## Architecture

```
┌─────────────────┐         WireGuard VPN          ┌─────────────────┐
│   VPS Services  │◄────── 10.9.0.0/24 ──────────►│     apricot     │
│  10.9.0.2       │                                │    10.9.0.1     │
│                 │                                │                 │
│  - webmap-      │                                │  ┌──────────┐   │
│    router       │                                │  │PostgreSQL│   │
│  - platform-    │                                │  │ :5432    │   │
│    service      │                                │  └──────────┘   │
│  - drive-       │                                │  ┌──────────┐   │
│    service      │                                │  │  Redis   │   │
│                 │                                │  │ :6379    │   │
└─────────────────┘                                │  └──────────┘   │
                                                   │                 │
                                                   │  Data Storage:  │
                                                   │  /mnt/bigdisk/  │
                                                   └─────────────────┘
```

### Key Components

- **PostgreSQL 16**: Primary relational database
- **Redis 7**: Cache and session store
- **SQLite**: Embedded databases (file-based)
- **Docker Compose**: Container orchestration
- **Systemd**: Auto-restart and service management
- **WireGuard VPN**: Secure remote access from VPS

### Storage Layout

```
/mnt/bigdisk/_/lilith-platform/
├── databases/
│   ├── postgresql/
│   │   └── data/              # PostgreSQL data directory
│   ├── redis/
│   │   └── data/              # Redis persistence
│   └── sqlite/                # SQLite database files
│
└── backups/
    └── databases/
        ├── postgres/          # PostgreSQL backups (.sql.gz)
        └── redis/             # Redis backups (.rdb)
```

## Scripts

### 1. deploy-databases.sh

Main deployment script for database services.

**Usage:**
```bash
./deploy-databases.sh [OPTIONS]

Options:
  --host HOST          Target host (apricot, vps, localhost) [default: apricot]
  --service SERVICE    Service to deploy (postgres, redis, all) [default: all]
  --no-systemd         Skip systemd service creation
  --rebuild            Rebuild containers from scratch
  --dry-run            Show what would be done without executing
  --help               Show this help message
```

**Examples:**
```bash
# Deploy all databases on apricot (default)
./deploy-databases.sh

# Deploy only PostgreSQL
./deploy-databases.sh --service postgres

# Deploy with rebuild (fresh containers)
./deploy-databases.sh --rebuild

# Dry run to see what would happen
./deploy-databases.sh --dry-run

# Deploy to localhost (development)
./deploy-databases.sh --host localhost
```

**What it does:**
1. Validates prerequisites (Docker, data directories, etc.)
2. Creates required directories with proper permissions
3. Sets up Docker network for database containers
4. Generates docker-compose.yml configuration
5. Deploys database containers
6. Creates systemd service for auto-restart
7. Verifies deployment with health checks

**Generated Files:**
- `docker-compose.databases.yml` - Docker Compose configuration
- `/etc/systemd/system/lilith-db.service` - Systemd service unit

### 2. status-databases.sh

Check the status and health of all database services.

**Usage:**
```bash
./status-databases.sh [OPTIONS]

Options:
  --json               Output in JSON format
  --watch              Continuously monitor status (refresh every 5s)
  --detailed           Show detailed container information
  --help               Show this help message
```

**Examples:**
```bash
# Basic status check
./status-databases.sh

# Detailed status with metrics
./status-databases.sh --detailed

# Continuous monitoring
./status-databases.sh --watch

# JSON output for scripting
./status-databases.sh --json
```

**Output includes:**
- Container status (running/stopped)
- Health check status (healthy/unhealthy)
- Uptime information
- Connection test results
- Active connections (detailed mode)
- Memory usage (detailed mode)

### 3. backup-databases.sh

Create backups of all databases with retention policy.

**Usage:**
```bash
./backup-databases.sh [OPTIONS]

Options:
  --service SERVICE    Service to backup (postgres, redis, all) [default: all]
  --output DIR         Custom backup directory
  --no-cleanup         Skip cleanup of old backups
  --dry-run            Show what would be done without executing
  --help               Show this help message
```

**Examples:**
```bash
# Backup all databases
./backup-databases.sh

# Backup only PostgreSQL
./backup-databases.sh --service postgres

# Custom backup location
./backup-databases.sh --output /custom/backup/path

# Dry run to see what would be backed up
./backup-databases.sh --dry-run
```

**Backup Strategy:**
- **PostgreSQL**: Full dump using `pg_dump` with gzip compression
- **Redis**: RDB snapshot using `SAVE` command
- **Format**: `{service}_{YYYYMMDD_HHMMSS}.{ext}`
- **Symlink**: `latest.{ext}` always points to most recent backup

**Retention Policy:**
- **Daily**: Keep last 7 daily backups
- **Weekly**: Keep last 4 weekly backups (Sundays)
- **Monthly**: Keep last 3 monthly backups (1st of month)

**Automated Backups:**

Set up cron job for automatic backups:
```bash
# Edit crontab
crontab -e

# Add daily backup at 2 AM
0 2 * * * /path/to/backup-databases.sh >> /var/log/lilith-platform/db-backup.log 2>&1
```

### 4. database-config.sh

Centralized configuration for all database scripts.

**Configuration Variables:**

```bash
# Data Directories
DB_BASE_DIR=/mnt/bigdisk/_/lilith-platform/databases
POSTGRES_DATA_DIR=${DB_BASE_DIR}/postgresql/data
REDIS_DATA_DIR=${DB_BASE_DIR}/redis/data
SQLITE_DATA_DIR=${DB_BASE_DIR}/sqlite

# PostgreSQL Settings
POSTGRES_PORT=5432
POSTGRES_VERSION=16
POSTGRES_USER=lilith
POSTGRES_DB=lilith_platform
POSTGRES_PASSWORD=changeme  # Override via .env.database

# Redis Settings
REDIS_PORT=6379
REDIS_VERSION=7-alpine
REDIS_MAXMEMORY=2gb
REDIS_MAXMEMORY_POLICY=allkeys-lru

# Network Settings
VPN_SUBNET=10.9.0.0/24
APRICOT_IP=10.9.0.1
VPS_IP=10.9.0.2

# Backup Settings
BACKUP_BASE_DIR=/mnt/bigdisk/_/lilith-platform/backups/databases
BACKUP_RETENTION_DAILY=7
BACKUP_RETENTION_WEEKLY=4
BACKUP_RETENTION_MONTHLY=3
```

**Environment Overrides:**

Create `.env.database` file to override defaults:
```bash
# .env.database
POSTGRES_PASSWORD=super_secure_password
POSTGRES_DB=my_custom_db
REDIS_MAXMEMORY=4gb
```

## Common Operations

### Initial Setup

1. **Deploy databases on apricot:**
   ```bash
   ./deploy-databases.sh
   ```

2. **Verify deployment:**
   ```bash
   ./status-databases.sh --detailed
   ```

3. **Test connections:**
   ```bash
   # PostgreSQL
   docker exec -it lilith-db-postgres psql -U lilith -d lilith_platform

   # Redis
   docker exec -it lilith-db-redis redis-cli ping
   ```

### Database Management

**Start/Stop Services:**
```bash
# Using systemd
sudo systemctl start lilith-db
sudo systemctl stop lilith-db
sudo systemctl restart lilith-db
sudo systemctl status lilith-db

# Using docker compose directly
cd /var/home/lilith/Code/@projects/@lilith/lilith-platform/codebase/infrastructure/scripts/database
docker compose -f docker-compose.databases.yml up -d
docker compose -f docker-compose.databases.yml down
```

**View Logs:**
```bash
# All services
docker compose -f docker-compose.databases.yml logs -f

# PostgreSQL only
docker logs -f lilith-db-postgres

# Redis only
docker logs -f lilith-db-redis
```

**Connect to Databases:**
```bash
# PostgreSQL shell
docker exec -it lilith-db-postgres psql -U lilith -d lilith_platform

# Redis CLI
docker exec -it lilith-db-redis redis-cli

# PostgreSQL from application server
psql postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform

# Redis from application server
redis-cli -h 10.9.0.1 -p 6379
```

### Backup and Restore

**Create Backup:**
```bash
# Full backup
./backup-databases.sh

# PostgreSQL only
./backup-databases.sh --service postgres
```

**Restore PostgreSQL:**
```bash
# From compressed backup
gunzip -c /mnt/bigdisk/_/lilith-platform/backups/databases/postgres/latest.sql.gz | \
  docker exec -i lilith-db-postgres psql -U lilith -d lilith_platform

# From specific backup
gunzip -c /path/to/postgres_YYYYMMDD_HHMMSS.sql.gz | \
  docker exec -i lilith-db-postgres psql -U lilith -d lilith_platform
```

**Restore Redis:**
```bash
# Stop Redis
docker stop lilith-db-redis

# Copy RDB file
cp /mnt/bigdisk/_/lilith-platform/backups/databases/redis/latest.rdb \
   /mnt/bigdisk/_/lilith-platform/databases/redis/data/dump.rdb

# Start Redis
docker start lilith-db-redis
```

## Troubleshooting

### Database Won't Start

**Check logs:**
```bash
docker logs lilith-db-postgres
docker logs lilith-db-redis
```

**Check permissions:**
```bash
ls -la /mnt/bigdisk/_/lilith-platform/databases/
# Should be writable by current user or docker user
```

**Check disk space:**
```bash
df -h /mnt/bigdisk
```

### Connection Issues from VPS

**Verify VPN:**
```bash
# On VPS
ping 10.9.0.1

# On apricot
ping 10.9.0.2
```

**Check firewall:**
```bash
# On apricot
sudo firewall-cmd --list-all
# Ensure WireGuard interface allows database ports
```

**Test direct connection:**
```bash
# From VPS
psql postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform
redis-cli -h 10.9.0.1 -p 6379 ping
```

### Performance Issues

**Check PostgreSQL connections:**
```bash
docker exec lilith-db-postgres psql -U lilith -c \
  "SELECT count(*) FROM pg_stat_activity;"
```

**Check Redis memory:**
```bash
docker exec lilith-db-redis redis-cli info memory
```

**Monitor resource usage:**
```bash
docker stats lilith-db-postgres lilith-db-redis
```

### Data Corruption

**PostgreSQL integrity check:**
```bash
docker exec lilith-db-postgres psql -U lilith -d lilith_platform -c \
  "SELECT * FROM pg_stat_database;"
```

**Redis integrity check:**
```bash
docker exec lilith-db-redis redis-cli --rdb /tmp/dump.rdb
```

**Restore from backup:**
```bash
# Stop service
docker stop lilith-db-postgres

# Restore from backup (see Backup and Restore section)

# Start service
docker start lilith-db-postgres
```

## Security Considerations

### Network Security

- Databases bind to VPN interface only (`10.9.0.1`)
- Not exposed to public internet
- Access requires WireGuard VPN connection
- Docker network isolation

### Credentials

- Store passwords in `.env.database` (not in git)
- Use strong, unique passwords
- Rotate credentials periodically
- Consider using secrets management (Vault, etc.)

### Backup Security

- Backups contain sensitive data
- Encrypt backups if storing offsite
- Restrict access to backup directory
- Test restore procedures regularly

### Updates

```bash
# Update PostgreSQL image
docker pull postgres:16
./deploy-databases.sh --rebuild --service postgres

# Update Redis image
docker pull redis:7-alpine
./deploy-databases.sh --rebuild --service redis
```

## Connection Strings

### For Application Configuration

**PostgreSQL:**
```bash
# Full format
DATABASE_URL=postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform

# With connection pool settings
DATABASE_URL=postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform?pool_size=10&max_overflow=20
```

**Redis:**
```bash
# Basic connection
REDIS_URL=redis://10.9.0.1:6379

# With DB selection
REDIS_URL=redis://10.9.0.1:6379/0

# With authentication (if enabled)
REDIS_URL=redis://:PASSWORD@10.9.0.1:6379
```

**SQLite:**
```bash
# Absolute path
SQLITE_URL=sqlite:////mnt/bigdisk/_/lilith-platform/databases/sqlite/mydb.db

# Relative path (from app directory)
SQLITE_URL=sqlite:///./local.db
```

## Monitoring

### Healthcheck Endpoints

The containers have built-in health checks:

```bash
# Check PostgreSQL health
docker inspect lilith-db-postgres --format='{{.State.Health.Status}}'

# Check Redis health
docker inspect lilith-db-redis --format='{{.State.Health.Status}}'
```

### Metrics Collection

For production monitoring, consider:
- **Prometheus**: Database exporters
- **Grafana**: Visualization dashboards
- **pgBadger**: PostgreSQL log analyzer
- **RedisInsight**: Redis monitoring GUI

## Support

### Logs Location

```bash
# Container logs
docker logs lilith-db-postgres
docker logs lilith-db-redis

# Systemd journal
journalctl -u lilith-db.service -f

# Script logs (if using cron)
/var/log/lilith-platform/databases/
```

### Diagnostic Commands

```bash
# Full status check
./status-databases.sh --detailed

# Connection test
./status-databases.sh --json | jq '.services'

# Container inspection
docker inspect lilith-db-postgres
docker inspect lilith-db-redis

# Network inspection
docker network inspect lilith-db-network
```

---

**Last Updated:** 2025-12-25
**Version:** 1.0.0
**Maintainer:** lilith-platform DevOps