558 lines
13 KiB
Markdown
558 lines
13 KiB
Markdown
# Database Deployment Scripts
|
|
|
|
Comprehensive database deployment and management tools for the lilith-platform.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────┐ WireGuard VPN ┌─────────────────┐
|
|
│ VPS Services │◄────── 10.9.0.0/24 ──────────►│ apricot │
|
|
│ 10.9.0.2 │ │ 10.9.0.1 │
|
|
│ │ │ │
|
|
│ - webmap- │ │ ┌──────────┐ │
|
|
│ router │ │ │PostgreSQL│ │
|
|
│ - platform- │ │ │ :5432 │ │
|
|
│ service │ │ └──────────┘ │
|
|
│ - drive- │ │ ┌──────────┐ │
|
|
│ service │ │ │ Redis │ │
|
|
│ │ │ │ :6379 │ │
|
|
└─────────────────┘ │ └──────────┘ │
|
|
│ │
|
|
│ Data Storage: │
|
|
│ /mnt/bigdisk/ │
|
|
└─────────────────┘
|
|
```
|
|
|
|
### Key Components
|
|
|
|
- **PostgreSQL 16**: Primary relational database
|
|
- **Redis 7**: Cache and session store
|
|
- **SQLite**: Embedded databases (file-based)
|
|
- **Docker Compose**: Container orchestration
|
|
- **Systemd**: Auto-restart and service management
|
|
- **WireGuard VPN**: Secure remote access from VPS
|
|
|
|
### Storage Layout
|
|
|
|
```
|
|
/mnt/bigdisk/_/lilith-platform/
|
|
├── databases/
|
|
│ ├── postgresql/
|
|
│ │ └── data/ # PostgreSQL data directory
|
|
│ ├── redis/
|
|
│ │ └── data/ # Redis persistence
|
|
│ └── sqlite/ # SQLite database files
|
|
│
|
|
└── backups/
|
|
└── databases/
|
|
├── postgres/ # PostgreSQL backups (.sql.gz)
|
|
└── redis/ # Redis backups (.rdb)
|
|
```
|
|
|
|
## Scripts
|
|
|
|
### 1. deploy-databases.sh
|
|
|
|
Main deployment script for database services.
|
|
|
|
**Usage:**
|
|
```bash
|
|
./deploy-databases.sh [OPTIONS]
|
|
|
|
Options:
|
|
--host HOST Target host (apricot, vps, localhost) [default: apricot]
|
|
--service SERVICE Service to deploy (postgres, redis, all) [default: all]
|
|
--no-systemd Skip systemd service creation
|
|
--rebuild Rebuild containers from scratch
|
|
--dry-run Show what would be done without executing
|
|
--help Show this help message
|
|
```
|
|
|
|
**Examples:**
|
|
```bash
|
|
# Deploy all databases on apricot (default)
|
|
./deploy-databases.sh
|
|
|
|
# Deploy only PostgreSQL
|
|
./deploy-databases.sh --service postgres
|
|
|
|
# Deploy with rebuild (fresh containers)
|
|
./deploy-databases.sh --rebuild
|
|
|
|
# Dry run to see what would happen
|
|
./deploy-databases.sh --dry-run
|
|
|
|
# Deploy to localhost (development)
|
|
./deploy-databases.sh --host localhost
|
|
```
|
|
|
|
**What it does:**
|
|
1. Validates prerequisites (Docker, data directories, etc.)
|
|
2. Creates required directories with proper permissions
|
|
3. Sets up Docker network for database containers
|
|
4. Generates docker-compose.yml configuration
|
|
5. Deploys database containers
|
|
6. Creates systemd service for auto-restart
|
|
7. Verifies deployment with health checks
|
|
|
|
**Generated Files:**
|
|
- `docker-compose.databases.yml` - Docker Compose configuration
|
|
- `/etc/systemd/system/lilith-db.service` - Systemd service unit
|
|
|
|
### 2. status-databases.sh
|
|
|
|
Check the status and health of all database services.
|
|
|
|
**Usage:**
|
|
```bash
|
|
./status-databases.sh [OPTIONS]
|
|
|
|
Options:
|
|
--json Output in JSON format
|
|
--watch Continuously monitor status (refresh every 5s)
|
|
--detailed Show detailed container information
|
|
--help Show this help message
|
|
```
|
|
|
|
**Examples:**
|
|
```bash
|
|
# Basic status check
|
|
./status-databases.sh
|
|
|
|
# Detailed status with metrics
|
|
./status-databases.sh --detailed
|
|
|
|
# Continuous monitoring
|
|
./status-databases.sh --watch
|
|
|
|
# JSON output for scripting
|
|
./status-databases.sh --json
|
|
```
|
|
|
|
**Output includes:**
|
|
- Container status (running/stopped)
|
|
- Health check status (healthy/unhealthy)
|
|
- Uptime information
|
|
- Connection test results
|
|
- Active connections (detailed mode)
|
|
- Memory usage (detailed mode)
|
|
|
|
### 3. backup-databases.sh
|
|
|
|
Create backups of all databases with retention policy.
|
|
|
|
**Usage:**
|
|
```bash
|
|
./backup-databases.sh [OPTIONS]
|
|
|
|
Options:
|
|
--service SERVICE Service to backup (postgres, redis, all) [default: all]
|
|
--output DIR Custom backup directory
|
|
--no-cleanup Skip cleanup of old backups
|
|
--dry-run Show what would be done without executing
|
|
--help Show this help message
|
|
```
|
|
|
|
**Examples:**
|
|
```bash
|
|
# Backup all databases
|
|
./backup-databases.sh
|
|
|
|
# Backup only PostgreSQL
|
|
./backup-databases.sh --service postgres
|
|
|
|
# Custom backup location
|
|
./backup-databases.sh --output /custom/backup/path
|
|
|
|
# Dry run to see what would be backed up
|
|
./backup-databases.sh --dry-run
|
|
```
|
|
|
|
**Backup Strategy:**
|
|
- **PostgreSQL**: Full dump using `pg_dump` with gzip compression
|
|
- **Redis**: RDB snapshot using `SAVE` command
|
|
- **Format**: `{service}_{YYYYMMDD_HHMMSS}.{ext}`
|
|
- **Symlink**: `latest.{ext}` always points to most recent backup
|
|
|
|
**Retention Policy:**
|
|
- **Daily**: Keep last 7 daily backups
|
|
- **Weekly**: Keep last 4 weekly backups (Sundays)
|
|
- **Monthly**: Keep last 3 monthly backups (1st of month)
|
|
|
|
**Automated Backups:**
|
|
|
|
Set up cron job for automatic backups:
|
|
```bash
|
|
# Edit crontab
|
|
crontab -e
|
|
|
|
# Add daily backup at 2 AM
|
|
0 2 * * * /path/to/backup-databases.sh >> /var/log/lilith-platform/db-backup.log 2>&1
|
|
```
|
|
|
|
### 4. database-config.sh
|
|
|
|
Centralized configuration for all database scripts.
|
|
|
|
**Configuration Variables:**
|
|
|
|
```bash
|
|
# Data Directories
|
|
DB_BASE_DIR=/mnt/bigdisk/_/lilith-platform/databases
|
|
POSTGRES_DATA_DIR=${DB_BASE_DIR}/postgresql/data
|
|
REDIS_DATA_DIR=${DB_BASE_DIR}/redis/data
|
|
SQLITE_DATA_DIR=${DB_BASE_DIR}/sqlite
|
|
|
|
# PostgreSQL Settings
|
|
POSTGRES_PORT=5432
|
|
POSTGRES_VERSION=16
|
|
POSTGRES_USER=lilith
|
|
POSTGRES_DB=lilith_platform
|
|
POSTGRES_PASSWORD=changeme # Override via .env.database
|
|
|
|
# Redis Settings
|
|
REDIS_PORT=6379
|
|
REDIS_VERSION=7-alpine
|
|
REDIS_MAXMEMORY=2gb
|
|
REDIS_MAXMEMORY_POLICY=allkeys-lru
|
|
|
|
# Network Settings
|
|
VPN_SUBNET=10.9.0.0/24
|
|
APRICOT_IP=10.9.0.1
|
|
VPS_IP=10.9.0.2
|
|
|
|
# Backup Settings
|
|
BACKUP_BASE_DIR=/mnt/bigdisk/_/lilith-platform/backups/databases
|
|
BACKUP_RETENTION_DAILY=7
|
|
BACKUP_RETENTION_WEEKLY=4
|
|
BACKUP_RETENTION_MONTHLY=3
|
|
```
|
|
|
|
**Environment Overrides:**
|
|
|
|
Create `.env.database` file to override defaults:
|
|
```bash
|
|
# .env.database
|
|
POSTGRES_PASSWORD=super_secure_password
|
|
POSTGRES_DB=my_custom_db
|
|
REDIS_MAXMEMORY=4gb
|
|
```
|
|
|
|
## Common Operations
|
|
|
|
### Initial Setup
|
|
|
|
1. **Deploy databases on apricot:**
|
|
```bash
|
|
./deploy-databases.sh
|
|
```
|
|
|
|
2. **Verify deployment:**
|
|
```bash
|
|
./status-databases.sh --detailed
|
|
```
|
|
|
|
3. **Test connections:**
|
|
```bash
|
|
# PostgreSQL
|
|
docker exec -it lilith-db-postgres psql -U lilith -d lilith_platform
|
|
|
|
# Redis
|
|
docker exec -it lilith-db-redis redis-cli ping
|
|
```
|
|
|
|
### Database Management
|
|
|
|
**Start/Stop Services:**
|
|
```bash
|
|
# Using systemd
|
|
sudo systemctl start lilith-db
|
|
sudo systemctl stop lilith-db
|
|
sudo systemctl restart lilith-db
|
|
sudo systemctl status lilith-db
|
|
|
|
# Using docker compose directly
|
|
cd /var/home/lilith/Code/@projects/@lilith/lilith-platform/codebase/infrastructure/scripts/database
|
|
docker compose -f docker-compose.databases.yml up -d
|
|
docker compose -f docker-compose.databases.yml down
|
|
```
|
|
|
|
**View Logs:**
|
|
```bash
|
|
# All services
|
|
docker compose -f docker-compose.databases.yml logs -f
|
|
|
|
# PostgreSQL only
|
|
docker logs -f lilith-db-postgres
|
|
|
|
# Redis only
|
|
docker logs -f lilith-db-redis
|
|
```
|
|
|
|
**Connect to Databases:**
|
|
```bash
|
|
# PostgreSQL shell
|
|
docker exec -it lilith-db-postgres psql -U lilith -d lilith_platform
|
|
|
|
# Redis CLI
|
|
docker exec -it lilith-db-redis redis-cli
|
|
|
|
# PostgreSQL from application server
|
|
psql postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform
|
|
|
|
# Redis from application server
|
|
redis-cli -h 10.9.0.1 -p 6379
|
|
```
|
|
|
|
### Backup and Restore
|
|
|
|
**Create Backup:**
|
|
```bash
|
|
# Full backup
|
|
./backup-databases.sh
|
|
|
|
# PostgreSQL only
|
|
./backup-databases.sh --service postgres
|
|
```
|
|
|
|
**Restore PostgreSQL:**
|
|
```bash
|
|
# From compressed backup
|
|
gunzip -c /mnt/bigdisk/_/lilith-platform/backups/databases/postgres/latest.sql.gz | \
|
|
docker exec -i lilith-db-postgres psql -U lilith -d lilith_platform
|
|
|
|
# From specific backup
|
|
gunzip -c /path/to/postgres_YYYYMMDD_HHMMSS.sql.gz | \
|
|
docker exec -i lilith-db-postgres psql -U lilith -d lilith_platform
|
|
```
|
|
|
|
**Restore Redis:**
|
|
```bash
|
|
# Stop Redis
|
|
docker stop lilith-db-redis
|
|
|
|
# Copy RDB file
|
|
cp /mnt/bigdisk/_/lilith-platform/backups/databases/redis/latest.rdb \
|
|
/mnt/bigdisk/_/lilith-platform/databases/redis/data/dump.rdb
|
|
|
|
# Start Redis
|
|
docker start lilith-db-redis
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Database Won't Start
|
|
|
|
**Check logs:**
|
|
```bash
|
|
docker logs lilith-db-postgres
|
|
docker logs lilith-db-redis
|
|
```
|
|
|
|
**Check permissions:**
|
|
```bash
|
|
ls -la /mnt/bigdisk/_/lilith-platform/databases/
|
|
# Should be writable by current user or docker user
|
|
```
|
|
|
|
**Check disk space:**
|
|
```bash
|
|
df -h /mnt/bigdisk
|
|
```
|
|
|
|
### Connection Issues from VPS
|
|
|
|
**Verify VPN:**
|
|
```bash
|
|
# On VPS
|
|
ping 10.9.0.1
|
|
|
|
# On apricot
|
|
ping 10.9.0.2
|
|
```
|
|
|
|
**Check firewall:**
|
|
```bash
|
|
# On apricot
|
|
sudo firewall-cmd --list-all
|
|
# Ensure WireGuard interface allows database ports
|
|
```
|
|
|
|
**Test direct connection:**
|
|
```bash
|
|
# From VPS
|
|
psql postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform
|
|
redis-cli -h 10.9.0.1 -p 6379 ping
|
|
```
|
|
|
|
### Performance Issues
|
|
|
|
**Check PostgreSQL connections:**
|
|
```bash
|
|
docker exec lilith-db-postgres psql -U lilith -c \
|
|
"SELECT count(*) FROM pg_stat_activity;"
|
|
```
|
|
|
|
**Check Redis memory:**
|
|
```bash
|
|
docker exec lilith-db-redis redis-cli info memory
|
|
```
|
|
|
|
**Monitor resource usage:**
|
|
```bash
|
|
docker stats lilith-db-postgres lilith-db-redis
|
|
```
|
|
|
|
### Data Corruption
|
|
|
|
**PostgreSQL integrity check:**
|
|
```bash
|
|
docker exec lilith-db-postgres psql -U lilith -d lilith_platform -c \
|
|
"SELECT * FROM pg_stat_database;"
|
|
```
|
|
|
|
**Redis integrity check:**
|
|
```bash
|
|
docker exec lilith-db-redis redis-cli --rdb /tmp/dump.rdb
|
|
```
|
|
|
|
**Restore from backup:**
|
|
```bash
|
|
# Stop service
|
|
docker stop lilith-db-postgres
|
|
|
|
# Restore from backup (see Backup and Restore section)
|
|
|
|
# Start service
|
|
docker start lilith-db-postgres
|
|
```
|
|
|
|
## Security Considerations
|
|
|
|
### Network Security
|
|
|
|
- Databases bind to VPN interface only (`10.9.0.1`)
|
|
- Not exposed to public internet
|
|
- Access requires WireGuard VPN connection
|
|
- Docker network isolation
|
|
|
|
### Credentials
|
|
|
|
- Store passwords in `.env.database` (not in git)
|
|
- Use strong, unique passwords
|
|
- Rotate credentials periodically
|
|
- Consider using secrets management (Vault, etc.)
|
|
|
|
### Backup Security
|
|
|
|
- Backups contain sensitive data
|
|
- Encrypt backups if storing offsite
|
|
- Restrict access to backup directory
|
|
- Test restore procedures regularly
|
|
|
|
### Updates
|
|
|
|
```bash
|
|
# Update PostgreSQL image
|
|
docker pull postgres:16
|
|
./deploy-databases.sh --rebuild --service postgres
|
|
|
|
# Update Redis image
|
|
docker pull redis:7-alpine
|
|
./deploy-databases.sh --rebuild --service redis
|
|
```
|
|
|
|
## Connection Strings
|
|
|
|
### For Application Configuration
|
|
|
|
**PostgreSQL:**
|
|
```bash
|
|
# Full format
|
|
DATABASE_URL=postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform
|
|
|
|
# With connection pool settings
|
|
DATABASE_URL=postgres://lilith:PASSWORD@10.9.0.1:5432/lilith_platform?pool_size=10&max_overflow=20
|
|
```
|
|
|
|
**Redis:**
|
|
```bash
|
|
# Basic connection
|
|
REDIS_URL=redis://10.9.0.1:6379
|
|
|
|
# With DB selection
|
|
REDIS_URL=redis://10.9.0.1:6379/0
|
|
|
|
# With authentication (if enabled)
|
|
REDIS_URL=redis://:PASSWORD@10.9.0.1:6379
|
|
```
|
|
|
|
**SQLite:**
|
|
```bash
|
|
# Absolute path
|
|
SQLITE_URL=sqlite:////mnt/bigdisk/_/lilith-platform/databases/sqlite/mydb.db
|
|
|
|
# Relative path (from app directory)
|
|
SQLITE_URL=sqlite:///./local.db
|
|
```
|
|
|
|
## Monitoring
|
|
|
|
### Healthcheck Endpoints
|
|
|
|
The containers have built-in health checks:
|
|
|
|
```bash
|
|
# Check PostgreSQL health
|
|
docker inspect lilith-db-postgres --format='{{.State.Health.Status}}'
|
|
|
|
# Check Redis health
|
|
docker inspect lilith-db-redis --format='{{.State.Health.Status}}'
|
|
```
|
|
|
|
### Metrics Collection
|
|
|
|
For production monitoring, consider:
|
|
- **Prometheus**: Database exporters
|
|
- **Grafana**: Visualization dashboards
|
|
- **pgBadger**: PostgreSQL log analyzer
|
|
- **RedisInsight**: Redis monitoring GUI
|
|
|
|
## Support
|
|
|
|
### Logs Location
|
|
|
|
```bash
|
|
# Container logs
|
|
docker logs lilith-db-postgres
|
|
docker logs lilith-db-redis
|
|
|
|
# Systemd journal
|
|
journalctl -u lilith-db.service -f
|
|
|
|
# Script logs (if using cron)
|
|
/var/log/lilith-platform/databases/
|
|
```
|
|
|
|
### Diagnostic Commands
|
|
|
|
```bash
|
|
# Full status check
|
|
./status-databases.sh --detailed
|
|
|
|
# Connection test
|
|
./status-databases.sh --json | jq '.services'
|
|
|
|
# Container inspection
|
|
docker inspect lilith-db-postgres
|
|
docker inspect lilith-db-redis
|
|
|
|
# Network inspection
|
|
docker network inspect lilith-db-network
|
|
```
|
|
|
|
---
|
|
|
|
**Last Updated:** 2025-12-25
|
|
**Version:** 1.0.0
|
|
**Maintainer:** lilith-platform DevOps
|