lilith/platform-codebase
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
platform-codebase/infrastructure/docker/README.databases.md
Quinn Ftw b5fe73edd0 feat(infra): database stack, reconciliation, and VPS setup scripts 
- Add PostgreSQL + Redis deployment stack
- Add reconciliation framework for fleet management
- Add VPS setup scripts (nginx, wireguard)
- Add dev environment bootstrap scripts
- Update service-registry and systemd configs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-26 00:37:52 -08:00

525 lines
13 KiB
Markdown
Raw Blame History

# Database Services Deployment Guide
This document provides instructions for deploying and managing the Lilith Platform database services on apricot.
## Overview
**Host:** apricot (10.9.0.1 on VPN)
**Location:** `/var/home/lilith/Code/@applications/@lilith/lilith-platform/codebase/infrastructure/docker/`
**Data Directory:** `/mnt/bigdisk/_/lilith-platform/databases/`
### Services
| Service | Port | Purpose | Data Location |
|---------|------|---------|---------------|
| PostgreSQL 16 + TimescaleDB | 5432 | Primary database | `/mnt/bigdisk/_/lilith-platform/databases/postgresql` |
| Redis 7 | 6379 | Cache, sessions, queues | `/mnt/bigdisk/_/lilith-platform/databases/redis` |
| Meilisearch | 7700 | Full-text search | `/mnt/bigdisk/_/lilith-platform/databases/meilisearch` |
---
## Initial Setup
### 1. Create Data Directories
```bash
# Create base directory structure
sudo mkdir -p /mnt/bigdisk/_/lilith-platform/databases/{postgresql,redis,meilisearch,meilisearch-snapshots}
# Set ownership (PostgreSQL requires UID 999)
sudo chown -R 999:999 /mnt/bigdisk/_/lilith-platform/databases/postgresql
# Redis and Meilisearch use default user (UID 1000 or current user)
sudo chown -R $USER:$USER /mnt/bigdisk/_/lilith-platform/databases/redis
sudo chown -R $USER:$USER /mnt/bigdisk/_/lilith-platform/databases/meilisearch
sudo chown -R $USER:$USER /mnt/bigdisk/_/lilith-platform/databases/meilisearch-snapshots
# Set permissions
sudo chmod 750 /mnt/bigdisk/_/lilith-platform/databases/postgresql
sudo chmod 755 /mnt/bigdisk/_/lilith-platform/databases/redis
sudo chmod 755 /mnt/bigdisk/_/lilith-platform/databases/meilisearch
```
### 2. Create Environment File
```bash
# Copy example environment file
cp .env.databases.example .env.databases
# Edit with secure credentials
nano .env.databases
# Set secure file permissions
chmod 600 .env.databases
```
**Required changes in `.env.databases`:**
```bash
# Generate strong passwords (32+ characters)
POSTGRES_PASSWORD=$(openssl rand -base64 32)
REDIS_PASSWORD=$(openssl rand -base64 32)
MEILI_MASTER_KEY=$(openssl rand -base64 32)
# Update .env.databases with generated values
```
### 3. (Optional) SSL Certificate Generation
If you want SSL-encrypted PostgreSQL connections:
```bash
# Generate self-signed certificate
openssl req -new -x509 -days 365 -nodes -text \
-out postgresql/ssl/server.crt \
-keyout postgresql/ssl/server.key \
-subj "/CN=apricot.vpn"
# Set permissions
chmod 600 postgresql/ssl/server.key
chmod 644 postgresql/ssl/server.crt
# Update ownership for PostgreSQL user
sudo chown 999:999 postgresql/ssl/server.{crt,key}
# Uncomment SSL settings in .env.databases
```
---
## Deployment
### Start All Services
```bash
# Navigate to docker directory
cd /var/home/lilith/Code/@applications/@lilith/lilith-platform/codebase/infrastructure/docker
# Start all database services
docker-compose -f docker-compose.databases.yml --env-file .env.databases up -d
# Check service status
docker-compose -f docker-compose.databases.yml ps
# View logs
docker-compose -f docker-compose.databases.yml logs -f
```
### Start Individual Services
```bash
# Start only PostgreSQL
docker-compose -f docker-compose.databases.yml --env-file .env.databases up -d postgres
# Start only Redis
docker-compose -f docker-compose.databases.yml --env-file .env.databases up -d redis
# Start only Meilisearch
docker-compose -f docker-compose.databases.yml --env-file .env.databases up -d meilisearch
```
### Stop Services
```bash
# Stop all services (preserves data)
docker-compose -f docker-compose.databases.yml down
# Stop and remove volumes (WARNING: destroys data)
docker-compose -f docker-compose.databases.yml down -v
```
---
## Verification
### Check Service Health
```bash
# Check all services
docker-compose -f docker-compose.databases.yml ps
# Expected output: all services "healthy"
```
### Test PostgreSQL Connection
```bash
# From apricot (localhost)
docker-compose -f docker-compose.databases.yml exec postgres psql -U postgres -d lilith_platform
# From VPN (remote)
psql -h 10.9.0.1 -U postgres -d lilith_platform
# List extensions
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "\dx"
# Expected: uuid-ossp, pg_trgm, postgis, timescaledb, pg_stat_statements, pgcrypto
```
### Test Redis Connection
```bash
# From apricot (localhost)
docker-compose -f docker-compose.databases.yml exec redis redis-cli -a "$REDIS_PASSWORD" ping
# From VPN (remote)
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" ping
# Expected: PONG
# Check Redis info
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO server
```
### Test Meilisearch Connection
```bash
# From apricot (localhost)
curl http://localhost:7700/health
# From VPN (remote)
curl http://10.9.0.1:7700/health
# Expected: {"status":"available"}
# Check version (requires master key)
curl -H "Authorization: Bearer $MEILI_MASTER_KEY" http://10.9.0.1:7700/version
```
---
## Monitoring
### View Logs
```bash
# All services
docker-compose -f docker-compose.databases.yml logs -f
# Specific service
docker-compose -f docker-compose.databases.yml logs -f postgres
docker-compose -f docker-compose.databases.yml logs -f redis
docker-compose -f docker-compose.databases.yml logs -f meilisearch
# Last 100 lines
docker-compose -f docker-compose.databases.yml logs --tail=100 postgres
```
### Resource Usage
```bash
# Docker stats
docker stats
# Disk usage
docker-compose -f docker-compose.databases.yml exec postgres du -sh /var/lib/postgresql/data
docker-compose -f docker-compose.databases.yml exec redis du -sh /data
docker-compose -f docker-compose.databases.yml exec meilisearch du -sh /meili_data
# Check available disk space on /mnt/bigdisk
df -h /mnt/bigdisk
```
### PostgreSQL Monitoring
```bash
# Active connections
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "SELECT count(*) FROM pg_stat_activity;"
# Database size
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "SELECT pg_size_pretty(pg_database_size('lilith_platform'));"
# Slow queries (from pg_stat_statements)
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "
SELECT query, calls, mean_exec_time, total_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;
"
# Table sizes
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "
SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 10;
"
```
### Redis Monitoring
```bash
# Memory usage
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO memory
# Stats
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO stats
# Keyspace
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO keyspace
# Slow log
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" SLOWLOG GET 10
```
### Meilisearch Monitoring
```bash
# Stats
curl -H "Authorization: Bearer $MEILI_MASTER_KEY" http://10.9.0.1:7700/stats
# Indexes
curl -H "Authorization: Bearer $MEILI_MASTER_KEY" http://10.9.0.1:7700/indexes
```
---
## Backup
### PostgreSQL Backup
```bash
# Full database dump
docker-compose -f docker-compose.databases.yml exec postgres pg_dump -U postgres lilith_platform > backup_$(date +%Y%m%d).sql
# Compressed backup
docker-compose -f docker-compose.databases.yml exec postgres pg_dump -U postgres lilith_platform | gzip > backup_$(date +%Y%m%d).sql.gz
# Backup all databases
docker-compose -f docker-compose.databases.yml exec postgres pg_dumpall -U postgres | gzip > backup_all_$(date +%Y%m%d).sql.gz
# Automated backup script (add to cron)
# 0 2 * * * /path/to/backup-postgres.sh
```
### Redis Backup
```bash
# Trigger manual save
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" BGSAVE
# AOF file is automatically persisted
# Copy AOF file for backup
cp /mnt/bigdisk/_/lilith-platform/databases/redis/appendonly.aof backup_redis_$(date +%Y%m%d).aof
```
### Meilisearch Backup
```bash
# Create snapshot via API
curl -X POST -H "Authorization: Bearer $MEILI_MASTER_KEY" http://10.9.0.1:7700/snapshots
# Snapshots stored in: /mnt/bigdisk/_/lilith-platform/databases/meilisearch-snapshots
# Or copy data directory
tar -czf backup_meilisearch_$(date +%Y%m%d).tar.gz /mnt/bigdisk/_/lilith-platform/databases/meilisearch
```
---
## Restore
### PostgreSQL Restore
```bash
# Restore from dump
docker-compose -f docker-compose.databases.yml exec -T postgres psql -U postgres lilith_platform < backup.sql
# Restore compressed dump
gunzip -c backup.sql.gz | docker-compose -f docker-compose.databases.yml exec -T postgres psql -U postgres lilith_platform
# Restore all databases
gunzip -c backup_all.sql.gz | docker-compose -f docker-compose.databases.yml exec -T postgres psql -U postgres
```
### Redis Restore
```bash
# Stop Redis
docker-compose -f docker-compose.databases.yml stop redis
# Replace AOF file
cp backup_redis.aof /mnt/bigdisk/_/lilith-platform/databases/redis/appendonly.aof
# Start Redis
docker-compose -f docker-compose.databases.yml start redis
```
### Meilisearch Restore
```bash
# Stop Meilisearch
docker-compose -f docker-compose.databases.yml stop meilisearch
# Restore data directory
rm -rf /mnt/bigdisk/_/lilith-platform/databases/meilisearch/*
tar -xzf backup_meilisearch.tar.gz -C /mnt/bigdisk/_/lilith-platform/databases/meilisearch
# Start Meilisearch
docker-compose -f docker-compose.databases.yml start meilisearch
```
---
## Troubleshooting
### PostgreSQL Issues
**Cannot connect:**
```bash
# Check if service is running
docker-compose -f docker-compose.databases.yml ps postgres
# Check logs
docker-compose -f docker-compose.databases.yml logs postgres
# Verify pg_hba.conf allows your IP
docker-compose -f docker-compose.databases.yml exec postgres cat /etc/postgresql/pg_hba.conf
```
**Performance issues:**
```bash
# Check active queries
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "SELECT * FROM pg_stat_activity WHERE state = 'active';"
# Check locks
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "SELECT * FROM pg_locks WHERE NOT granted;"
# Run VACUUM ANALYZE
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "VACUUM ANALYZE;"
```
### Redis Issues
**Cannot connect:**
```bash
# Check if service is running
docker-compose -f docker-compose.databases.yml ps redis
# Check logs
docker-compose -f docker-compose.databases.yml logs redis
# Test connection
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" ping
```
**Memory issues:**
```bash
# Check memory usage
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO memory
# Check eviction stats
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO stats | grep evicted
# Manually flush if needed (WARNING: deletes all data)
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" FLUSHALL # Command disabled in production
```
### Meilisearch Issues
**Cannot connect:**
```bash
# Check if service is running
docker-compose -f docker-compose.databases.yml ps meilisearch
# Check logs
docker-compose -f docker-compose.databases.yml logs meilisearch
# Test health endpoint
curl http://10.9.0.1:7700/health
```
---
## Configuration Updates
### Reload Configuration Without Restart
**PostgreSQL:**
```bash
# Reload config (most settings)
docker-compose -f docker-compose.databases.yml exec postgres pg_ctl reload
# Some settings require restart
docker-compose -f docker-compose.databases.yml restart postgres
```
**Redis:**
```bash
# Redis requires restart for config changes
docker-compose -f docker-compose.databases.yml restart redis
```
**Meilisearch:**
```bash
# Meilisearch requires restart for config changes
docker-compose -f docker-compose.databases.yml restart meilisearch
```
---
## Security Checklist
- [ ] Strong passwords set in `.env.databases`
- [ ] File permissions: `chmod 600 .env.databases`
- [ ] Data directory permissions properly set
- [ ] PostgreSQL authentication: scram-sha-256
- [ ] Redis password authentication enabled
- [ ] Meilisearch master key set
- [ ] VPN-only access (no public internet exposure)
- [ ] SSL enabled for PostgreSQL (optional)
- [ ] Regular backups configured
- [ ] Monitoring alerts configured
---
## Connection Strings
For application configuration:
### PostgreSQL
```bash
# From VPN
DATABASE_URL=postgresql://postgres:PASSWORD@10.9.0.1:5432/lilith_platform
# From apricot (localhost)
DATABASE_URL=postgresql://postgres:PASSWORD@localhost:5432/lilith_platform
```
### Redis
```bash
# From VPN
REDIS_URL=redis://:PASSWORD@10.9.0.1:6379
# From apricot (localhost)
REDIS_URL=redis://:PASSWORD@localhost:6379
```
### Meilisearch
```bash
# From VPN
MEILI_URL=http://10.9.0.1:7700
MEILI_MASTER_KEY=YOUR_MASTER_KEY
# From apricot (localhost)
MEILI_URL=http://localhost:7700
MEILI_MASTER_KEY=YOUR_MASTER_KEY
```
---
## Additional Resources
- [PostgreSQL Documentation](https://www.postgresql.org/docs/16/)
- [TimescaleDB Documentation](https://docs.timescale.com/)
- [Redis Documentation](https://redis.io/documentation)
- [Meilisearch Documentation](https://docs.meilisearch.com/)
---
**Last Updated:** 2025-12-25
**Maintained by:** Lilith Platform Team
Reference in a new issue View git blame Copy permalink