platform-codebase/infrastructure/docker/README.databases.md

Database Services Deployment Guide

This document provides instructions for deploying and managing the Lilith Platform database services on apricot.

Overview

Host: apricot (10.9.0.1 on VPN) Location: /var/home/lilith/Code/@applications/@lilith/lilith-platform/codebase/infrastructure/docker/ Data Directory: /mnt/bigdisk/_/lilith-platform/databases/

Services

Service	Port	Purpose	Data Location
PostgreSQL 16 + TimescaleDB	5432	Primary database	/mnt/bigdisk/_/lilith-platform/databases/postgresql
Redis 7	6379	Cache, sessions, queues	/mnt/bigdisk/_/lilith-platform/databases/redis
Meilisearch	7700	Full-text search	/mnt/bigdisk/_/lilith-platform/databases/meilisearch

---

Initial Setup

1. Create Data Directories

# Create base directory structure
sudo mkdir -p /mnt/bigdisk/_/lilith-platform/databases/{postgresql,redis,meilisearch,meilisearch-snapshots}

# Set ownership (PostgreSQL requires UID 999)
sudo chown -R 999:999 /mnt/bigdisk/_/lilith-platform/databases/postgresql

# Redis and Meilisearch use default user (UID 1000 or current user)
sudo chown -R $USER:$USER /mnt/bigdisk/_/lilith-platform/databases/redis
sudo chown -R $USER:$USER /mnt/bigdisk/_/lilith-platform/databases/meilisearch
sudo chown -R $USER:$USER /mnt/bigdisk/_/lilith-platform/databases/meilisearch-snapshots

# Set permissions
sudo chmod 750 /mnt/bigdisk/_/lilith-platform/databases/postgresql
sudo chmod 755 /mnt/bigdisk/_/lilith-platform/databases/redis
sudo chmod 755 /mnt/bigdisk/_/lilith-platform/databases/meilisearch

2. Create Environment File

# Copy example environment file
cp .env.databases.example .env.databases

# Edit with secure credentials
nano .env.databases

# Set secure file permissions
chmod 600 .env.databases

Required changes in .env.databases:

# Generate strong passwords (32+ characters)
POSTGRES_PASSWORD=$(openssl rand -base64 32)
REDIS_PASSWORD=$(openssl rand -base64 32)
MEILI_MASTER_KEY=$(openssl rand -base64 32)

# Update .env.databases with generated values

3. (Optional) SSL Certificate Generation

If you want SSL-encrypted PostgreSQL connections:

# Generate self-signed certificate
openssl req -new -x509 -days 365 -nodes -text \
  -out postgresql/ssl/server.crt \
  -keyout postgresql/ssl/server.key \
  -subj "/CN=apricot.vpn"

# Set permissions
chmod 600 postgresql/ssl/server.key
chmod 644 postgresql/ssl/server.crt

# Update ownership for PostgreSQL user
sudo chown 999:999 postgresql/ssl/server.{crt,key}

# Uncomment SSL settings in .env.databases

---

Deployment

Start All Services

# Navigate to docker directory
cd /var/home/lilith/Code/@applications/@lilith/lilith-platform/codebase/infrastructure/docker

# Start all database services
docker-compose -f docker-compose.databases.yml --env-file .env.databases up -d

# Check service status
docker-compose -f docker-compose.databases.yml ps

# View logs
docker-compose -f docker-compose.databases.yml logs -f

Start Individual Services

# Start only PostgreSQL
docker-compose -f docker-compose.databases.yml --env-file .env.databases up -d postgres

# Start only Redis
docker-compose -f docker-compose.databases.yml --env-file .env.databases up -d redis

# Start only Meilisearch
docker-compose -f docker-compose.databases.yml --env-file .env.databases up -d meilisearch

Stop Services

# Stop all services (preserves data)
docker-compose -f docker-compose.databases.yml down

# Stop and remove volumes (WARNING: destroys data)
docker-compose -f docker-compose.databases.yml down -v

---

Verification

Check Service Health

# Check all services
docker-compose -f docker-compose.databases.yml ps

# Expected output: all services "healthy"

Test PostgreSQL Connection

# From apricot (localhost)
docker-compose -f docker-compose.databases.yml exec postgres psql -U postgres -d lilith_platform

# From VPN (remote)
psql -h 10.9.0.1 -U postgres -d lilith_platform

# List extensions
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "\dx"

# Expected: uuid-ossp, pg_trgm, postgis, timescaledb, pg_stat_statements, pgcrypto

Test Redis Connection

# From apricot (localhost)
docker-compose -f docker-compose.databases.yml exec redis redis-cli -a "$REDIS_PASSWORD" ping

# From VPN (remote)
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" ping

# Expected: PONG

# Check Redis info
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO server

Test Meilisearch Connection

# From apricot (localhost)
curl http://localhost:7700/health

# From VPN (remote)
curl http://10.9.0.1:7700/health

# Expected: {"status":"available"}

# Check version (requires master key)
curl -H "Authorization: Bearer $MEILI_MASTER_KEY" http://10.9.0.1:7700/version

---

Monitoring

View Logs

# All services
docker-compose -f docker-compose.databases.yml logs -f

# Specific service
docker-compose -f docker-compose.databases.yml logs -f postgres
docker-compose -f docker-compose.databases.yml logs -f redis
docker-compose -f docker-compose.databases.yml logs -f meilisearch

# Last 100 lines
docker-compose -f docker-compose.databases.yml logs --tail=100 postgres

Resource Usage

# Docker stats
docker stats

# Disk usage
docker-compose -f docker-compose.databases.yml exec postgres du -sh /var/lib/postgresql/data
docker-compose -f docker-compose.databases.yml exec redis du -sh /data
docker-compose -f docker-compose.databases.yml exec meilisearch du -sh /meili_data

# Check available disk space on /mnt/bigdisk
df -h /mnt/bigdisk

PostgreSQL Monitoring

# Active connections
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "SELECT count(*) FROM pg_stat_activity;"

# Database size
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "SELECT pg_size_pretty(pg_database_size('lilith_platform'));"

# Slow queries (from pg_stat_statements)
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "
  SELECT query, calls, mean_exec_time, total_exec_time
  FROM pg_stat_statements
  ORDER BY mean_exec_time DESC
  LIMIT 10;
"

# Table sizes
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "
  SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
  FROM pg_tables
  WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
  ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
  LIMIT 10;
"

Redis Monitoring

# Memory usage
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO memory

# Stats
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO stats

# Keyspace
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO keyspace

# Slow log
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" SLOWLOG GET 10

Meilisearch Monitoring

# Stats
curl -H "Authorization: Bearer $MEILI_MASTER_KEY" http://10.9.0.1:7700/stats

# Indexes
curl -H "Authorization: Bearer $MEILI_MASTER_KEY" http://10.9.0.1:7700/indexes

---

Backup

PostgreSQL Backup

# Full database dump
docker-compose -f docker-compose.databases.yml exec postgres pg_dump -U postgres lilith_platform > backup_$(date +%Y%m%d).sql

# Compressed backup
docker-compose -f docker-compose.databases.yml exec postgres pg_dump -U postgres lilith_platform | gzip > backup_$(date +%Y%m%d).sql.gz

# Backup all databases
docker-compose -f docker-compose.databases.yml exec postgres pg_dumpall -U postgres | gzip > backup_all_$(date +%Y%m%d).sql.gz

# Automated backup script (add to cron)
# 0 2 * * * /path/to/backup-postgres.sh

Redis Backup

# Trigger manual save
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" BGSAVE

# AOF file is automatically persisted
# Copy AOF file for backup
cp /mnt/bigdisk/_/lilith-platform/databases/redis/appendonly.aof backup_redis_$(date +%Y%m%d).aof

Meilisearch Backup

# Create snapshot via API
curl -X POST -H "Authorization: Bearer $MEILI_MASTER_KEY" http://10.9.0.1:7700/snapshots

# Snapshots stored in: /mnt/bigdisk/_/lilith-platform/databases/meilisearch-snapshots

# Or copy data directory
tar -czf backup_meilisearch_$(date +%Y%m%d).tar.gz /mnt/bigdisk/_/lilith-platform/databases/meilisearch

---

Restore

PostgreSQL Restore

# Restore from dump
docker-compose -f docker-compose.databases.yml exec -T postgres psql -U postgres lilith_platform < backup.sql

# Restore compressed dump
gunzip -c backup.sql.gz | docker-compose -f docker-compose.databases.yml exec -T postgres psql -U postgres lilith_platform

# Restore all databases
gunzip -c backup_all.sql.gz | docker-compose -f docker-compose.databases.yml exec -T postgres psql -U postgres

Redis Restore

# Stop Redis
docker-compose -f docker-compose.databases.yml stop redis

# Replace AOF file
cp backup_redis.aof /mnt/bigdisk/_/lilith-platform/databases/redis/appendonly.aof

# Start Redis
docker-compose -f docker-compose.databases.yml start redis

Meilisearch Restore

# Stop Meilisearch
docker-compose -f docker-compose.databases.yml stop meilisearch

# Restore data directory
rm -rf /mnt/bigdisk/_/lilith-platform/databases/meilisearch/*
tar -xzf backup_meilisearch.tar.gz -C /mnt/bigdisk/_/lilith-platform/databases/meilisearch

# Start Meilisearch
docker-compose -f docker-compose.databases.yml start meilisearch

---

Troubleshooting

PostgreSQL Issues

Cannot connect:

# Check if service is running
docker-compose -f docker-compose.databases.yml ps postgres

# Check logs
docker-compose -f docker-compose.databases.yml logs postgres

# Verify pg_hba.conf allows your IP
docker-compose -f docker-compose.databases.yml exec postgres cat /etc/postgresql/pg_hba.conf

Performance issues:

# Check active queries
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "SELECT * FROM pg_stat_activity WHERE state = 'active';"

# Check locks
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "SELECT * FROM pg_locks WHERE NOT granted;"

# Run VACUUM ANALYZE
psql -h 10.9.0.1 -U postgres -d lilith_platform -c "VACUUM ANALYZE;"

Redis Issues

Cannot connect:

# Check if service is running
docker-compose -f docker-compose.databases.yml ps redis

# Check logs
docker-compose -f docker-compose.databases.yml logs redis

# Test connection
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" ping

Memory issues:

# Check memory usage
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO memory

# Check eviction stats
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" INFO stats | grep evicted

# Manually flush if needed (WARNING: deletes all data)
redis-cli -h 10.9.0.1 -a "$REDIS_PASSWORD" FLUSHALL  # Command disabled in production

Meilisearch Issues

Cannot connect:

# Check if service is running
docker-compose -f docker-compose.databases.yml ps meilisearch

# Check logs
docker-compose -f docker-compose.databases.yml logs meilisearch

# Test health endpoint
curl http://10.9.0.1:7700/health

---

Configuration Updates

Reload Configuration Without Restart

PostgreSQL:

# Reload config (most settings)
docker-compose -f docker-compose.databases.yml exec postgres pg_ctl reload

# Some settings require restart
docker-compose -f docker-compose.databases.yml restart postgres

Redis:

# Redis requires restart for config changes
docker-compose -f docker-compose.databases.yml restart redis

Meilisearch:

# Meilisearch requires restart for config changes
docker-compose -f docker-compose.databases.yml restart meilisearch

---

Security Checklist

• Strong passwords set in .env.databases
• File permissions: chmod 600 .env.databases
• Data directory permissions properly set
• PostgreSQL authentication: scram-sha-256
• Redis password authentication enabled
• Meilisearch master key set
• VPN-only access (no public internet exposure)
• SSL enabled for PostgreSQL (optional)
• Regular backups configured
• Monitoring alerts configured

---

Connection Strings

For application configuration:

PostgreSQL

# From VPN
DATABASE_URL=postgresql://postgres:PASSWORD@10.9.0.1:5432/lilith_platform

# From apricot (localhost)
DATABASE_URL=postgresql://postgres:PASSWORD@localhost:5432/lilith_platform

Redis

# From VPN
REDIS_URL=redis://:PASSWORD@10.9.0.1:6379

# From apricot (localhost)
REDIS_URL=redis://:PASSWORD@localhost:6379

Meilisearch

# From VPN
MEILI_URL=http://10.9.0.1:7700
MEILI_MASTER_KEY=YOUR_MASTER_KEY

# From apricot (localhost)
MEILI_URL=http://localhost:7700
MEILI_MASTER_KEY=YOUR_MASTER_KEY

---

Additional Resources

• PostgreSQL Documentation
• TimescaleDB Documentation
• Redis Documentation
• Meilisearch Documentation

---

Last Updated: 2025-12-25 Maintained by: Lilith Platform Team