platform-codebase/infrastructure/nginx/README.md

Nginx Configuration - 7-Domain Architecture

Production-grade nginx configuration for lilith-platform's multi-domain SEO strategy.

Architecture Overview

Domain Portfolio

Domain	Purpose	SEO Strategy	Primary App
lilith.io	Primary brand domain	Tech/SaaS credibility, startup positioning	Portal
getlilith.com	Acquisition funnel	"Get [product]" search pattern capture	Portal (conversion-optimized)
lilith.store	E-commerce funnel	Shopping/product intent traffic	Storefront
lilithapps.com	Product suite platform	Multi-app platform positioning	Portal
lilith.fan	Creator platform	Creator discovery and fan engagement	Portal, Fan-Club
lilith.toys	Physical products	Creator merchandise storefronts	Storefront
trustedmeet.com	Discovery/booking marketplace	Service provider listings	SEO App, Marketplace
nasty.sh	Infrastructure domain	Backend services (image generation, ML)	ML Services

---

File Structure

infrastructure/nginx/
├── nginx.conf                          # Main configuration file
├── snippets/
│   ├── ssl-params.conf                 # SSL/TLS settings
│   ├── security-headers.conf           # Security headers (HSTS, CSP, etc.)
│   └── proxy-params.conf               # Reverse proxy settings
└── conf.d/
    ├── 0-rate-limiting.conf            # Rate limiting zones
    ├── 1-upstreams.conf                # Backend service definitions
    └── 7-domain-routing.conf           # Domain-specific routing rules

Load Order: Files in conf.d/ are loaded alphabetically (hence the numeric prefixes).

---

Prerequisites

1. System Requirements

• Nginx: 1.18.0+ (with HTTP/2 and SSL modules)
• OS: Linux (Ubuntu 20.04+ or RHEL 8+)
• SSL: Let's Encrypt certbot installed

2. Required Modules

Check if modules are compiled in:

nginx -V 2>&1 | grep -o with-http_ssl_module
nginx -V 2>&1 | grep -o with-http_v2_module

If missing, install nginx with modules:

# Ubuntu/Debian
sudo apt install nginx-full

# RHEL/CentOS
sudo yum install nginx

3. Directory Structure

Create required directories:

# Cache directories
sudo mkdir -p /var/cache/nginx/{seo,static}
sudo chown -R nginx:nginx /var/cache/nginx

# SSL directory for default certificate
sudo mkdir -p /etc/nginx/ssl

# Certbot webroot
sudo mkdir -p /var/www/certbot

# Log directories (should already exist)
sudo mkdir -p /var/log/nginx

---

Installation Steps

Step 1: Generate DH Parameters (One-Time Setup)

Generate Diffie-Hellman parameters for DHE cipher suites:

sudo openssl dhparam -out /etc/nginx/dhparam.pem 2048

Expected time: 2-5 minutes (depending on CPU)

Uncomment in snippets/ssl-params.conf:

ssl_dhparam /etc/nginx/dhparam.pem;

Step 2: Generate Default SSL Certificate

Create self-signed certificate for the default server block:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout /etc/nginx/ssl/default.key \
  -out /etc/nginx/ssl/default.crt \
  -subj "/C=US/ST=State/L=City/O=Organization/CN=default"

Step 3: Install Configuration Files

Copy nginx configuration files to system directories:

# Main config
sudo cp nginx.conf /etc/nginx/nginx.conf

# Snippets
sudo mkdir -p /etc/nginx/snippets
sudo cp snippets/*.conf /etc/nginx/snippets/

# Domain configs
sudo cp conf.d/*.conf /etc/nginx/conf.d/

Step 4: Obtain SSL Certificates (Let's Encrypt)

For each domain, obtain SSL certificates:

# Example for lilith.io (repeat for each domain)
sudo certbot certonly --webroot \
  -w /var/www/certbot \
  -d lilith.io -d www.lilith.io

# Repeat for other domains:
# - getlilith.com, www.getlilith.com
# - lilith.store, www.lilith.store, *.lilith.store (wildcard requires DNS challenge)
# - lilithapps.com, www.lilithapps.com, *.lilithapps.com
# - lilith.fan, www.lilith.fan, *.lilith.fan
# - lilith.toys, www.lilith.toys, *.lilith.toys
# - trustedmeet.com, www.trustedmeet.com, *.trustedmeet.com
# - nasty.sh, *.nasty.sh

Wildcard certificates (for *.lilith.store, etc.):

sudo certbot certonly --dns-cloudflare \
  --dns-cloudflare-credentials ~/.secrets/certbot/cloudflare.ini \
  -d lilith.store -d *.lilith.store

Cloudflare DNS credentials (~/.secrets/certbot/cloudflare.ini):

dns_cloudflare_api_token = YOUR_CLOUDFLARE_API_TOKEN

Step 5: Configure Backend Services

Update upstream backend addresses in conf.d/1-upstreams.conf:

# Example: If API runs on different host/port
upstream api_backend {
    server 192.168.1.100:3001 max_fails=3 fail_timeout=30s;
    keepalive 64;
}

Default values (assumes services on localhost):

• Portal: 127.0.0.1:3100
• API: 127.0.0.1:3001
• Storefront: 127.0.0.1:3200
• SEO App: 127.0.0.1:3800
• ML Services: 127.0.0.1:4000

Step 6: Test Configuration

Validate nginx configuration syntax:

sudo nginx -t

Expected output:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Step 7: Enable 7-Domain Configuration

In nginx.conf, uncomment the modular config includes:

# Uncomment these lines:
include /etc/nginx/conf.d/0-rate-limiting.conf;
include /etc/nginx/conf.d/1-upstreams.conf;
include /etc/nginx/conf.d/7-domain-routing.conf;

Step 8: Reload Nginx

Apply the new configuration:

# Test again
sudo nginx -t

# Reload (zero downtime)
sudo systemctl reload nginx

# Or full restart if major changes
sudo systemctl restart nginx

Step 9: Verify Domains

Test each domain:

# Check HTTP → HTTPS redirect
curl -I http://lilith.io

# Check HTTPS response
curl -I https://lilith.io

# Check SSL certificate
echo | openssl s_client -servername lilith.io -connect lilith.io:443 2>/dev/null | openssl x509 -noout -dates

Expected:

• HTTP returns 301 Moved Permanently with Location: https://
• HTTPS returns 200 OK with security headers
• Certificate is valid and matches domain

---

DNS Configuration

Required DNS Records

For each domain, configure DNS records:

Example for lilith.io:

Type    Name    Value               TTL
A       @       YOUR_SERVER_IP      300
A       www     YOUR_SERVER_IP      300
AAAA    @       YOUR_IPV6_IP        300
AAAA    www     YOUR_IPV6_IP        300

Wildcard subdomains (for *.lilith.store, *.lilithapps.com, etc.):

Type    Name    Value               TTL
A       *       YOUR_SERVER_IP      300

Cloudflare Configuration (Recommended)

If using Cloudflare:

1. Proxy Status: Orange cloud (proxied) for DDoS protection
2. SSL/TLS Mode: Full (Strict) - requires valid SSL on origin
3. Always Use HTTPS: Enabled
4. Automatic HTTPS Rewrites: Enabled
5. HTTP Strict Transport Security (HSTS): Enabled
   • Max Age: 6 months
   • Include subdomains: Yes
   • Preload: Yes (after testing)

---

Security Hardening

1. Firewall Rules

Allow only HTTP/HTTPS traffic:

# UFW (Ubuntu)
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

# firewalld (RHEL)
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

2. SELinux (RHEL/CentOS)

Allow nginx to connect to backend services:

sudo setsebool -P httpd_can_network_connect 1

3. Fail2Ban Integration

Protect against brute force attacks:

Install fail2ban:

sudo apt install fail2ban  # Ubuntu
sudo yum install fail2ban  # RHEL

Create nginx jail (/etc/fail2ban/jail.d/nginx.conf):

[nginx-limit-req]
enabled = true
filter = nginx-limit-req
logpath = /var/log/nginx/error.log
maxretry = 5
findtime = 60
bantime = 3600

Create filter (/etc/fail2ban/filter.d/nginx-limit-req.conf):

[Definition]
failregex = limiting requests, excess: .* by zone .*, client: <HOST>
ignoreregex =

Restart fail2ban:

sudo systemctl restart fail2ban
sudo fail2ban-client status nginx-limit-req

---

Monitoring & Maintenance

1. Log Rotation

Configure logrotate (/etc/logrotate.d/nginx):

/var/log/nginx/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 0640 nginx adm
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 `cat /var/run/nginx.pid`
        fi
    endscript
}

2. SSL Certificate Renewal

Certbot auto-renewal should be configured by default. Verify:

# Test renewal (dry run)
sudo certbot renew --dry-run

# Check renewal timer (systemd)
sudo systemctl status certbot.timer

# Manual renewal
sudo certbot renew

After renewal, reload nginx:

sudo systemctl reload nginx

3. Performance Monitoring

Enable nginx status module (add to nginx.conf):

server {
    listen 127.0.0.1:8080;
    server_name localhost;

    location /nginx_status {
        stub_status;
        access_log off;
    }
}

Query status:

curl http://127.0.0.1:8080/nginx_status

Sample output:

Active connections: 42
server accepts handled requests
 12345 12345 54321
Reading: 0 Writing: 3 Waiting: 39

4. Access Log Analysis

Analyze traffic patterns:

# Top requesting IPs
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# Top requested URLs
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# HTTP status codes
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

Use GoAccess for real-time dashboard:

sudo apt install goaccess
goaccess /var/log/nginx/access.log -o report.html --log-format=COMBINED

---

Performance Tuning

1. Kernel Parameters

Optimize network stack (/etc/sysctl.conf):

# TCP optimization
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

# File descriptor limits
fs.file-max = 2097152

# Apply changes
sudo sysctl -p

2. Nginx Worker Tuning

Adjust based on CPU cores:

# In nginx.conf
worker_processes auto;  # Auto-detect cores
worker_connections 4096;  # Increase for high traffic

Calculate max connections: worker_processes * worker_connections = total capacity

3. Connection Keepalive

Tune keepalive for backend connections:

# In upstream blocks
keepalive 64;  # Maintain 64 persistent connections
keepalive_timeout 65;  # Keep connections alive for 65 seconds
keepalive_requests 100;  # Max requests per connection

---

Troubleshooting

Common Issues

1. 502 Bad Gateway

• Cause: Backend service is down or unreachable
• Fix: Check upstream backend status:

  curl http://127.0.0.1:3001/health  # Test API backend directly
  sudo systemctl status api-service  # Check service status
  

2. SSL Certificate Errors

• Cause: Certificate path is incorrect or expired
• Fix: Verify certificate paths in nginx config match certbot output:

  sudo certbot certificates
  

3. Rate Limiting Too Aggressive

• Cause: Legitimate users hitting rate limits
• Fix: Increase burst values in conf.d/7-domain-routing.conf:

  limit_req zone=api burst=50 nodelay;  # Increase burst from 20 to 50
  

4. WebSocket Connections Failing

• Cause: Missing connection upgrade header
• Fix: Verify map $http_upgrade $connection_upgrade is defined in nginx.conf

5. 413 Request Entity Too Large

• Cause: File upload exceeds client_max_body_size
• Fix: Increase limit in nginx.conf or specific location block:

  client_max_body_size 500M;  # Allow 500MB uploads
  

Debug Mode

Enable detailed error logging:

# In nginx.conf
error_log /var/log/nginx/error.log debug;

Warning: Debug mode generates massive logs. Disable after troubleshooting:

error_log /var/log/nginx/error.log warn;

---

Rollback Plan

If issues arise after deployment:

1. Quick Rollback

Restore previous configuration:

# Restore backup
sudo cp /etc/nginx/nginx.conf.bak /etc/nginx/nginx.conf

# Test and reload
sudo nginx -t && sudo systemctl reload nginx

2. Comment Out 7-Domain Config

In nginx.conf, comment out the new includes:

# include /etc/nginx/conf.d/0-rate-limiting.conf;
# include /etc/nginx/conf.d/1-upstreams.conf;
# include /etc/nginx/conf.d/7-domain-routing.conf;

Then reload:

sudo systemctl reload nginx

---

Production Checklist

Before deploying to production:

• All 8 SSL certificates obtained (7 domains + nasty.sh)
• DNS records configured for all domains
• DH parameters generated (/etc/nginx/dhparam.pem)
• Backend services running and health checks passing
• Nginx configuration tested (nginx -t)
• Firewall rules configured (ports 80, 443)
• Log rotation configured
• Certbot auto-renewal tested
• fail2ban configured and active
• Monitoring/alerting set up
• Backup plan documented
• Rollback tested in staging environment

---

Additional Resources

• Nginx Documentation: https://nginx.org/en/docs/
• Let's Encrypt Documentation: https://letsencrypt.org/docs/
• SSL Labs Test: https://www.ssllabs.com/ssltest/
• Mozilla SSL Configuration Generator: https://ssl-config.mozilla.org/

---

Last Updated: 2025-12-11 Maintained By: The Collective Version: 1.0.0 - 7-Domain Architecture