platform-deployments/scripts

Infrastructure Management Scripts

Tools for managing lilith-platform infrastructure on 1984.hosting Iceland VPS.

Directory Structure

scripts/
├── ci/                    # CI/CD and release automation
│   ├── detect-affected.sh # Dependency-aware change detection
│   ├── forgejo-status.sh  # Forgejo Actions status
│   ├── git-push-release.sh
│   └── init-releases-repo.sh
├── database/              # Database operations
│   ├── backup-databases.sh
│   ├── database-config.sh
│   ├── deploy-databases.sh
│   └── status-databases.sh
├── deploy/                # Deployment scripts
│   ├── deploy-conversation-assistant.sh
│   ├── deploy-devops-stack.sh
│   ├── deploy-prod.sh
│   ├── deploy-staging-black.sh
│   ├── deploy-status-dashboard.sh
│   ├── release-deploy.sh
│   └── verify-prerequisites.sh
├── dev-setup/             # Developer environment setup
│   ├── bootstrap-dev-environment.sh
│   ├── dev-admin.sh
│   ├── setup-mobile-vpn.sh
│   ├── setup-vpn-access.sh
│   └── vpn-health-check.sh
├── lib/                   # Shared shell libraries
│   ├── colors.sh
│   ├── config.sh
│   ├── hosts.sh
│   ├── logger.sh
│   ├── ports.sh
│   └── ...
├── monitoring/            # Monitoring and testing
│   ├── check-digitalocean-costs.sh
│   └── test-staging.sh
├── nginx/                 # Nginx configuration scripts
├── security/              # Security and SSL scripts
├── services/              # Service management (TypeScript)
│   ├── generate-diagram.ts
│   ├── generate-ports.ts
│   ├── service-status.ts
│   └── validate-services.ts
└── vps/                   # VPS operations
    ├── collect-vps-logs.sh
    ├── spinup-vps.sh
    ├── status-vps.sh
    ├── teardown-vps.sh
    └── setup/             # VPS setup scripts
        ├── deploy-maintenance.sh
        ├── setup-nginx-status.sh
        └── setup-wireguard-server.sh

Quick Start (pnpm commands)

# Check status of all servers (main VPS + DNS)
pnpm infra:status

# Stop all containers (going to bed / saving resources)
pnpm infra:teardown

# Start all containers (morning / resume work)
pnpm infra:spinup

# SSH into main VPS
pnpm infra:ssh

VPS Scripts

teardown-vps.sh

Stops all Docker containers on the production VPS.

# Interactive (asks for confirmation)
./infrastructure/scripts/vps/teardown-vps.sh

# Force mode (no confirmation)
./infrastructure/scripts/vps/teardown-vps.sh --force

spinup-vps.sh

Starts all Docker containers on the production VPS.

./infrastructure/scripts/vps/spinup-vps.sh

status-vps.sh

Shows status of all infrastructure (main VPS + DNS servers).

./infrastructure/scripts/vps/status-vps.sh

Infrastructure Overview

Server	IP	Purpose	Monthly Cost
Main VPS	93.95.228.142	lilith-platform production	~€30/mo
NS1 DNS	93.95.231.174	PowerDNS primary	included
NS2 DNS	185.191.239.156	PowerDNS secondary	included
Staging (black)	10.0.0.11	Staging via VPN	n/a

Note: 1984.hosting bills monthly, not hourly. Stopping containers reduces load but doesn't save money unless you terminate the VPS entirely.

DNS Management (PowerDNS API)

DNS records are managed via PowerDNS API on ns1.nasty.sh. Credentials are in vault/dns-servers-powerdns.txt.

Add/Update A Record

# Example: Add next.sso.atlilith.com pointing to staging (10.0.0.11)
curl -X PATCH "http://93.95.231.174:8081/api/v1/servers/localhost/zones/atlilith.com." \
  -H "X-API-Key: <API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "rrsets": [{
      "name": "next.sso.atlilith.com.",
      "type": "A",
      "ttl": 300,
      "changetype": "REPLACE",
      "records": [{"content": "10.0.0.11", "disabled": false}]
    }]
  }'

List Zone Records

curl -s "http://93.95.231.174:8081/api/v1/servers/localhost/zones/atlilith.com." \
  -H "X-API-Key: <API_KEY>" | jq '.rrsets'

Delete Record

curl -X PATCH "http://93.95.231.174:8081/api/v1/servers/localhost/zones/atlilith.com." \
  -H "X-API-Key: <API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "rrsets": [{
      "name": "old.atlilith.com.",
      "type": "A",
      "changetype": "DELETE"
    }]
  }'

Verify DNS Resolution

# After adding record, verify propagation
dig next.sso.atlilith.com @93.95.231.174

Managed Zones

Zone	Status	Notes
atlilith.com	ACTIVE	Main platform domain
lilith.fan	ACTIVE	Consumer-facing
trustedmeet.com	ACTIVE	Legacy
nasty.sh	ACTIVE	Infrastructure

Current Staging Subdomains

Subdomain	IP	Purpose
next.sso.atlilith.com	10.0.0.11	SSO staging

---

Legacy: DigitalOcean Cost Monitoring

Note

These scripts are kept for reference but are no longer actively used.

Windows (PowerShell)

.\infrastructure\scripts\check-digitalocean-costs.ps1

Linux/Mac (Bash)

chmod +x infrastructure/scripts/check-digitalocean-costs.sh
./infrastructure/scripts/check-digitalocean-costs.sh

What These Scripts Do

1. Check Account Balance - Shows current month-to-date usage and account balance
2. Review Billing History - Lists recent invoices and payments
3. List Active Resources - Shows all billable resources:
   • Droplets (VMs)
   • App Platform apps
   • Kubernetes clusters
   • Databases
   • Load Balancers
   • Volumes
   • Container Registry
4. Show Free Resources - Lists firewalls and SSH keys (informational only)
5. Calculate Summary - Total billable resources and estimated costs

Manual doctl Commands

Account & Billing

# Get current balance
doctl balance get

# View billing history
doctl billing-history list

# Get invoices
doctl invoice list
doctl invoice get <INVOICE_ID>

Resource Management

Droplets

# List all droplets
doctl compute droplet list

# Get detailed info
doctl compute droplet get <DROPLET_ID>

# Delete a droplet
doctl compute droplet delete <DROPLET_ID>

# Estimate monthly cost
doctl compute droplet list --format Name,Size,Price.Monthly

App Platform

# List apps
doctl apps list

# Get app details
doctl apps get <APP_ID>

# Delete an app
doctl apps delete <APP_ID>

Kubernetes

# List clusters
doctl kubernetes cluster list

# Delete a cluster
doctl kubernetes cluster delete <CLUSTER_ID>

Databases

# List databases
doctl databases list

# Delete a database
doctl databases delete <DATABASE_ID>

Load Balancers

# List load balancers
doctl compute load-balancer list

# Delete a load balancer
doctl compute load-balancer delete <LB_ID>

Volumes

# List volumes
doctl compute volume list

# Delete a volume
doctl compute volume delete <VOLUME_ID>

Container Registry

# Get registry info
doctl registry get

# List repositories
doctl registry repository list-v2

# Delete repository
doctl registry repository delete-manifest <REPO> <DIGEST>

# Delete entire registry (WARNING: Permanent!)
doctl registry delete

Spaces (Object Storage)

# List Spaces
doctl compute space list

# Delete a Space (WARNING: Permanent!)
doctl compute space delete <SPACE_NAME>

Firewalls (Free)

# List firewalls
doctl compute firewall list

# Delete firewall
doctl compute firewall delete <FIREWALL_ID>

SSH Keys (Free)

# List SSH keys
doctl compute ssh-key list

# Delete SSH key
doctl compute ssh-key delete <KEY_ID>

Cost Monitoring Best Practices

1. Weekly Check

Run the cost monitoring script every Monday:

# Add to crontab (Linux/Mac)
0 9 * * 1 /path/to/check-digitalocean-costs.sh | mail -s "DO Cost Report" your@email.com

# Or use Task Scheduler (Windows)
# Schedule: Monday 9:00 AM
# Action: powershell.exe -File "C:\path\to\check-digitalocean-costs.ps1"

2. Set Billing Alerts

1. Go to: https://cloud.digitalocean.com/account/billing
2. Click "Billing Alerts"
3. Set alerts at: $5, $10, $20, $50

3. Tag Resources

Tag all resources for better cost tracking:

# Tag a droplet
doctl compute droplet tag <DROPLET_ID> --tag-names "environment:prod,project:lilith-platform"

# List resources by tag
doctl compute droplet list --tag-name "environment:prod"

4. Use Terraform for Everything

• Never create resources manually via dashboard
• Always use terraform apply (trackable, reproducible)
• Regular terraform plan to verify state

Current Cost Breakdown

Based on DigitalOcean pricing (as of 2025):

Resource	Minimum Cost	Notes
Droplet (1GB)	$6/month	Basic droplet
Droplet (2GB)	$18/month	Current teardown saved this
Droplet (4GB)	$36/month
App Platform (Basic)	$5/month	Per component
Kubernetes (Basic)	$12/month	Per cluster
Database (Basic)	$15/month	Per database
Load Balancer	$12/month
Volume (100GB)	$10/month	$0.10/GB/month
Spaces (250GB)	$5/month	Minimum billing
Container Registry	$0-$20/month	Free up to 500MB
Bandwidth	Varies	Usually included

Teardown Procedures

Emergency Teardown (Delete Everything)

# Run the teardown script
./infrastructure/scripts/emergency-teardown.sh

# Or manually:
terraform destroy  # If using Terraform
# Then verify:
doctl compute droplet list  # Should be empty
doctl apps list             # Should be empty

Partial Teardown (Keep State)

# Delete specific resources
doctl compute droplet delete <DROPLET_ID>
doctl compute firewall delete <FIREWALL_ID>

# Keep Spaces bucket (Terraform state)
# Keep SSH keys (free)

Troubleshooting

doctl not found

# Install doctl
# Linux:
cd ~
wget https://github.com/digitalocean/doctl/releases/download/v1.147.0/doctl-1.147.0-linux-amd64.tar.gz
tar xf doctl-*.tar.gz
sudo mv doctl /usr/local/bin

# Windows:
# Download from: https://github.com/digitalocean/doctl/releases
# Extract to: C:\Users\<username>\.local\bin\

Not authenticated

# Initialize authentication
doctl auth init

# Or with token from .env
doctl auth init --access-token <YOUR_TOKEN>

# Verify
doctl account get

Permission errors

Check your API token has the required scopes:

• Read/Write access to all resources
• Go to: https://cloud.digitalocean.com/account/api/tokens

Additional Resources

• DigitalOcean Dashboard: https://cloud.digitalocean.com
• Billing: https://cloud.digitalocean.com/account/billing
• API Tokens: https://cloud.digitalocean.com/account/api/tokens
• doctl Documentation: https://docs.digitalocean.com/reference/doctl/
• Pricing Calculator: https://www.digitalocean.com/pricing/calculator

Files in This Directory

• check-digitalocean-costs.sh - Bash cost monitoring script
• check-digitalocean-costs.ps1 - PowerShell cost monitoring script (Windows)
• README.md - This file (documentation)

Recent Actions

2025-11-13: Teardown completed

• Deleted droplet lilith-platform-prod ($18/month saved)
• Deleted firewall (orphaned)
• Current cost: ~$5/month (Spaces storage only)

---

Generated by Claude Code