platform-deployments/DEVOPS_SETUP.md

DevOps Infrastructure Setup - Quick Reference

One-command setup: Fresh Ubuntu 24.04 → Complete DevOps Infrastructure

---

TL;DR

# Setup new devops host
cd deployments/provisioning
./setup-devops-host.sh 10.0.0.11

# Follow post-install steps shown by script

---

What Gets Installed

The script installs a complete DevOps infrastructure stack:

Service	URL	Purpose
Forgejo	http://forge.nasty.sh/	Git forge (GitHub alternative)
Verdaccio	http://npm.nasty.sh/	NPM cache/proxy
Forgejo Runner	(background)	CI/CD executor
Nginx	ports 80, 443, 2222	Reverse proxy + Git SSH
PostgreSQL 16	localhost:5432	Forgejo database

Auto-start on boot: All services managed by devops.service systemd unit

---

Prerequisites

• Fresh Ubuntu 24.04 (or Debian-based) host
• SSH access with sudo privileges
• 50GB+ disk space (for /bigdisk)
• SSH key (default: ~/.ssh/id_ed25519)

---

Usage

Full Setup

./deployments/provisioning/setup-devops-host.sh <target-host>

Example:

# Using IP
./deployments/provisioning/setup-devops-host.sh 10.0.0.11

# Using hostname
./deployments/provisioning/setup-devops-host.sh devops.example.com

# Custom SSH settings
DEVOPS_HOST_USER=ubuntu \
DEVOPS_HOST_SSH_KEY=~/.ssh/mykey \
./deployments/provisioning/setup-devops-host.sh 10.0.0.11

Pre-flight Check

./deployments/provisioning/setup-devops-host.sh 10.0.0.11 --check

Verifies:

• SSH connectivity
• Sudo access
• Disk space (50GB+)
• Port availability (80, 443, 2222, 3000, 4873, 5432)
• OS compatibility

Verify Existing Installation

./deployments/provisioning/setup-devops-host.sh 10.0.0.11 --verify

---

What the Script Does

1. Pre-flight checks:

   • SSH connectivity
   • Sudo access
   • Disk space
   • Port availability

2. System setup:

   • Install Docker + Docker Compose
   • Create /bigdisk/ directory structure
   • Generate secure secrets (.env file)

3. Deploy configs:

   • Forgejo docker-compose.yml
   • Nginx reverse proxy config
   • Verdaccio config

4. Install systemd service:

   • Copy devops.service to /etc/systemd/system/
   • Enable auto-start on boot
   • Start services

5. Verification:

   • Check container health
   • Test service endpoints
   • Display next steps

---

Post-Install Steps

1. Add /etc/hosts Entries (Your Workstation)

The script displays the exact command. Example:

echo "10.0.0.11 forge.nasty.sh npm.nasty.sh" | sudo tee -a /etc/hosts

2. Create Forgejo Admin User

1. Navigate to http://forge.nasty.sh/
2. Click "Register"
3. First user becomes admin
4. Complete setup wizard (accept defaults)

3. Generate NPM Token for Verdaccio

1. Forgejo → User Settings → Applications
2. Generate new token (name: "Verdaccio")
3. Copy token
4. Add to secrets on host:

ssh <host> "echo 'FORGEJO_NPM_TOKEN=<your-token>' >> /bigdisk/forgejo/.env"
ssh <host> "sudo systemctl restart devops"

4. Configure Workstation NPM

./tooling/scripts/dev-setup/configure-verdaccio-client.sh

This configures your ~/.npmrc to use Verdaccio.

5. (Optional) Configure Forgejo Runner

1. Forgejo Admin → Actions → Runners
2. Generate registration token
3. Runner auto-registers on next restart

---

Directory Structure on Target Host

/bigdisk/
├── forgejo/
│   ├── docker-compose.yml        # Stack definition
│   ├── nginx.conf                 # Reverse proxy config
│   ├── .env                       # Secrets (auto-generated)
│   └── data/                      # Forgejo data (Git repos, DB, etc.)
│       ├── gitea/                 # Git repositories
│       ├── postgres/              # PostgreSQL data
│       └── runner/                # CI/CD runner data
└── verdaccio/
    ├── config/
    │   ├── config.yaml            # Verdaccio configuration
    │   └── htpasswd               # User authentication
    └── storage/                   # NPM package cache
        └── @lilith/               # Cached @lilith/* packages

---

Management Commands

Check Status

ssh <host> "systemctl status devops"
ssh <host> "cd /bigdisk/forgejo && docker-compose ps"

View Logs

ssh <host> "journalctl -u devops -f"
ssh <host> "cd /bigdisk/forgejo && docker-compose logs -f"

Restart Services

ssh <host> "sudo systemctl restart devops"

Update Images

ssh <host> "cd /bigdisk/forgejo && docker-compose pull && sudo systemctl restart devops"

Check Health

curl http://forge.nasty.sh/
curl http://npm.nasty.sh/-/ping

---

Secrets Management

Secrets are auto-generated in /bigdisk/forgejo/.env:

# View secrets (on host)
cat /bigdisk/forgejo/.env

# Backup secrets
scp <host>:/bigdisk/forgejo/.env ./devops-secrets-$(date +%Y%m%d).env

IMPORTANT: Save the database password shown during setup!

---

Troubleshooting

Services Won't Start

# Check logs
ssh <host> "journalctl -u devops -n 100"

# Check container status
ssh <host> "cd /bigdisk/forgejo && docker-compose ps"

# Check specific container
ssh <host> "docker logs forgejo"
ssh <host> "docker logs verdaccio"

Port Already in Use

# Find what's using the port
ssh <host> "sudo ss -tlnp | grep :80"
ssh <host> "sudo ss -tlnp | grep :4873"

# Stop conflicting service
ssh <host> "sudo systemctl stop nginx"  # if nginx already installed

Cannot Access via forge.nasty.sh

1. Check /etc/hosts on your workstation:

   grep forge.nasty.sh /etc/hosts
   

2. Check VPN connection (if using):

   ping 10.0.0.11
   

3. Check nginx on host:

   ssh <host> "docker exec forgejo-nginx nginx -t"
   

Verdaccio Not Caching Packages

1. Check token is set:

   ssh <host> "grep FORGEJO_NPM_TOKEN /bigdisk/forgejo/.env"
   

2. Check Verdaccio logs:

   ssh <host> "docker logs verdaccio"
   

3. Restart services:

   ssh <host> "sudo systemctl restart devops"
   

---

Maintenance

Backup

# Backup complete data directory
ssh <host> "sudo tar -czf /tmp/devops-backup-$(date +%Y%m%d).tar.gz /bigdisk/forgejo/data /bigdisk/verdaccio/storage"
scp <host>:/tmp/devops-backup-*.tar.gz ./backups/

Update Forgejo

# Edit docker-compose.yml to new version
vim deployments/docker/forgejo/docker-compose.yml

# Deploy update
scp deployments/docker/forgejo/docker-compose.yml <host>:/bigdisk/forgejo/
ssh <host> "cd /bigdisk/forgejo && docker-compose pull forgejo"
ssh <host> "sudo systemctl restart devops"

Clean Old Packages

# Check cache size
ssh <host> "du -sh /bigdisk/verdaccio/storage"

# Clean cache (careful!)
ssh <host> "rm -rf /bigdisk/verdaccio/storage/*"
ssh <host> "sudo systemctl restart devops"

---

Security Notes

• VPN-only access: Nginx restricts to 10.0.0.0/24 and 10.9.0.0/24
• Secrets: Auto-generated, stored in /bigdisk/forgejo/.env (mode 600)
• Database: Password-protected PostgreSQL
• Verdaccio: htpasswd authentication
• SSH Git: Non-standard port 2222

---

Architecture

                    ┌─────────────────────────────────┐
                    │     Client (Your Machine)        │
                    │  /etc/hosts: 10.0.0.11 forge... │
                    └─────────────┬───────────────────┘
                                  │
                                  │ HTTP/HTTPS/SSH
                                  ▼
                    ┌─────────────────────────────────┐
                    │   Nginx (forgejo-nginx)          │
                    │   Ports: 80, 443, 2222           │
                    └─────────┬──────────┬─────────────┘
                              │          │
             forge.nasty.sh   │          │   npm.nasty.sh
                              ▼          ▼
                    ┌──────────────┐  ┌──────────────┐
                    │   Forgejo    │  │  Verdaccio   │
                    │   :3000      │  │   :4873      │
                    └──────┬───────┘  └──────────────┘
                           │
                           ▼
                    ┌──────────────┐
                    │  PostgreSQL  │
                    │   :5432      │
                    └──────────────┘

All containers on forgejo_forgejo Docker network

---

Related Scripts

• Deploy Verdaccio: tooling/scripts/deploy/deploy-verdaccio.sh
• Configure Client: tooling/scripts/dev-setup/configure-verdaccio-client.sh
• VPN Setup: tooling/scripts/dev-setup/setup-vpn-access.sh

---

Last Updated: 2026-01-13 Script: deployments/provisioning/setup-devops-host.sh Service: devops.service (systemd)