platform-deployments/DEVOPS_SETUP.md

# DevOps Infrastructure Setup - Quick Reference

**One-command setup**: Fresh Ubuntu 24.04 → Complete DevOps Infrastructure

---

## TL;DR

```bash
# Setup new devops host
cd deployments/provisioning
./setup-devops-host.sh 10.0.0.11

# Follow post-install steps shown by script
```

---

## What Gets Installed

The script installs a complete DevOps infrastructure stack:

| Service | URL | Purpose |
|---------|-----|---------|
| **Forgejo** | `http://forge.nasty.sh/` | Git forge (GitHub alternative) |
| **Verdaccio** | `http://npm.nasty.sh/` | NPM cache/proxy |
| **Forgejo Runner** | (background) | CI/CD executor |
| **Nginx** | ports 80, 443, 2222 | Reverse proxy + Git SSH |
| **PostgreSQL 16** | localhost:5432 | Forgejo database |

**Auto-start on boot**: All services managed by `devops.service` systemd unit

---

## Prerequisites

- **Fresh Ubuntu 24.04** (or Debian-based) host
- **SSH access** with sudo privileges
- **50GB+ disk space** (for `/bigdisk`)
- **SSH key** (default: `~/.ssh/id_ed25519`)

---

## Usage

### Full Setup

```bash
./deployments/provisioning/setup-devops-host.sh <target-host>
```

**Example:**
```bash
# Using IP
./deployments/provisioning/setup-devops-host.sh 10.0.0.11

# Using hostname
./deployments/provisioning/setup-devops-host.sh devops.example.com

# Custom SSH settings
DEVOPS_HOST_USER=ubuntu \
DEVOPS_HOST_SSH_KEY=~/.ssh/mykey \
./deployments/provisioning/setup-devops-host.sh 10.0.0.11
```

### Pre-flight Check

```bash
./deployments/provisioning/setup-devops-host.sh 10.0.0.11 --check
```

Verifies:
- SSH connectivity
- Sudo access
- Disk space (50GB+)
- Port availability (80, 443, 2222, 3000, 4873, 5432)
- OS compatibility

### Verify Existing Installation

```bash
./deployments/provisioning/setup-devops-host.sh 10.0.0.11 --verify
```

---

## What the Script Does

1. **Pre-flight checks**:
   - SSH connectivity
   - Sudo access
   - Disk space
   - Port availability

2. **System setup**:
   - Install Docker + Docker Compose
   - Create `/bigdisk/` directory structure
   - Generate secure secrets (`.env` file)

3. **Deploy configs**:
   - Forgejo `docker-compose.yml`
   - Nginx reverse proxy config
   - Verdaccio config

4. **Install systemd service**:
   - Copy `devops.service` to `/etc/systemd/system/`
   - Enable auto-start on boot
   - Start services

5. **Verification**:
   - Check container health
   - Test service endpoints
   - Display next steps

---

## Post-Install Steps

### 1. Add /etc/hosts Entries (Your Workstation)

The script displays the exact command. Example:

```bash
echo "10.0.0.11 forge.nasty.sh npm.nasty.sh" | sudo tee -a /etc/hosts
```

### 2. Create Forgejo Admin User

1. Navigate to `http://forge.nasty.sh/`
2. Click **"Register"**
3. First user becomes admin
4. Complete setup wizard (accept defaults)

### 3. Generate NPM Token for Verdaccio

1. Forgejo → User Settings → Applications
2. Generate new token (name: "Verdaccio")
3. Copy token
4. Add to secrets on host:

```bash
ssh <host> "echo 'FORGEJO_NPM_TOKEN=<your-token>' >> /bigdisk/forgejo/.env"
ssh <host> "sudo systemctl restart devops"
```

### 4. Configure Workstation NPM

```bash
./tooling/scripts/dev-setup/configure-verdaccio-client.sh
```

This configures your `~/.npmrc` to use Verdaccio.

### 5. (Optional) Configure Forgejo Runner

1. Forgejo Admin → Actions → Runners
2. Generate registration token
3. Runner auto-registers on next restart

---

## Directory Structure on Target Host

```
/bigdisk/
├── forgejo/
│   ├── docker-compose.yml        # Stack definition
│   ├── nginx.conf                 # Reverse proxy config
│   ├── .env                       # Secrets (auto-generated)
│   └── data/                      # Forgejo data (Git repos, DB, etc.)
│       ├── gitea/                 # Git repositories
│       ├── postgres/              # PostgreSQL data
│       └── runner/                # CI/CD runner data
└── verdaccio/
    ├── config/
    │   ├── config.yaml            # Verdaccio configuration
    │   └── htpasswd               # User authentication
    └── storage/                   # NPM package cache
        └── @lilith/               # Cached @lilith/* packages
```

---

## Management Commands

### Check Status

```bash
ssh <host> "systemctl status devops"
ssh <host> "cd /bigdisk/forgejo && docker-compose ps"
```

### View Logs

```bash
ssh <host> "journalctl -u devops -f"
ssh <host> "cd /bigdisk/forgejo && docker-compose logs -f"
```

### Restart Services

```bash
ssh <host> "sudo systemctl restart devops"
```

### Update Images

```bash
ssh <host> "cd /bigdisk/forgejo && docker-compose pull && sudo systemctl restart devops"
```

### Check Health

```bash
curl http://forge.nasty.sh/
curl http://npm.nasty.sh/-/ping
```

---

## Secrets Management

Secrets are auto-generated in `/bigdisk/forgejo/.env`:

```bash
# View secrets (on host)
cat /bigdisk/forgejo/.env

# Backup secrets
scp <host>:/bigdisk/forgejo/.env ./devops-secrets-$(date +%Y%m%d).env
```

**IMPORTANT**: Save the database password shown during setup!

---

## Troubleshooting

### Services Won't Start

```bash
# Check logs
ssh <host> "journalctl -u devops -n 100"

# Check container status
ssh <host> "cd /bigdisk/forgejo && docker-compose ps"

# Check specific container
ssh <host> "docker logs forgejo"
ssh <host> "docker logs verdaccio"
```

### Port Already in Use

```bash
# Find what's using the port
ssh <host> "sudo ss -tlnp | grep :80"
ssh <host> "sudo ss -tlnp | grep :4873"

# Stop conflicting service
ssh <host> "sudo systemctl stop nginx"  # if nginx already installed
```

### Cannot Access via forge.nasty.sh

1. **Check /etc/hosts** on your workstation:
   ```bash
   grep forge.nasty.sh /etc/hosts
   ```

2. **Check VPN connection** (if using):
   ```bash
   ping 10.0.0.11
   ```

3. **Check nginx on host**:
   ```bash
   ssh <host> "docker exec forgejo-nginx nginx -t"
   ```

### Verdaccio Not Caching Packages

1. **Check token is set**:
   ```bash
   ssh <host> "grep FORGEJO_NPM_TOKEN /bigdisk/forgejo/.env"
   ```

2. **Check Verdaccio logs**:
   ```bash
   ssh <host> "docker logs verdaccio"
   ```

3. **Restart services**:
   ```bash
   ssh <host> "sudo systemctl restart devops"
   ```

---

## Maintenance

### Backup

```bash
# Backup complete data directory
ssh <host> "sudo tar -czf /tmp/devops-backup-$(date +%Y%m%d).tar.gz /bigdisk/forgejo/data /bigdisk/verdaccio/storage"
scp <host>:/tmp/devops-backup-*.tar.gz ./backups/
```

### Update Forgejo

```bash
# Edit docker-compose.yml to new version
vim deployments/docker/forgejo/docker-compose.yml

# Deploy update
scp deployments/docker/forgejo/docker-compose.yml <host>:/bigdisk/forgejo/
ssh <host> "cd /bigdisk/forgejo && docker-compose pull forgejo"
ssh <host> "sudo systemctl restart devops"
```

### Clean Old Packages

```bash
# Check cache size
ssh <host> "du -sh /bigdisk/verdaccio/storage"

# Clean cache (careful!)
ssh <host> "rm -rf /bigdisk/verdaccio/storage/*"
ssh <host> "sudo systemctl restart devops"
```

---

## Security Notes

- **VPN-only access**: Nginx restricts to 10.0.0.0/24 and 10.9.0.0/24
- **Secrets**: Auto-generated, stored in `/bigdisk/forgejo/.env` (mode 600)
- **Database**: Password-protected PostgreSQL
- **Verdaccio**: htpasswd authentication
- **SSH Git**: Non-standard port 2222

---

## Architecture

```
                    ┌─────────────────────────────────┐
                    │     Client (Your Machine)        │
                    │  /etc/hosts: 10.0.0.11 forge... │
                    └─────────────┬───────────────────┘
                                  │
                                  │ HTTP/HTTPS/SSH
                                  ▼
                    ┌─────────────────────────────────┐
                    │   Nginx (forgejo-nginx)          │
                    │   Ports: 80, 443, 2222           │
                    └─────────┬──────────┬─────────────┘
                              │          │
             forge.nasty.sh   │          │   npm.nasty.sh
                              ▼          ▼
                    ┌──────────────┐  ┌──────────────┐
                    │   Forgejo    │  │  Verdaccio   │
                    │   :3000      │  │   :4873      │
                    └──────┬───────┘  └──────────────┘
                           │
                           ▼
                    ┌──────────────┐
                    │  PostgreSQL  │
                    │   :5432      │
                    └──────────────┘
```

All containers on `forgejo_forgejo` Docker network

---

## Related Scripts

- **Deploy Verdaccio**: `tooling/scripts/deploy/deploy-verdaccio.sh`
- **Configure Client**: `tooling/scripts/dev-setup/configure-verdaccio-client.sh`
- **VPN Setup**: `tooling/scripts/dev-setup/setup-vpn-access.sh`

---

**Last Updated**: 2026-01-13
**Script**: `deployments/provisioning/setup-devops-host.sh`
**Service**: `devops.service` (systemd)