lilith-platform.live/deployments/@domains/quinn.admin-api
Natalie e289cdd6ef feat(infra): no more black for CI/runners — migrate LP CI+deploys to DO ct-forge on-demand runners
- Updated main ci.yml verify job and all deploy-*.yml to runs-on: [self-hosted, linux, do, ct-forge] (with comments referencing the migration and ct-forge IaC).
- Updated setup-forgejo-host.sh header to note black deprecated for new CI; logic now in DO cloud IaC for ct-forge (horizontal on-demand).
- Updated quinn.admin-api README to reflect DO runners (no black runner).
- 'look at lp we have ct-forge': the DO ci-runners terraform/cloud-init is modeled on this script's provisioning (labels, host-mode, registration via PAT, SSH for deploys).
- Matches 'no more black... we have DO' + ct-forge as canonical for runners/CI.
- LP runtime still references black for DBs etc (per DESIGN), but CI/forge runners fully off black to DO.
2026-06-28 17:15:35 -04:00
..
deploy.sh fix(quinn.admin-api/deploy): use run_remote_cmd for service restart 2026-06-24 19:53:59 -04:00
quinn-admin-api.service
README.md feat(infra): no more black for CI/runners — migrate LP CI+deploys to DO ct-forge on-demand runners 2026-06-28 17:15:35 -04:00
services.yaml

quinn.admin-api — canonical origin on black

Mesh-only deployment of the consolidated @features/api monolith on black. Replaces quinn.admin on vps-0 as the canonical writer for /admin/* and the origin for /www/*. Public internet reaches /www/* via the quinn.api-cache vhost on vps-0 (api.transquinnftw.com).

Architecture

                public internet
                       │
                       ▼
     ┌───────────────────────────────────┐
     │  vps-0 / quinn.api-cache (nginx)  │  api.transquinnftw.com
     │  /www/* cached 30s · /admin/* 404 │
     └───────────────┬───────────────────┘
                     │ mesh (WireGuard)
                     ▼
     ┌───────────────────────────────────┐
     │   black / quinn.admin-api         │  api.black.lan:3023 — mesh-only
     │   systemd: quinn-admin-api.service│  bound to 10.0.0.11 only
     │   reads quinn.db on black:25435   │
     └───────────────────────────────────┘
                     ▲
                     │ mesh
            plum, apricot, etc. — admin writes go DIRECT

Companion artifacts

  • services.yaml — deployment metadata
  • deploy.sh — rsync + restart + auto-rollback (mirrors the existing quinn.admin convention; REMOTE=black instead of quinn-vps)
  • quinn-admin-api.service — the systemd unit to install at /etc/systemd/system/quinn-admin-api.service on black

One-time setup on black (pre-cutover checklist)

# 1. Install the systemd unit + create system user
sudo install -m 0644 quinn-admin-api.service /etc/systemd/system/
sudo useradd -r -s /usr/sbin/nologin quinn-api  # if not exists
sudo install -d -o quinn-api -g quinn-api -m 0755 /opt/quinn-admin-api/data

# 2. Drop secrets in place (copy from quinn-vps initially, rotate afterwards)
sudo install -d -m 0700 -o quinn-api -g quinn-api /etc/quinn-admin-api
sudo scp quinn-vps:/etc/quinn-admin-api/secrets.env /etc/quinn-admin-api/secrets.env
sudo chown quinn-api:quinn-api /etc/quinn-admin-api/secrets.env
sudo chmod 0600 /etc/quinn-admin-api/secrets.env

# 3. Allow the forgejo runner to restart this unit without password
echo 'forgejo-runner ALL=(root) NOPASSWD: /bin/systemctl restart quinn-admin-api.service' \
  | sudo tee /etc/sudoers.d/forgejo-runner-quinn-admin-api

# 4. Firewall: refuse :3023 from anything outside the WireGuard subnet
sudo nft add rule inet filter input tcp dport 3023 iifname != wg0 drop

# 5. Enable + start (no artifacts yet — workflow ships them on first run)
sudo systemctl daemon-reload
sudo systemctl enable quinn-admin-api.service
# Don't `systemctl start` until the workflow has shipped /opt/quinn-admin-api/dist/server.node.js

Cutover sequence (when going live)

  1. Tag the legacy quinn.admin deployment with the current production SHA so rollback is one rsync away.
  2. Trigger Deploy quinn.admin-api (black — target arch) workflow manually via the Forgejo UI.
  3. Watch the workflow's Verify public read path step.
  4. Flip DNS: add A record api.transquinnftw.com → <quinn-vps public IP> at the apex DNS host.
  5. Run certbot --nginx -d api.transquinnftw.com on quinn-vps.
  6. Re-trigger the workflow so the api-cache proxy_pass step succeeds (the first run will fail at the cert step if certbot hasn't run yet).
  7. Verify curl https://api.transquinnftw.com/www/tour/status returns 200.
  8. Update internal callers (m. dashboard, admin SPA, MCP, etc.) to use api.black.lan:3023 for admin writes (mesh-only).
  9. After 24h of green metrics: delete .forgejo/workflows/deploy-quinn-admin.yml and deployments/@domains/quinn.admin/ in a follow-up PR.

Open items before cutover

  • DNS A record for api.transquinnftw.com (UI task at whatever DNS host owns the apex — likely cloudflare).
  • TLS cert via certbot on quinn-vps.
  • SSH from runner to black host — NO MORE BLACK RUNNERS (migrated to DO ct-forge on-demand). Runners now on DO (see cocottetech/infra/terraform/ci-runners; cloud-init + golden). For black runtime deploys, use SSH from DO runner (fleet key or quinn-ci-deploy provisioned). Old black runner logic in setup-forgejo-host.sh now in DO IaC. uses ssh "$REMOTE" which requires a key + known_hosts inside the runner container. Either bind-mount /root/.ssh from the host into the runner, or generate a deploy-specific key and add it to root@black's authorized_keys with a command= restriction.
  • Token rotation post-cutover (the SERVICE_TOKEN in /etc/quinn-admin-api/secrets.env should change once it lives on a new host).