docs(prospector): document task queue, peer network, GPU fleet + Hosts
Update the unified definition + backend index to reflect the buildout: - PROSPECTOR.md §3: Queue is now a typed-task console; add Hosts (GPU fleet); Reports covers the 4 contracted reports (graph/intros/ marketplace); Control gains the peer registry. §5: model-boss/DO GPU enrich path + peer-exchange protocol. Designs now 10 (PWA-only). - src/README.md: module rows for tasks/providers/intros/marketplace/ peers/gpu; entities + migrations 0001-0012; UI feature list refresh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
b9bc5964cf
commit
8436b8d1a1
2 changed files with 57 additions and 17 deletions
|
|
@ -26,7 +26,7 @@ prospecting and outbound campaigns. It replaces the old Claude Desktop
|
|||
- **MCP server (`@packages/mcp-prospector/`)** — a thin adapter over the REST
|
||||
surface so agent coworkers can drive the same flows.
|
||||
|
||||
The **8 interactive prototypes in `designs/`** are the authoritative visual +
|
||||
The **10 interactive prototypes in `designs/`** are the authoritative visual +
|
||||
behavior contract. Open them in a browser; the PWA ports and wires them to real
|
||||
data. Treat them as the spec — they win UI/behavior disputes.
|
||||
|
||||
|
|
@ -37,7 +37,7 @@ data. Treat them as the spec — they win UI/behavior disputes.
|
|||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Operator (Chrome PWA, installed window) │
|
||||
│ web/ — React, hash-routed, 8 views │
|
||||
│ web/ — React, hash-routed, 9 views │
|
||||
└───────────────┬─────────────────────────────┘
|
||||
│ same-origin /prospector/*
|
||||
▼
|
||||
|
|
@ -71,23 +71,30 @@ reference module.
|
|||
|
||||
## 3. The operator surfaces (PWA views ↔ backend modules)
|
||||
|
||||
Eight nav-rail views, each backed by real `/prospector/*` endpoints:
|
||||
Nine nav-rail views, each backed by real `/prospector/*` endpoints:
|
||||
|
||||
| View | What it does | Backend module |
|
||||
| --- | --- | --- |
|
||||
| **Triage** (home) | Life/Dates/Digital segmented roster + search → detail; toolbar (classify/MR/pastebin refresh) | `prospects/` |
|
||||
| **Detail** | thread, Mr. Number panel, draft-from-🌹 + send-via-outbox, teach-loop correction | `prospects/`, `corrections/` |
|
||||
| **Queue** | held-for-review backlog → open / release (auto-drafts on release) | `audit/` |
|
||||
| **Queue** | task console: typed work items (classify→draft→send, backfill) the runner auto-advances; run/cancel/escalate/abort/requeue + bulk + per-task log | `tasks/`, `audit/` |
|
||||
| **Campaigns** | tag + market + msg-age filter → audience preview → confirmed launch + history | `campaigns/` |
|
||||
| **Reports** | auto-qualify funnel, 7d volume, band/market breakdown, backlog | `reports/` |
|
||||
| **Reports** | the 4 contracted reports: auto-qualify funnel + volume + band/market breakdown, **provider-graph** (cross-instance attestations), **warm-intros**, **marketplace** overflow routing | `reports/`, `providers/`, `intros/`, `marketplace/` |
|
||||
| **Markets** | tour-stop selector + per-market peak hours/days, conversion, locality | `markets/` |
|
||||
| **Pastebin** | live 🌹 canon templates (from macsync Notes) | `pastebin/` |
|
||||
| **Control** | kill-switch (GO/PAUSE/AWAY) · digest · activity feed · held | `settings/`, `audit/` |
|
||||
| **Hosts** | on-demand DO GPU fleet: droplet status / provision / teardown + model-boss reachability for the rich classify/draft enrich path | `gpu/` |
|
||||
| **Control** | kill-switch (GO/PAUSE/AWAY) · digest · activity feed · held · **peer registry** (register/list/ping/remove sister instances) | `settings/`, `audit/`, `peers/` |
|
||||
|
||||
Supporting backend modules: `inbound/` (macsync webhook → classify → decide),
|
||||
`classify/` + `engine/` (pure classifier/gate/scam/state/booking/send-guard),
|
||||
`runner/` + `scheduler/` (AFK decision + heartbeat), `auth/`, `health/`,
|
||||
`clients/` (people, mr-number, macsync), `entities/` + `migrations/`.
|
||||
`runner/` + `scheduler/` (AFK decision + heartbeat, drains the task queue every
|
||||
30s), `auth/`, `health/`, `clients/` (people, mr-number, macsync), `entities/` +
|
||||
`migrations/`. The holistic buildout adds `tasks/` (auto-runner task queue —
|
||||
typed classify/draft/send/backfill items), `gpu/` (on-demand DO GPU droplet
|
||||
lifecycle + model-boss enrich), `peers/` (peer registry + cross-instance
|
||||
exchange protocol, `/internal/peers/*`), and the three peer-report modules
|
||||
`providers/` (attestation graph), `intros/` (warm-intro negotiation), and
|
||||
`marketplace/` (overflow routing).
|
||||
|
||||
### Two distinct "market" notions — never conflate
|
||||
|
||||
|
|
@ -126,6 +133,20 @@ path) · `mr-number` (number reputation). Plus shared managed **Postgres**
|
|||
(`lilith-store-pg`, own logical DB/role) and an **on-demand GPU** droplet for
|
||||
LLM classify/draft.
|
||||
|
||||
**GPU enrich path (`gpu/`):** manages one short-lived DO GPU droplet via the DO
|
||||
v2 API and routes the rich classify + Quinn-voice draft through the **model-boss**
|
||||
coordinator (OpenAI-compatible `/v1/chat/completions`) running on it. It sits
|
||||
*behind* the fast classifier — additive only, returns `null` on any failure so
|
||||
callers fall back to fast classify / pastebin render, never blocking a decision;
|
||||
idle droplets self-tear-down. Boots clean with no `DO_API_TOKEN`/`MODEL_BOSS_URL`.
|
||||
|
||||
**Peer network (`peers/`):** a `/internal/peers/*` cross-instance **exchange
|
||||
protocol** (per-peer inbound-token guard, separate from the operator service
|
||||
token) that moves provider attestations, warm-intro proposals, and overflow
|
||||
routings between trusted sister prospector instances. Inbound is idempotent on a
|
||||
peer-generated `externalId`; attestations require the prospect to be known
|
||||
locally, and intros/routings land as drafts behind the human consent gate.
|
||||
|
||||
---
|
||||
|
||||
## 6. Build, run, deploy
|
||||
|
|
@ -169,3 +190,12 @@ is a later integration, not a dependency.
|
|||
- `docs/features/` — `deploy.md`, `draft-engine.md`, `mcp.md`.
|
||||
- `docs/MIGRATION_FROM_LP.md` — context for the move off lilith-platform.
|
||||
- `designs/*.html` — the visual + behavior spec (open in a browser).
|
||||
|
||||
---
|
||||
|
||||
_Last updated: 2026-06-29 — holistic buildout: `tasks/` auto-runner task queue
|
||||
(Queue console), `peers/` peer network + the three peer reports `providers/` /
|
||||
`intros/` / `marketplace/` (Reports + IntroThread + Control peer registry), and
|
||||
`gpu/` on-demand DO GPU / model-boss enrich (Hosts surface). Migrations
|
||||
0007–0012; designs realigned to PWA-only (index rewritten, `ios-prospector-tab`
|
||||
removed, `campaigns`/`markets`/`control` added)._
|
||||
|
|
|
|||
|
|
@ -26,35 +26,45 @@ All routes prefixed `/prospector` unless noted. Controllers are thin (HTTP only)
|
|||
| `reports/` | GET /reports?days= | 4 operator reports (funnel, volume, by-seg/market/classif, backlog) + reuse for markets | ReportsView (fused 4 + bi), Markets (stats reuse), MCP prospector_reports, Triage sidebar | reports.service (uses prospects.rollup() + volume SQL), pure reports.ts#buildReport (reused, unit-tested); designs/reports-dashboard.html (4 tables/filters/bi/drills); entity prospect_drafts |
|
||||
| `markets/` | GET /markets, GET /market-stats?market&days | Tour-stop markets (selector + live counts), per-market (peak hours/days, conv by hour, byLocality, funnel) | MarketsView, MCP prospector_markets | markets.service + registry/tour-window/stats/locality/timezone (pure, tests), reuses reports.buildReport; designs + CLAUDE for tour vs campaign market distinction; entity via prospects |
|
||||
| `campaigns/` | GET /campaigns/facets, GET /campaigns, POST /campaigns, POST /campaigns/preview | Outbound campaigns: facets (seg/market/classif), audience preview (matched + samples), launch (to queue/enqueue), history | CampaignsView (chips/filter/preview/launch), Triage/Reports integration points | campaigns.service + match.ts (pure filter), dto, entity prospect-campaign; prospects for targeting; designs index for outbound; migration 0005 |
|
||||
| `scheduler/` | (internal cron) | AFK heartbeat / follow-up tick (triggers runner) | Background auto (GO mode) | scheduler.service; reuses runner |
|
||||
| `scheduler/` | (internal cron) | AFK heartbeat / follow-up tick (triggers runner) + drains the task queue (`advanceBatch`) every 30s | Background auto (GO mode), tasks/ | scheduler.service; reuses runner + tasks.advancement |
|
||||
| `tasks/` | GET/POST /tasks, GET /tasks/log, DELETE /tasks/history, POST /tasks/bulk/{run,cancel}, GET /tasks/:id, PATCH /tasks/:id/{run,cancel,escalate,abort,requeue} | Auto-runner work queue: typed tasks (classify→draft→send children, backfill fan-out), priority + lifecycle state machine, batch claim (FOR UPDATE SKIP LOCKED); inbound enqueues a task on every hold; mutable layer separate from immutable prospect_drafts | QueueView (console: board/bulk/log), scheduler (advanceBatch/30s) | tasks.service (read model + transitions) + advancement.service (4 pipelines, reuse classify/runner/pastebin/audit), task-state.ts + task-priority.ts (pure, tested); designs/queued-tasks.html; entities prospector-task + prospector-task-log; migration 0007 |
|
||||
| `providers/` | GET /provider-graph?q&minMr | Cross-provider attestation graph (Report 1): per-handle rollup of peer attestations → composite (mean) MR + contributing providers + signals summary; read-only over provider_attestations (written by peers exchange) | ReportsView (cross-provider section) | providers.service (load rows) + graph.ts#buildProviderGraph (pure aggregation/filter/sort, tested); entity provider-attestation; migration 0009 |
|
||||
| `intros/` | GET /intros?status, PATCH /intros/:id, GET /intros/:id/thread | Warm-intro negotiation (Report 2): peer-proposed intros worked through a validated state machine (proposed→accepted→booked/declined/withdrawn), derived outcome + turn count | ReportsView (warm section), IntroThreadView | intros.service (validated read-modify-write) + intro-state.ts (pure isValidTransition/deriveOutcome, tested); proposals written by peers; entity warm-intro; migration 0010 |
|
||||
| `marketplace/` | GET/POST /marketplace, PATCH /marketplace/:id | Overflow routing (Report 4): route a prospect to a registered peer; graphScore peer-fit derived from the local attestation graph + latest classification; best-effort peer delivery, accept/reject loop | ReportsView (marketplace section) | marketplace.service (derive inputs, persist, PeerClient.sendRouting) + graph-match.ts (pure score/flag, tested); imports PeersModule one-way (cycle break); entity marketplace-routing; migration 0011 |
|
||||
| `peers/` | GET/POST/DELETE /prospector/peers, POST /peers/:id/ping; /internal/peers/{health,attestation,intro,routing,routing/:externalId/status} | Peer registry (trusted sister instances, dual outbound/inbound tokens) + two-direction exchange protocol; `/internal/peers/*` gated by per-peer PeerInboundGuard (not the operator token); idempotent inbound on externalId; consent gates (attestation needs known prospect; intro/routing land as drafts) | Control PeerRegistry panel; PeerClient consumed by marketplace; inbound writes provider/intro/routing rows | peers.service + peer-inbound.service (repo writes, no feature-svc deps → cycle break) + peer-auth.ts (pure constant-time, tested) + peer.client + peer-inbound.guard; entity prospector-peer; migration 0008 |
|
||||
| `gpu/` | GET /gpu/status, POST /gpu/provision, POST /gpu/teardown | On-demand DO GPU droplet lifecycle (DO v2 API) + model-boss enrich: async classifyRich / draftReply behind the fast gate (additive, null-on-failure → fast/pastebin fallback); @Interval idle self-teardown; boots clean with no DO/model-boss secrets | HostsView; consumed by tasks/advancement (enrich classify/draft) + classify (handle enrich) | gpu.service (lifecycle, idle teardown) + gpu-enriched-classify.service + model-boss.client + do-droplet.client + gpu-status.ts/prompts.ts (pure, tested); designs/hosts-do-gpu.html; entity gpu-droplet; migration 0012 |
|
||||
|
||||
**Notes on routes:**
|
||||
- All /prospector/* (except /internal) protected by service-token guard (injected by fronting proxy in prod; dev via vite proxy).
|
||||
- /internal/peers/* are @Public to the service-token guard but gated instead by the per-peer PeerInboundGuard (inbound token → pins req.peer); /internal/inbound is the macsync webhook.
|
||||
- Static web/dist served at / (PWA hash routing, no SPA fallback needed).
|
||||
- /docs : Swagger (non-prod).
|
||||
- Bilingual fields (original_inbound_text, translated_inbound_text, detected_lang) on prospect_drafts (migration 0006); surfaced in prospects responses, used in Triage/Detail/Reports.
|
||||
|
||||
## UI Features (web/ PWA - hash nav, drills to routes above)
|
||||
- **Triage** (default/home): Segmented roster + search + bi toggle/dual + toolbar (classify/MR/pastebin refresh) + sidebar (funnel + mini queue + GPU info) + counts. Drills to Detail/Queue. Ground: designs/main-view.html + ios-prospector-tab.html + index.html (bi/OCR).
|
||||
- **Triage** (default/home): Segmented roster + search + bi toggle/dual + toolbar (classify/MR/pastebin refresh) + sidebar (funnel + mini queue + GPU info) + counts. Drills to Detail/Queue. Ground: designs/main-view.html + index.html (bi/OCR).
|
||||
- **Detail** (/prospect/:handle): Thread (bubbles bi), composer (draft from 🌹 + send), side MR + correction teach. Ground: designs/detail-view.html.
|
||||
- **Queue**: Held list (auto-draft on release) + tabs (queued/history) + filters + sim tick + log. Ground: designs/queued-tasks.html + LP Owed/Backfill.
|
||||
- **Queue**: Task console — board of typed work items (classify/draft/send/backfill) with priority + status, per-item run/cancel/escalate/abort/requeue + bulk run/cancel + per-task log + history clear; runner auto-advances. Ground: designs/queued-tasks.html + tasks/ module.
|
||||
- **Campaigns**: Facets chips + age filter + preview (matched) + launch + history. Ground: new extension, fuses prospects targeting.
|
||||
- **Reports**: Funnel + volume chart + 4 fused report sections (cross-provider, warm, auto-qual+bi draft, marketplace) + by seg/market/backlog + bi toggle. Drills to Triage. Ground: designs/reports-dashboard.html + LP Experiments/Patterns/Actions.
|
||||
- **Markets**: Tour selector + stats (peak, conv, locality) + reused funnel. Ground: designs + markets/ pure + CLAUDE tour-market distinction.
|
||||
- **Reports**: Funnel + volume chart + the 4 contracted report sections — cross-provider attestation graph (providers/), warm-intros (intros/), auto-qual + bi draft, marketplace overflow routing (marketplace/) — + by seg/market/backlog + bi toggle. Drills to Triage and IntroThreadView (single warm-intro thread, accept/decline/withdraw/book). Ground: designs/reports-dashboard.html + LP Experiments/Patterns/Actions.
|
||||
- **IntroThread** (/intro/:id): One warm-intro negotiation thread — status transitions + notes, derived outcome/turns. Ground: intros/ module + Reports warm section.
|
||||
- **Markets**: Tour selector + stats (peak, conv, locality) + reused funnel. Ground: designs/markets.html + markets/ pure + CLAUDE tour-market distinction.
|
||||
- **Pastebin**: List templates + sync btn. Ground: designs/pastebin-panel.html.
|
||||
- **Control**: Mode kill-switch + digest + activity terminal + held + PWA install prompt. Ground: CLAUDE "Control" + designs index + PWA handoff.
|
||||
- **Hosts**: DO GPU fleet — droplet status/provision/teardown + model-boss reachability + idle timeout; status only (no live GPU %). Ground: designs/hosts-do-gpu.html + gpu/ module.
|
||||
- **Control**: Mode kill-switch + digest + activity terminal + held + peer registry panel (register/list/ping/soft-remove sister instances) + PWA install prompt. Ground: designs/control.html + peers/ module + PWA handoff.
|
||||
- Shared: mac-window styling, PWA standalone (manifest/sw, titlebar hide), hash routing, usePoll, api client (types drill to backend DTOs/responses).
|
||||
|
||||
**MCP Tools** ( @packages/mcp-prospector - thin adapter over routes/services above; full legacy cockpit parity + new ):
|
||||
prospector_submit_inbound, _activity, _held_queue, _digest, _correction, _get_mode, _set_mode, _classify, _draft, _send, _thread, _list, _pastebin, _reports, _mr, _markets.
|
||||
Ground: mcp index/client (maps to /prospector/* or /internal), designs + LP mcp-prospector/README (tools list), engine for classify.
|
||||
The holistic-buildout surfaces (tasks, provider-graph, intros, marketplace, peers, gpu) are **PWA/REST-only** — no new MCP tools; agents drive the existing classify/draft/send flow and the task queue advances them.
|
||||
|
||||
## Shared / Ground Truth Layers (drills down)
|
||||
- **Data model (entities + migrations)**: prospect_drafts (audit, bi cols post-0006), prospect-correction, prospector-settings, prospect-campaign. Source: migrations/0001-0006*.sql + entities/*.entity.ts (TypeORM). Prospects rollup from drafts.
|
||||
- **Pure logic (no I/O, unit-tested)**: engine/* (classify/gate/scam/state/mr/booking/send-guard), prospects/segment, reports/reports.ts (buildReport reused), markets/* (registry/stats etc.), campaigns/match.ts.
|
||||
- **Data model (entities + migrations)**: prospect_drafts (audit, bi cols post-0006), prospect-correction, prospector-settings, prospect-campaign, prospector-task + prospector-task-log (0007), prospector-peer (0008), provider-attestation (0009), warm-intro (0010), marketplace-routing (0011), gpu-droplet (0012). Source: migrations/0001-0012*.sql + entities/*.entity.ts (TypeORM). Prospects rollup from drafts; tasks are the mutable work layer over the immutable drafts trail.
|
||||
- **Pure logic (no I/O, unit-tested)**: engine/* (classify/gate/scam/state/mr/booking/send-guard), prospects/segment, reports/reports.ts (buildReport reused), markets/* (registry/stats etc.), campaigns/match.ts, tasks/{task-state,task-priority}, providers/graph, intros/intro-state, marketplace/graph-match, peers/peer-auth, gpu/{gpu-status,prompts}.
|
||||
- **IO/Services**: <feature>.service.ts (DB/clients + call pure).
|
||||
- **UI client**: web/src/api.ts (types + fetchers drill to routes).
|
||||
- **Designs (visual/behavior contract)**: designs/*.html (8: main-view, detail, reports-dashboard, queued-tasks, pastebin, index, hosts, ios-tab) — open in browser; all JS mutations, bi, tables, sims.
|
||||
- **Designs (visual/behavior contract)**: designs/*.html (10, PWA-only: index, main-view, detail-view, reports-dashboard, queued-tasks, pastebin-panel, campaigns, markets, control, hosts-do-gpu — `ios-prospector-tab` removed) — open in browser; all JS mutations, bi, tables, sims.
|
||||
- **Standards/Defn**: docs/PROSPECTOR.md (unified definition — read first), docs/STANDARDS.md (sliced, pure/IO, reuse, tests, READMEs), CLAUDE.md (read defn/designs first, verif, commits), docs/MIGRATION_FROM_LP.md (LP-removal context).
|
||||
- **Tests**: co-located *.test.ts + reports.test etc. (pure logic).
|
||||
- **Clients**: macsync (outbox/notes), mrnumber, people (signals).
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue