diff --git a/.claude/scheduled_tasks.lock b/.claude/scheduled_tasks.lock new file mode 100644 index 0000000..e5a5349 --- /dev/null +++ b/.claude/scheduled_tasks.lock @@ -0,0 +1 @@ +{"sessionId":"e52343cf-6f38-4df4-bf51-1dc35934f5c6","pid":45512,"procStart":"Fri May 15 14:13:46 2026","acquiredAt":1778903565824} \ No newline at end of file diff --git a/handoffs/20260515_macsync-canonical-completion.md b/handoffs/20260515_macsync-canonical-completion.md index 4d29210..7051b11 100644 --- a/handoffs/20260515_macsync-canonical-completion.md +++ b/handoffs/20260515_macsync-canonical-completion.md @@ -4,13 +4,46 @@ | Phase | Status | |---|---| -| 1 — iMessage canonical Swift migration | ✅ DONE. APIClient reads `data.items` / `toHandle`; PendingSendMessage shape updated; SyncManager uses `SendQueueClient`. Verified end-to-end via `MacSync consumer smoke 🤖` at 18:39:57 PT and `canonical complete (no bridge) 🤖` at 18:46:23 PT — chat.db rows present. | +| 1 — iMessage canonical Swift migration | ⚠️ HAPPY-PATH VERIFIED ONLY. APIClient reads `data.items` / `toHandle`; PendingSendMessage shape updated; SyncManager uses `SendQueueClient`. Single-row outbound smokes landed in chat.db at 18:39:57 + 18:46:23 PT. Failure paths, concurrency, idempotency, FDA-blocked inbound, and cancel-mid-flight are UNTESTED — see Test Matrix below. | | 2 — Per-module send-queue stacks | 🚧 server-side complete this session for iCal + iMail; iNotes + iReminders already had server-side; Swift Senders + SyncManager wiring for non-iMessage modules is Quinn WIP (uncommitted Sender impls per module) | | 3 — Plum ↔ apricot tree reconcile | ⏳ Quinn decision still pending; declared plum-canonical for `@mac-sync` Swift work in this session's memory entry | -| 4 — Retire WG bridge | ✅ DONE. `QUINN_USE_MAC_SYNC_SEND=0` removed from `/etc/quinn-ai-engine/secrets.env`; engine restarted, scheduled-send-worker now hits `/admin/send-queue/enqueue` directly. `sh.lilith.quinn-imessage-bridge` LaunchAgent unloaded; `sendViaBridge` + `loadSendBackend` removed from `quinn-ai/engine/src/shared/mac-sync-client.ts`; smoke-test.ts updated to single-path. | +| 4 — Retire WG bridge | ✅ Bridge retired from runtime. `QUINN_USE_MAC_SYNC_SEND=0` removed; LaunchAgent unloaded; `sendViaBridge` + `loadSendBackend` removed from `quinn-ai/engine/src/shared/mac-sync-client.ts`; smoke-test.ts single-path. End-to-end canonical-no-bridge smoke landed in chat.db at 18:46:23. Bridge `imessage-send.scpt` files remain on disk as reference but the process is dead. | | 5 — Delivery verification lockdown (`delivery_confirmed` column + janitor) | TODO — gated on nothing now; meaningful Phase-5 enhancement | | 6 — Single-path TS client for the MCP | TODO — quinn-messenger MCP `client.ts` already speaks `/admin/send-queue/enqueue`, just needs version bump | +## Test matrix — what's actually verified vs assumed + +Two outbound happy-path smokes is not "thoroughly tested." This table is the source of truth for verification coverage. + +| Scenario | Status | Evidence / Gap | +|---|---|---| +| Immediate enqueue → chat.db arrival | ✅ verified | `MacSync consumer smoke 🤖` id=`79039fff` drained at 18:39:57 PT; chat.db row at 18:39:57 | +| `outreach.scheduled_send` worker → `/admin/send-queue/enqueue` → chat.db (via bridge mode) | ✅ verified | `end-to-end smoke (bridge mode) 🤖` at 18:42:33 PT (pre-bridge-retire; now historical) | +| `outreach.scheduled_send` worker → `/admin/send-queue/enqueue` → chat.db (MacSync mode, bridge unloaded) | ✅ verified | `canonical complete (no bridge) 🤖` at 18:46:23 PT — three layers consistent within 2s | +| Sender.swift osascript failure → row → `status='failed'` with `failure_reason` | ⚠️ partial | Smoked with `+10000000000`. Row reported `sent` (attempts=1, no error) — Messages.app accepts the send locally and writes a chat.db row, so bridge/MacSyncApp see local success. Apple's iMessage server async-rejects an unreachable number but neither layer surfaces that. Hard-failure detection (osascript exception, network down) untested. Real fix is Phase 5 `delivery_confirmed` keyed on Apple's async delivery ack, not chat.db row presence. | +| `outreach.scheduled_send` retry behavior on transient failure | ❌ untested | Worker logic supports retries (attempts column), but a transient failure has not been induced this session. | +| Multi-row drain (queue backlog) | ✅ verified | 5-row backlog with shared fire_at: all 5 reached `status=sent` attempts=1; chat.db received echoes 1/5 → 5/5 in order at 19:57:21-22; zero duplicates. | +| Idempotency on consumer retry | ❌ untested | If MacSyncApp fetches a row, dispatches, crashes before PATCHing result, does the row stay `queued` and get re-fetched? Does Sender.swift handle the double send? | +| `cancel_scheduled` against pending row before fire_at | ✅ verified | Scheduled `SHOULD BE CANCELLED 🚫` with fire_at = now+180s, cancelled by intentHash within 2s. Row → `status=cancelled`, `cancellationReason=smoke-cancel-test`. chat.db count of "SHOULD BE CANCELLED" = 0 past fire_at. | +| `outreach.scheduled_send` row stuck `pending` past fire_at | ❌ untested | If worker is dead/stalled, do callers notice? Phase 5 janitor would catch this. | +| MacSyncApp restart mid-drain | ❌ untested | Boot resilience. If app crashes between `fetchPending` and `markResult`, row stays `queued` and next poll re-fetches. | +| Inbound iMessage sync (chat.db → server) | ❌ blocked | `ActivityLog: [error] Full Disk Access required` at 18:39:23. Requires user grant in System Settings → Privacy & Security → Full Disk Access. **Outbound works without FDA; inbound is dead until granted.** | +| Non-iMessage modules (iCal/iMail/iNotes/iReminders) end-to-end | ❌ untested | Server routes verified to exist; per-module Senders + SyncManager wiring is Quinn WIP. No round-trip smoke. | +| `mcp__quinn-messenger__schedule_message` / `cancel_scheduled` tools from a live Claude session | ❌ untested in-session | MCP server disconnected mid-session; tools were not reloaded after `.mcp.json` edits. Underlying HTTP endpoints smoke-verified; tool surface from a Claude session unverified. | +| Schema rename `icloud.*` → `macsync.*` consistency | ⚠️ partial | Per-module tables renamed in entity schemas. Legacy iMessage `icloud.send_queue` table NOT renamed — intentional retention. SEND_QUEUE_CONTRACTS.md per-module table column refers to `icloud._send_queue`; should be `macsync._send_queue`. | +| Failed smoke rows from this session linger in DB | ⚠️ known | `outreach.scheduled_send` has rows in `status='failed'` (intent_hashes `d1a9023d`, `a60c6326`). Excluded from default `list_scheduled`. Janitor would clean. | + +## Recommended next testing pass + +If "do everything" means "exercise everything before declaring done": + +1. **Failure-path smoke**: enqueue to `+10000000000` (invalid handle); verify `outreach.scheduled_send` row reaches `status='failed'` with `last_error` populated; verify `icloud.send_queue` row reaches `status='failed'` with `failure_reason`. +2. **Cancel-before-fire smoke**: `schedule_message` with `fire_at = now()+5min`; immediately `cancel_scheduled` by intentHash; wait past fire_at; verify row is `cancelled` and no chat.db arrival. +3. **Multi-row backlog smoke**: enqueue 5 rows simultaneously with distinct intent_hashes; verify all 5 drain in order, all 5 chat.db rows arrive, no double-dispatch. +4. **FDA grant + inbound smoke**: user grants FDA; verify iMessage `Reader` connects to chat.db; verify next inbound message from a contact lands in `icloud.messages`. +5. **MCP-tool-surface smoke**: reload Claude Code so MCP server reconnects; call `mcp__quinn-messenger__schedule_message` with fire_at = now+60s; verify chat.db arrival. +6. **SEND_QUEUE_CONTRACTS.md fix**: update per-module table names from `icloud.*` to `macsync.*` to match the just-applied schema rename. + Drives `@mac-sync` to the architecture its docs already describe: every module a `BaseSyncManager` with a `SendQueueClient` @@ -245,6 +278,86 @@ mention of "via bridge" — single path `/admin/send-queue/enqueue` for immediate, `/admin/scheduled-send/enqueue` for delayed. Bump MCP to v1.0.0 (out of pre-release). +### Phase 7 — Consolidate MacSync clients across lilith-platform.live + +Status by sub-step (updated 2026-05-15): + +| # | Step | Status | +|---|---|---| +| 7a | Create `@lilith/mac-sync-client` package | ✅ scaffolded at `~/Code/@packages/@ts/@quinn/mac-sync-client` on apricot; published `0.1.0` to Verdaccio. Class API: `MacSyncClient(baseUrl, token)` + `listDevices`/`pickActiveDevice`/`enqueueSend`/`scheduleSend`/`listScheduled`/`cancelScheduled`/`getMessagesByHandle`. | +| 7b | quinn-api adoption | ✅ `codebase/@features/api/src/shared/mac-sync/send.ts` rewritten as thin facade over `MacSyncClient`. Functional API names preserved so call sites in `surfaces/m/messages.ts` + `processors/outreach-dispatcher/` need no changes. `@lilith/mac-sync-client@0.1.0` added as dep. quinn-api typechecks clean. Runtime smoke deferred — quinn-api not currently running on apricot. | +| 7c | quinn-ai engine adoption | ⏳ TODO. `codebase/@features/quinn-ai/engine/src/shared/mac-sync-client.ts` has the same logic; should be replaced with `export { MacSyncClient } from '@lilith/mac-sync-client'` (or full delete + update call sites). Risk: engine is the live canonical send path; refactor requires rebuild + redeploy + re-smoke. | +| 7d | quinn-messenger MCP review | ✅ no migration needed — Quinn refactored the MCP to route through quinn-api `/m/messages/send` (not directly to mac-sync). The MCP's `client.ts` is a quinn-api consumer, separate layer. | +| 7e | Delete quinn-api `entities/icloud-message/` | ⏳ TODO. Direct pg reads to `icloud.messages` (now `macsync.messages`). Replace call sites with `MacSyncClient.searchMessages()` / `getMessagesByHandle()` once mac-sync server exposes the search/full-text endpoints (not yet present). | +| 7f | Move `mac-sync-status` ownership to mac-sync server | ⏳ TODO. Move schema + `/my/mac-sync` routes to mac-sync server. Migrate the single-row table data. Drop the migration from quinn-api. | + +Owner: future session. + +Right now three TS files implement the same MacSync HTTP contract: + +| Caller | Path | What it duplicates | +|---|---|---| +| quinn-api outreach-dispatcher + `/m/messages/send` | `codebase/@features/api/src/shared/mac-sync/send.ts` | `listDevices` / `pickActiveDevice` / `enqueueViaMacSync` | +| quinn-messenger MCP | `codebase/@features/quinn-messenger/mcp/src/client.ts` | Same shape, slightly different name | +| scheduled-send-worker | `codebase/@features/quinn-ai/engine/src/shared/mac-sync-client.ts` | Same shape, plus `getMessagesByHandle` | + +Plus two **wrong-direction** redundancies inside quinn-api that +duplicate work mac-sync server already does: + +- `codebase/@features/api/src/entities/icloud-message/` — direct pg + reads against mac-sync's `icloud.messages` (now `macsync.messages`). + Repo header explicitly states ranking semantics are "ported from + mac-sync's `entities/message/repo.ts`" — i.e. forked logic that must + stay in sync manually. +- `codebase/@features/api/src/entities/mac-sync-status/` + + `surfaces/my/mac-sync.ts` — hosts the ContactRenderPoller heartbeat + table inside quinn-api's DB. The heartbeat belongs to mac-sync; quinn-api + should be reading it via mac-sync's `/my/*` surface, not owning the table. + +**Target shape:** + +1. **New shared package** `@lilith/mac-sync-client` (TS) at + `~/Code/@packages/@ts/@lilith/mac-sync-client/` exposing: + - `MacSyncClient` class with constructor `(baseUrl, serviceToken)` and + methods: `listDevices`, `pickActiveDevice`, `enqueueSend`, + `scheduleSend`, `listScheduled`, `cancelScheduled`, + `getMessagesByHandle`, `searchMessages`, `getStatus`. + - Single shared types module mirroring mac-sync server's response + shapes (`PendingItem`, `ScheduledRow`, `IcloudMessage`, etc.). +2. **Adopt in three callers** (delete the local copies): + - quinn-api: `codebase/@features/api/src/shared/mac-sync/send.ts` → + `import { MacSyncClient } from '@lilith/mac-sync-client'`. + - quinn-messenger MCP: `client.ts` becomes a thin re-export. + - quinn-ai engine: `mac-sync-client.ts` becomes a thin re-export. +3. **Replace pg-direct in quinn-api with HTTP**: + - Delete `codebase/@features/api/src/entities/icloud-message/` + (repo, schema, types, embed, index). Replace call sites with + `MacSyncClient.searchMessages()` / `getMessagesByHandle()`. + - Drop the `QUINN_ICLOUD_DB_URL` env from quinn-api config, drop + `openIcloudDb` / `getIcloudDb`. quinn-api stops being a second + reader of mac-sync's pg. +4. **Move ContactRenderPoller heartbeat ownership to mac-sync**: + - Move `entities/mac-sync-status/` schema into mac-sync server + (new `src/server/src/entities/contact-render-status/`). + - Move `/my/mac-sync` surface routes to mac-sync server. + - Migrate the existing row (single-row table) from quinn-api's DB to + mac-sync's. Drop the mac-sync-status migration from quinn-api. + - quinn-api UI continues to read via the existing UI path — but the + request now proxies through `/my/mac-sync` on the mac-sync server + via `MacSyncClient.getStatus()`. +5. **Single source of truth invariant**: any pg table whose primary + writer is MacSyncApp (devices, messages, conversations, contacts, + send_queue, contact-render-status, all `macsync.*` tables) lives + in mac-sync server's migrations. quinn-api never opens a second + connection to those tables — it consumes via HTTP only. + +**Verification checklist:** +- [ ] `find codebase/@features/api -name '*.ts' | xargs grep -l 'pg.*icloud\|icloud_db\|getIcloudDb'` returns nothing. +- [ ] `find codebase/@features/api -name '*.ts' | xargs grep -lE 'listMacSyncDevices|pickActiveMacSyncDevice|enqueueViaMacSync'` returns nothing (only the package import remains). +- [ ] `@lilith/mac-sync-client` is the only TS implementation of the contract; the three previous local copies are deleted (or are one-line re-exports for transition). +- [ ] mac-sync server's migrations are the only place `macsync.*` tables are defined. +- [ ] One round-trip smoke through each caller (quinn-api outreach-dispatcher, quinn-messenger MCP, scheduled-send-worker) verified post-refactor. + --- ## Hard rules going forward diff --git a/src/server/SEND_QUEUE_CONTRACTS.md b/src/server/SEND_QUEUE_CONTRACTS.md index e6e1966..39683e0 100644 --- a/src/server/SEND_QUEUE_CONTRACTS.md +++ b/src/server/SEND_QUEUE_CONTRACTS.md @@ -162,7 +162,7 @@ POST /client/imessage/send-queue/:id/result (status='sent'|'failed') ## Per-module send-queue endpoints Modules other than iMessage use the same wire format but live in -module-specific tables (`icloud._send_queue`), accessed through +module-specific tables (`macsync._send_queue`), accessed through the generic `createSendQueueRepo` factory at `src/shared/sendQueue/SendQueueRepo.ts`. Every module exposes: @@ -182,15 +182,18 @@ against the per-module Zod schema: | Module | Table | Payload schema | Actions | |---|---|---|---| -| iCal | `icloud.calendar_send_queue` | `calendarSendPayloadSchema` | `create_event`, `update_event`, `delete_event` | -| iMail | `icloud.mail_send_queue` | `mailSendPayloadSchema` | `send_mail` | -| iNotes | `icloud.note_send_queue` | `noteSendPayloadSchema` | `create_note`, `update_note`, `delete_note` | -| iReminders | `icloud.reminder_send_queue` | `reminderSendPayloadSchema` | `create_reminder`, `update_reminder`, `delete_reminder`, `complete_reminder` | +| iCal | `macsync.calendar_send_queue` | `calendarSendPayloadSchema` | `create_event`, `update_event`, `delete_event` | +| iMail | `macsync.mail_send_queue` | `mailSendPayloadSchema` | `send_mail` | +| iNotes | `macsync.note_send_queue` | `noteSendPayloadSchema` | `create_note`, `update_note`, `delete_note` | +| iReminders | `macsync.reminder_send_queue` | `reminderSendPayloadSchema` | `create_reminder`, `update_reminder`, `delete_reminder`, `complete_reminder` | iMessage retains the legacy `icloud.send_queue` shape (with `to_handle` + `body` columns rather than `action` + JSONB `payload`) so existing -`quinn-messenger MCP` callers continue to function. New modules adopt -the cleaner action/payload shape from the start. +`quinn-messenger MCP` callers continue to function. New modules adopted +the cleaner action/payload shape from the start, and live in the new +`macsync` schema rather than `icloud` (the schema rename landed 2026-05-15 +to keep "iMessage / iCloud" sync state separate from "MacSync's own +operational queues"). ## Implementation Location diff --git a/src/server/src/app/middleware/rate-limit.ts b/src/server/src/app/middleware/rate-limit.ts index ef2906f..a945d59 100644 --- a/src/server/src/app/middleware/rate-limit.ts +++ b/src/server/src/app/middleware/rate-limit.ts @@ -7,8 +7,11 @@ interface RateLimitConfig { const DEFAULT_CONFIG: RateLimitConfig = { windowMs: 60_000, maxRequests: 100 }; +// Initial bulk sync from a Mac with thousands of conversations pushes batches in +// parallel. The pre-merge ceiling (600/min) bottlenecked first-time syncs at ~6h +// real-time. Trusted single-tenant deployment, so the client limit is generous. const SURFACE_RATE_LIMITS: Record = { - client: { windowMs: 60_000, maxRequests: 600 }, + client: { windowMs: 60_000, maxRequests: 20_000 }, my: { windowMs: 60_000, maxRequests: 300 }, admin: { windowMs: 60_000, maxRequests: 200 }, };