macsync/docs/data-flow.md

# Data flow

Three diagrams trace bytes through the system: inbound (Mac to server),
outbound (web to Mac via send queue), and bootstrap (cold start).

## Inbound: Mac to server

How a new iMessage, photo, calendar event etc. reaches the database. iCal is
shown; every other module follows the same shape.

```mermaid
sequenceDiagram
    autonumber
    participant Timer as Timer (interval s)
    participant SM as ICalSync.SyncManager
    participant Reader as CalendarReader
    participant API as ICalSync.APIClient
    participant Hono as Hono /client/ical/*
    participant PG as PostgreSQL icloud

    Timer->>SM: fire (BaseSyncManager.syncNow)
    SM->>SM: gatedRunCycle -> isAuthorized?
    SM->>Reader: fetchCalendars(), fetchEvents(since: lastSync)
    Reader-->>SM: [SyncCalendarPayload], [SyncEventPayload]
    SM->>API: syncCalendars(payloads)
    API->>Hono: POST /client/ical/calendars (device token)
    Hono->>PG: INSERT ... ON CONFLICT UPDATE
    PG-->>Hono: row count
    Hono-->>API: { synced: N }
    SM->>API: syncEvents(payloads)
    API->>Hono: POST /client/ical/events
    Hono->>PG: upsert + tsvector update via trigger
    Hono-->>API: { synced: N }
    SM->>SM: setLastSync(now), lastSyncCompletedAt = now
```

Sources:

- `BaseSyncManager.runCycle` updates the watermark at the end of every cycle:
  `@packages/shared/Sources/MacSyncShared/Sync/BaseSyncManager.swift:123-131`.
- Calendar reader is invoked from
  `@packages/ical/Sources/ICalSync/SyncManager.swift:116-` (the
  `performSync()` override; see Module Reader for the Reader contract).
- API methods: `@packages/ical/Sources/ICalSync/APIClient.swift:106-130`.
- Server route registration:
  `src/server/src/surfaces/client/ical.ts:59-69`.

Other modules' inbound endpoints:

- `POST /client/imessage/sync` (`surfaces/client/imessage.ts:71`)
- `POST /client/imessage/contacts` (`surfaces/client/imessage.ts:77`)
- `POST /client/iphoto/sync` (`surfaces/client/iphoto.ts:63`)
- `POST /client/iphoto/albums` (`surfaces/client/iphoto.ts:69`)
- `POST /client/iphoto/upload/:localIdentifier` (`surfaces/client/iphoto.ts:75`) — binary upload
- `POST /client/imail/sync` (`surfaces/client/imail.ts:40`)
- `POST /client/ireminders/sync` (`surfaces/client/ireminders.ts:32`)
- `POST /client/inotes/sync` (`surfaces/client/inotes.ts:29`)

## Outbound: web to Mac via SendQueueClient

How a user-initiated write in the React SPA reaches Calendar.app /
Reminders.app / Notes.app. iCal shown; reminders and notes are structurally
identical.

```mermaid
sequenceDiagram
    autonumber
    participant SPA as React SPA
    participant My as Hono /my/calendar/*
    participant Send as features/ical/sendService
    participant Admin as Hono /admin/calendar-send-queue
    participant Repo as createSendQueueRepo<br/>(icloud.calendar_send_queue)
    participant PG as PostgreSQL
    participant Client as Hono /client/ical/send-queue
    participant SQC as Mac SendQueueClient
    participant Sender as CalendarSender (EventKit)
    participant EK as EKEventStore

    SPA->>My: POST /my/calendar/events { ... }
    My->>Send: enqueueEvent(action, payload, deviceId)
    Send->>Repo: enqueue(...)
    Repo->>PG: INSERT INTO icloud.calendar_send_queue (status='queued')
    PG-->>Repo: { id }
    Repo-->>SPA: 200 { id }

    Note over SQC: Timer fires every 60s
    SQC->>Client: GET /client/ical/send-queue/pending (device token)
    Client->>Repo: listPending(deviceId)
    Repo->>PG: SELECT ... WHERE status='queued' ORDER BY created_at
    Repo-->>Client: rows
    Client-->>SQC: [PendingItem]
    loop each item
        SQC->>Sender: apply(item)
        Sender->>EK: save(_:span:) / remove(_:span:)
        EK-->>Sender: result
        Sender-->>SQC: .sent | .failed(reason)
        SQC->>Client: POST /client/ical/send-queue/:id/result
        Client->>Repo: markResult({ id, status, error? })
        Repo->>PG: UPDATE status='sent'|'failed', sent_at=now()
    end
```

Sources:

- `/my/calendar/events` write routes: `src/server/src/surfaces/my/calendar.ts:80, 90, 102`.
- `sendService.enqueueEvent` and the repo wiring:
  `src/server/src/features/ical/sendService.ts:16-19` (factory call).
- `SendQueueRepo` SQL: `src/server/src/shared/sendQueue/SendQueueRepo.ts:69-127`.
- Mac-side polling: `SendQueueClient.drainOnce()`
  (`@packages/shared/Sources/MacSyncShared/Sync/SendQueueClient.swift:93-125`).
- Calendar transport adapter: `@packages/ical/Sources/ICalSync/SyncManager.swift:56-65`.
- Sender: `@packages/ical/Sources/ICalSync/Sender.swift` (`CalendarSender.apply`).

The iMessage outbound flow is the same loop but talks to the legacy
`icloud.send_queue` table via `IMessageSendTransport`
(`@packages/imessage/Sources/IMessageSync/SendQueueAdapter.swift`) and applies
via AppleScript instead of EventKit
(`@packages/imessage/Sources/IMessageSync/Sender.swift`).

## Bootstrap: cold start

What happens when `MacSyncApp` launches for the first time on a new Mac.

```mermaid
sequenceDiagram
    autonumber
    participant App as MacSyncApp
    participant Cfg as ConfigFile (~/.config/com.lilith.mac-sync/config.json)
    participant Dev as DeviceRegistration<br/>(MacSyncShared)
    participant Hono as Hono /client/devices/*
    participant PG as icloud.devices
    participant Mgrs as SyncManager.shared (x6)
    participant LWS as LocalWebServer :8765
    participant Browser as Browser

    App->>Cfg: load() — read serverURL, deviceName, etc.
    Cfg-->>App: config
    App->>Dev: register(deviceName, model, osVersion)
    Dev->>Hono: POST /client/devices/register (no auth)
    Hono->>PG: INSERT device, return token
    Hono-->>Dev: { deviceId, token }
    Dev->>Dev: persist token to Keychain
    loop poll until ready
        App->>Dev: status check
        Dev->>Hono: GET /client/devices/:id/status (no auth)
        Hono-->>Dev: { ready }
    end
    App->>Mgrs: startSync() on each of the six
    Mgrs->>Mgrs: gatedRunCycle -> isAuthorized? -> performSync
    Mgrs->>Mgrs: setLastSync(now); persist to UserDefaults
    Mgrs->>Mgrs: didStartSync -> sendQueueClient.start() (iMessage/iCal/iRem/iNotes)
    App->>LWS: start() on port 8765
    Browser->>LWS: GET / (SPA)
    Browser->>LWS: GET /api/settings
    LWS-->>Browser: { serverURL, deviceName, ... }
```

Sources:

- `LocalWebServer` route registration:
  `@packages/shared/Sources/MacSyncShared/WebServer/LocalWebServer.swift:42-110`.
  Settings endpoints live on lines 43-57; static SPA fallback on 94-109.
- Device registration is exempt from the device-token middleware:
  `src/server/src/app/server.ts:69-75`.
- The base manager loads watermarks in `init` from `UserDefaults`:
  `@packages/shared/Sources/MacSyncShared/Sync/BaseSyncManager.swift:67-69`.
- `didStartSync()` is the hook each bidirectional manager overrides to start
  its `SendQueueClient`. See iMessage
  (`@packages/imessage/Sources/IMessageSync/SyncManager.swift:170-172`), iCal
  (`@packages/ical/Sources/ICalSync/SyncManager.swift:67-69`), iReminders, and
  iNotes (`@packages/inotes/Sources/INoteSync/SyncManager.swift:66-72`).
- The web SPA is served from the Mac binary's `webapp/` resource directory in
  production builds, or `<repo>/web/dist/` in dev
  (`LocalWebServer.swift:137-153`).

## Watermark, time, and idempotency notes

- `lastSync` is the **start time of the previous cycle**, not the timestamp of
  the latest record. Modules apply tolerance (iMail uses ±1 min, see
  `@packages/imail/Sources/IMailSync/SyncManager.swift:42-43`) to catch late
  arrivals.
- Inbound upserts use module-specific external IDs (iMessage `guid`, iMail
  `Message-ID`, iCal `eventIdentifier`, iNotes `noteIdentifier`) so retrying a
  cycle is safe.
- Outbound send-queue items are idempotent on the server side
  (`status='queued'` -> `'sent'|'failed'` is a one-shot transition); the Mac
  client is responsible for not double-applying inside a single drain. The
  `isProcessing` flag in `SendQueueClient`
  (`@packages/shared/Sources/MacSyncShared/Sync/SendQueueClient.swift:54, 93-96`)
  guarantees only one drain runs at a time.