macsync/docs/data-flow.md

Data flow

Three diagrams trace bytes through the system: inbound (Mac to server), outbound (web to Mac via send queue), and bootstrap (cold start).

Inbound: Mac to server

How a new iMessage, photo, calendar event etc. reaches the database. iCal is shown; every other module follows the same shape.

sequenceDiagram
    autonumber
    participant Timer as Timer (interval s)
    participant SM as ICalSync.SyncManager
    participant Reader as CalendarReader
    participant API as ICalSync.APIClient
    participant Hono as Hono /client/ical/*
    participant PG as PostgreSQL icloud

    Timer->>SM: fire (BaseSyncManager.syncNow)
    SM->>SM: gatedRunCycle -> isAuthorized?
    SM->>Reader: fetchCalendars(), fetchEvents(since: lastSync)
    Reader-->>SM: [SyncCalendarPayload], [SyncEventPayload]
    SM->>API: syncCalendars(payloads)
    API->>Hono: POST /client/ical/calendars (device token)
    Hono->>PG: INSERT ... ON CONFLICT UPDATE
    PG-->>Hono: row count
    Hono-->>API: { synced: N }
    SM->>API: syncEvents(payloads)
    API->>Hono: POST /client/ical/events
    Hono->>PG: upsert + tsvector update via trigger
    Hono-->>API: { synced: N }
    SM->>SM: setLastSync(now), lastSyncCompletedAt = now

Sources:

• BaseSyncManager.runCycle updates the watermark at the end of every cycle: @packages/shared/Sources/MacSyncShared/Sync/BaseSyncManager.swift:123-131.
• Calendar reader is invoked from @packages/ical/Sources/ICalSync/SyncManager.swift:116- (the performSync() override; see Module Reader for the Reader contract).
• API methods: @packages/ical/Sources/ICalSync/APIClient.swift:106-130.
• Server route registration: src/server/src/surfaces/client/ical.ts:59-69.

Other modules' inbound endpoints:

• POST /client/imessage/sync (surfaces/client/imessage.ts:71)
• POST /client/imessage/contacts (surfaces/client/imessage.ts:77)
• POST /client/iphoto/sync (surfaces/client/iphoto.ts:63)
• POST /client/iphoto/albums (surfaces/client/iphoto.ts:69)
• POST /client/iphoto/upload/:localIdentifier (surfaces/client/iphoto.ts:75) — binary upload
• POST /client/imail/sync (surfaces/client/imail.ts:40)
• POST /client/ireminders/sync (surfaces/client/ireminders.ts:32)
• POST /client/inotes/sync (surfaces/client/inotes.ts:29)

Outbound: web to Mac via SendQueueClient

How a user-initiated write in the React SPA reaches Calendar.app / Reminders.app / Notes.app. iCal shown; reminders and notes are structurally identical.

sequenceDiagram
    autonumber
    participant SPA as React SPA
    participant My as Hono /my/calendar/*
    participant Send as features/ical/sendService
    participant Admin as Hono /admin/calendar-send-queue
    participant Repo as createSendQueueRepo<br/>(icloud.calendar_send_queue)
    participant PG as PostgreSQL
    participant Client as Hono /client/ical/send-queue
    participant SQC as Mac SendQueueClient
    participant Sender as CalendarSender (EventKit)
    participant EK as EKEventStore

    SPA->>My: POST /my/calendar/events { ... }
    My->>Send: enqueueEvent(action, payload, deviceId)
    Send->>Repo: enqueue(...)
    Repo->>PG: INSERT INTO icloud.calendar_send_queue (status='queued')
    PG-->>Repo: { id }
    Repo-->>SPA: 200 { id }

    Note over SQC: Timer fires every 60s
    SQC->>Client: GET /client/ical/send-queue/pending (device token)
    Client->>Repo: listPending(deviceId)
    Repo->>PG: SELECT ... WHERE status='queued' ORDER BY created_at
    Repo-->>Client: rows
    Client-->>SQC: [PendingItem]
    loop each item
        SQC->>Sender: apply(item)
        Sender->>EK: save(_:span:) / remove(_:span:)
        EK-->>Sender: result
        Sender-->>SQC: .sent | .failed(reason)
        SQC->>Client: POST /client/ical/send-queue/:id/result
        Client->>Repo: markResult({ id, status, error? })
        Repo->>PG: UPDATE status='sent'|'failed', sent_at=now()
    end

Sources:

• /my/calendar/events write routes: src/server/src/surfaces/my/calendar.ts:80, 90, 102.
• sendService.enqueueEvent and the repo wiring: src/server/src/features/ical/sendService.ts:16-19 (factory call).
• SendQueueRepo SQL: src/server/src/shared/sendQueue/SendQueueRepo.ts:69-127.
• Mac-side polling: SendQueueClient.drainOnce() (@packages/shared/Sources/MacSyncShared/Sync/SendQueueClient.swift:93-125).
• Calendar transport adapter: @packages/ical/Sources/ICalSync/SyncManager.swift:56-65.
• Sender: @packages/ical/Sources/ICalSync/Sender.swift (CalendarSender.apply).

The iMessage outbound flow is the same loop but talks to the legacy icloud.send_queue table via IMessageSendTransport (@packages/imessage/Sources/IMessageSync/SendQueueAdapter.swift) and applies via AppleScript instead of EventKit (@packages/imessage/Sources/IMessageSync/Sender.swift).

Bootstrap: cold start

What happens when MacSyncApp launches for the first time on a new Mac.

sequenceDiagram
    autonumber
    participant App as MacSyncApp
    participant Cfg as ConfigFile (~/.config/com.lilith.mac-sync/config.json)
    participant Dev as DeviceRegistration<br/>(MacSyncShared)
    participant Hono as Hono /client/devices/*
    participant PG as icloud.devices
    participant Mgrs as SyncManager.shared (x6)
    participant LWS as LocalWebServer :8765
    participant Browser as Browser

    App->>Cfg: load() — read serverURL, deviceName, etc.
    Cfg-->>App: config
    App->>Dev: register(deviceName, model, osVersion)
    Dev->>Hono: POST /client/devices/register (no auth)
    Hono->>PG: INSERT device, return token
    Hono-->>Dev: { deviceId, token }
    Dev->>Dev: persist token to Keychain
    loop poll until ready
        App->>Dev: status check
        Dev->>Hono: GET /client/devices/:id/status (no auth)
        Hono-->>Dev: { ready }
    end
    App->>Mgrs: startSync() on each of the seven
    Mgrs->>Mgrs: gatedRunCycle -> isAuthorized? -> performSync
    Mgrs->>Mgrs: setLastSync(now); persist to UserDefaults
    Mgrs->>Mgrs: didStartSync -> sendQueueClient.start() (iMessage/iCal/iRem/iNotes)
    App->>LWS: start() on port 8765
    Browser->>LWS: GET / (SPA)
    Browser->>LWS: GET /api/settings
    LWS-->>Browser: { serverURL, deviceName, ... }

Sources:

• LocalWebServer route registration: @packages/shared/Sources/MacSyncShared/WebServer/LocalWebServer.swift:42-110. Settings endpoints live on lines 43-57; static SPA fallback on 94-109.
• Device registration is exempt from the device-token middleware: src/server/src/app/server.ts:69-75.
• The base manager loads watermarks in init from UserDefaults: @packages/shared/Sources/MacSyncShared/Sync/BaseSyncManager.swift:67-69.
• didStartSync() is the hook each bidirectional manager overrides to start its SendQueueClient. See iMessage (@packages/imessage/Sources/IMessageSync/SyncManager.swift:170-172), iCal (@packages/ical/Sources/ICalSync/SyncManager.swift:67-69), iReminders, and iNotes (@packages/inotes/Sources/INoteSync/SyncManager.swift:66-72).
• The web SPA is served from the Mac binary's webapp/ resource directory in production builds, or <repo>/web/dist/ in dev (LocalWebServer.swift:137-153).

Calls (iCalls) are read-only: the Swift reader (with WAL snapshot) feeds /client/icalls/sync; the server surface is at /my/calls (and exposed via MCP recent_calls + local LocalWebServer /api/calls). No send queue.

Watermark, time, and idempotency notes

• lastSync is the start time of the previous cycle, not the timestamp of the latest record. Modules apply tolerance (iMail uses ±1 min, see @packages/imail/Sources/IMailSync/SyncManager.swift:42-43) to catch late arrivals.
• Inbound upserts use module-specific external IDs (iMessage guid, iMail Message-ID, iCal eventIdentifier, iNotes noteIdentifier) so retrying a cycle is safe.
• Outbound send-queue items are idempotent on the server side (status='queued' -> 'sent'|'failed' is a one-shot transition); the Mac client is responsible for not double-applying inside a single drain. The isProcessing flag in SendQueueClient (@packages/shared/Sources/MacSyncShared/Sync/SendQueueClient.swift:54, 93-96) guarantees only one drain runs at a time.