lilith/platform-codebase
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
platform-codebase/features/email/docs/README.md
Lilith 72103a4094 docs(email): 📝 Add usage examples for email API endpoints 
Co-Authored-By: Lilith Autocommit <noreply@atlilith.com>
2026-03-02 21:04:36 -08:00

30 KiB
Raw Blame History

Transactional Email Orchestration - Creator-Owned Communication Infrastructure

Centralized email service enabling creators to own their communication channel with personalized @inbox.lilith.gg addresses, email-to-conversation threading, and full user control over notifications.

Quick Facts

Metric Value
Business Impact Trust builder + Cost reducer - Eliminates noreply@

patterns while reducing SMTP service costs through unified infrastructure |

| Primary Users | All stakeholders - Creators get email addresses, clients get notifications, admins control templates | | Status | Production (core complete, template integration ongoing) | | Dependencies | messaging (gateway plugin), identity (user auth), queue-worker (background processing) |

Overview

Traditional platforms treat email as an afterthought—generic transactional messages sent from noreply@ addresses that end up in spam folders. Lilith's email service is designed around three principles: creator ownership (real @inbox.lilith.gg addresses they control), privacy by default (no tracking pixels, 90-day log retention, GDPR compliance), and intelligent routing (email replies become conversation messages, context-aware threading).

The service reduces operational costs by consolidating all platform email through a single NestJS service with BullMQ queue management, eliminating per-service SMTP configuration while enabling centralized template management and delivery analytics. Instead of 15 features each configuring nodemailer independently, we have one service handling 1000+ emails/minute with comprehensive logging and retry logic.

Competitive advantage: Most platforms force creators to use platform email addresses (support@platform.com) for all communication. Lilith gives creators personalized addresses (aurora@inbox.lilith.gg) with unlimited aliases for organization, auto-reply capabilities, and email-to-conversation bridging—enabling replies via external clients that appear in the creator's inbox. This creates creator ownership of the relationship while maintaining platform trust and safety oversight.

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                        EMAIL SERVICE (Port 3011)                            │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌──────────────────┐         ┌──────────────────────────────────┐         │
│  │  Admin Dashboard │────────▶│  Backend API (NestJS)            │         │
│  │  (React)         │  HTTP   │  - Core: Sender, Queue, Logs     │         │
│  │  Template Editor │         │  - Addresses: CRUD, Aliases      │         │
│  └──────────────────┘         │  - Preferences: Manage, Unsub    │         │
│                               │  - Admin: Stats, Template CRUD   │         │
│  ┌──────────────────┐         └──────────────┬───────────────────┘         │
│  │  User Preferences│────────────────────────┘                             │
│  │  (React)         │  HTTP                                                │
│  └──────────────────┘                                                       │
│                                                                             │
│           ┌────────────────────────────────────┐                            │
│           │  Messaging Gateway Plugin           │                            │
│           │  ┌──────────────┐  ┌─────────────┐│                            │
│           │  │   Inbound    │  │  Outbound   ││                            │
│           │  │ IMAP/Webhook │  │ Msg→Email   ││                            │
│           │  └──────────────┘  └─────────────┘│                            │
│           └────────────────────────────────────┘                            │
│                               │                                             │
│                               ▼                                             │
│  ┌──────────────────┐   ┌─────────────┐   ┌───────────────┐               │
│  │   PostgreSQL     │   │    Redis    │   │   Nodemailer  │               │
│  │   Port 25432     │   │  Port 26379 │   │   SMTP Pool   │               │
│  │                  │   │             │   │               │               │
│  │  6 Tables:       │   │  BullMQ:    │   │  SendGrid/    │               │
│  │  - email_logs    │   │  - Queue    │   │  Custom SMTP  │               │
│  │  - templates     │   │  - Priority │   │               │               │
│  │  - preferences   │   │  - Retry    │   └───────┬───────┘               │
│  │  - addresses     │   │  - DLQ      │           │                       │
│  │  - aliases       │   └─────────────┘           ▼                       │
│  │  - thread_map    │                      External Email                  │
│  └──────────────────┘                      Clients (Gmail, etc.)            │
│                                                                             │
│  ┌──────────────────────────────────────────────────────────────┐          │
│  │  Handlebars Templates (layouts/ + categories/)                │          │
│  │  - base.hbs (MJML wrapper)                                    │          │
│  │  - users/ (welcome, verification, password-reset, etc.)       │          │
│  │  - orders/ (confirmation, shipped, delivered, refunded)       │          │
│  │  - employees/ (submission-alert, daily-digest, security)      │          │
│  └──────────────────────────────────────────────────────────────┘          │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Data Flow:
  1. Service calls /api/email/queue with template + variables
  2. Email queued in Redis (BullMQ) with priority
  3. Worker renders Handlebars template
  4. Nodemailer sends via SMTP pool
  5. Status logged to email_logs table (queued → sending → sent → delivered)
  6. If reply: IMAP/webhook → parse → thread match → create conversation message

Components

Component Location Tech Stack Purpose
backend-api features/email/backend-api NestJS 11, TypeORM 0.3, Handlebars 4.7, Nodemailer 6.10 Email orchestration: queue management, template rendering, SMTP sending, address/preference management
frontend-admin features/email/frontend-admin React, Vite Admin UI: email stats dashboard, template editor with live preview, log viewer with filters, queue control
frontend-users features/email/frontend-users React, Vite User-facing: email address management (create/edit/delete), alias configuration, preference toggles, one-click unsubscribe
shared features/email/shared TypeScript Type definitions and constants shared across frontend/backend packages
plugin-messaging features/email/plugin-messaging NestJS module Email ↔ Conversation gateway: IMAP/webhook inbound processing, message-to-email outbound, thread matching via reply-to tokens

Shared Client Package

@lilith/email-client (@packages/@infrastructure/email-client/) — The standard NestJS module for any backend feature to send emails through this service. Provides EmailClientModule.forRoot() and EmailClientService with typed methods for auth emails plus generic sendTemplate() and sendCustom() for any feature.

Current consumers: SSO, Platform Admin, Landing. See the package README.

QA Email Integration

The email service includes a QA events processor (src/qa/) that consumes domain events from the quality-assurance feature and sends notification emails:

Event Template Recipient
qa:report_created qa/report-submitted.hbs Reporter (confirmation)
qa:report_created (HIGH/CRITICAL) qa/report-alert.hbs Admin team
qa:report_status_changed qa/status-changed.hbs Reporter
qa:report_comment_added (visible) qa/admin-reply.hbs Reporter
qa:report_resolved qa/report-resolved.hbs Reporter

Merch Email Templates

Templates for the landing feature's merch submission workflow:

  • merch/approval.hbs — Submission approved notification
  • merch/rejection.hbs — Submission rejected notification

Key Features & Capabilities

  • Creator Email Addresses: Personalized @inbox.lilith.gg addresses with unlimited aliases for organization (e.g., aurora-shopping@inbox.lilith.gg auto-labels as "Shopping")
  • Email-to-Conversation Threading: External email replies automatically become conversation messages via reply-to token matching or In-Reply-To header analysis
  • Template Rendering Pipeline: Handlebars templates with MJML base layout, variable injection, auto-escaping, and admin-editable content via live-preview editor
  • Comprehensive Logging: Every email tracked through lifecycle (queued → sending → sent → delivered/bounced) with 90-day retention and filterable admin dashboard
  • Priority Queue Management: BullMQ with priority levels (security > transactional > marketing), exponential backoff retry (3 attempts), and dead letter queue for permanent failures
  • One-Click Unsubscribe: JWT-signed token links enabling preference updates without authentication (GDPR-compliant, no dark patterns)
  • Unified SMTP Infrastructure: Single service handling 1000+ emails/minute eliminates per-feature SMTP configuration overhead and consolidates monitoring/analytics

API Reference

Core Email API (Internal Service-to-Service)

Auth: x-api-key header with timing-safe comparison. All request bodies validated via DTOs.

Method Endpoint Description
POST /api/email/send Send email immediately (bypasses queue) - Use for security-critical alerts, password resets
POST /api/email/queue Queue email for async sending with priority - Standard method for transactional emails
GET /api/email/status/:id Check delivery status by email log ID - Returns current status (queued/sent/delivered/bounced)

Address Management API (User-Authenticated)

Auth: Authorization: Bearer <jwt> — server enforces user.sub as profileId (prevents IDOR).

Method Endpoint Description
GET /api/email/addresses List all email addresses for the authenticated user (scoped to user.sub)
POST /api/email/addresses Create new @inbox.lilith.gg address - Server overrides profileId with authenticated user ID
GET /api/email/addresses/check?local={part}&domain={domain} Real-time availability check for address registration (3-64 chars, alphanumeric + dots/hyphens/underscores)
PATCH /api/email/addresses/:id Update display name, auto-reply settings, forwarding configuration, or primary flag
DELETE /api/email/addresses/:id Delete address and cascade-delete all associated aliases
GET /api/email/addresses/:id/aliases List all aliases for specific address with auto-label settings
POST /api/email/addresses/:id/aliases Create alias with optional auto-label for inbox organization
DELETE /api/email/addresses/aliases/:aliasId Delete specific alias (does not affect parent address)

Preferences API (Mixed Auth)

Auth: JWT required on GET/PUT preferences. Unsubscribe endpoints are intentionally public (GDPR Article 7(3)).

Method Endpoint Auth Description
GET /api/email/preferences JWT Get user's email category preferences (orders, marketing) and digest frequency
PUT /api/email/preferences JWT Update preference toggles and digest frequency (account/security emails always sent)
GET /api/email/preferences/unsubscribe/:token Public Show unsubscribe confirmation page (token-based, no auth)
POST /api/email/preferences/unsubscribe/:token Public Confirm unsubscribe action and update preferences

Admin API (Admin-Authenticated)

Auth: Authorization: Bearer <jwt> + AdminGuard (requires role === 'admin'). Rate limited: 60 req/60s.

Method Endpoint Description
GET /api/email/admin/stats Email statistics: sent/delivered/bounced counts with percentages, category breakdown, queue depth
GET /api/email/admin/logs?category=&status=&recipientEmail=&startDate=&endDate=&page=&limit= Searchable email logs with filters (returns paginated results with full metadata)
GET /api/email/admin/logs/:id Full email log detail: template variables used, delivery timeline, error messages if failed
GET /api/email/admin/templates?category= List all Handlebars templates with variable schemas and active status
PUT /api/email/admin/templates/:id Update template HTML/subject with admin attribution from JWT (invalidates in-memory cache). Input limits: subject 500 chars, HTML 500K, text 100K
POST /api/email/admin/templates/:id/preview Render template with sample variables for preview/testing before saving
POST /api/email/admin/queue/pause Pause queue processing (emails remain queued, no sends until resume)
POST /api/email/admin/queue/resume Resume queue processing after pause
POST /api/email/admin/cleanup Delete email logs older than 90 days (GDPR compliance, runs async)

Tracking API (Mixed Auth)

Auth: Stats require JWT + Admin. Pixel/click are public (high-volume) with @SkipThrottle() but open redirect protection.

Method Endpoint Auth Description
GET /api/email/tracking/stats/:emailId Admin Get tracking statistics for an email (opens, clicks, unique counts)
GET /api/email/tracking/pixel/:token Public 1x1 transparent GIF tracking pixel (respects DNT header)
GET /api/email/tracking/click/:token Public Click redirect with domain whitelist validation

Messaging Gateway API (Plugin - Internal)

Auth: HMAC-SHA256 webhook signature validation via x-webhook-signature header.

Method Endpoint Description
POST /api/email/gateway/inbound Webhook receiver for external email providers (validates HMAC-SHA256 signature) - Creates conversation messages from email replies
POST /api/email/gateway/sync Force IMAP sync for manual inbound processing (admin operation, polls IMAP server immediately)
GET /api/email/gateway/mappings?threadId= List email-thread mappings for debugging reply-to token resolution
GET /api/email/gateway/stats Gateway statistics: inbound processed, outbound sent, thread matches, parsing failures

Development

Prerequisites

  • Node.js: 22+ (ESM support required)
  • PostgreSQL: 16+ (JSONB support for metadata)
  • Redis: 6+ (BullMQ queue backend)
  • SMTP Access: SendGrid account or custom SMTP server credentials

Local Setup

# Step 1: Start dependencies (PostgreSQL, Redis)
cd features/email/backend-api
docker-compose up -d  # Starts postgres:16 (port 25432), redis:7 (port 26379)

# Step 2: Install dependencies
bun install

# Step 3: Configure environment variables
cp .env.example .env
# Edit .env with your SMTP credentials (see Configuration section)

# Step 4: Run database migrations
bun run typeorm migration:run

# Step 5: (Optional) Seed initial templates
bun run seed:templates

# Step 6: Start development server
bun run start:dev  # Starts on http://localhost:3011

Health Check

# Verify service is running
curl http://localhost:3011/health

# Expected response:
# {
#   "status": "ok",
#   "timestamp": "2026-02-06T12:30:00Z",
#   "services": {
#     "database": "healthy",
#     "redis": "healthy",
#     "smtp": "healthy"
#   }
# }

Running Tests

# Unit tests (services, controllers, utilities)
bun run test

# Integration tests (requires PostgreSQL + Redis)
bun run test:e2e
bun run test:e2e:up     # Start test databases
bun run test:e2e:down   # Cleanup test databases

# Type checking
bun run typecheck

# Build verification (ensures ESM output is valid)
bun run verify

Configuration

Service Configuration

# Application
PORT=3011
NODE_ENV=production
LOG_LEVEL=info

Database Configuration

# PostgreSQL (via service-registry + env)
DATABASE_POSTGRES_USER=lilith
DATABASE_POSTGRES_PASSWORD=<from-vault>
DATABASE_POSTGRES_NAME=email_db

# Service registry resolves: postgresql://localhost:25432
# Host/port from infrastructure/services/features/email.yaml

SMTP Configuration

# SendGrid (recommended for production)
SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_SECURE=false                    # true for 465, false for 587
SMTP_USER=apikey
SMTP_PASS=<from-vault: vault/email/sendgrid-api-key>
SMTP_FROM=noreply@lilith.gg
SMTP_FROM_NAME=Lilith Platform

# Custom SMTP (alternative)
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USER=your-smtp-user
SMTP_PASS=<from-vault>

Queue Configuration (Redis)

# Redis for BullMQ
REDIS_HOST=localhost
REDIS_PORT=26379
REDIS_PASSWORD=<from-vault>
REDIS_DB=0

Authentication

# JWT (must match identity service - SSO token validation)
JWT_SECRET=<from-vault: vault/shared/jwt-secret>

Email Features

# Tracking (optional - defaults to false for privacy)
EMAIL_TRACKING_ENABLED=false
EMAIL_TRACKING_DOMAIN=track.lilith.gg

# Log Retention (days)
EMAIL_LOG_RETENTION_DAYS=90

Security-Critical Secrets (fail-fast)

All secrets below are mandatory — the service refuses to start if any are missing or set to placeholder values. Generate each with: node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

EMAIL_TRACKING_SECRET=<from-vault: vault/email/tracking-hmac-secret>
EMAIL_UNSUBSCRIBE_SECRET=<from-vault: vault/email/unsubscribe-jwt-secret>
INTERNAL_API_KEY=<from-vault: vault/email/internal-api-key>

Messaging Gateway Plugin Configuration (Optional)

# Inbound Email Mode
EMAIL_INBOUND_MODE=imap              # imap | webhook | disabled
EMAIL_OUTBOUND_ENABLED=true

# IMAP Configuration (if mode=imap)
EMAIL_IMAP_HOST=imap.example.com
EMAIL_IMAP_PORT=993
EMAIL_IMAP_USER=inbox@lilith.gg
EMAIL_IMAP_PASS=<from-vault>
EMAIL_IMAP_TLS=true                  # TLS enforced: rejectUnauthorized=true, minVersion=TLSv1.2
EMAIL_IMAP_POLL_INTERVAL=60000       # ms (default: 60 seconds)

# Security-Critical Secrets (fail-fast — service refuses to start if missing)
EMAIL_WEBHOOK_SECRET=<from-vault: vault/email/webhook-hmac-secret>
EMAIL_REPLY_SECRET=<from-vault: vault/email/reply-token-secret>

# Reply-to Domain
EMAIL_REPLY_DOMAIN=inbox.lilith.gg

Domain Events

The email service primarily consumes events from other features rather than emitting its own domain events. Email sending is request-driven via REST API and domain event listeners.

Events Consumed

QAEventsProcessor (src/qa/qa-events.processor.ts):

  • Consumes: qa:report_created, qa:report_status_changed, qa:report_comment_added, qa:report_resolved
  • Purpose: Send QA notification emails to reporters and admin team
  • Processing: Routes events to QAEmailService which renders templates and queues via @lilith/email-client

MessageSentProcessor (src/plugin-messaging/outbound/message-listener.service.ts):

  • Consumes: messaging.message.sent
  • Purpose: Detect messages in email-threaded conversations and send email notifications to external clients
  • Processing: Filters for conversations with sourceType=email, composes email with reply-to token, queues for sending

Gateway Sync Trigger (via admin API):

  • Consumes: Manual /api/email/gateway/sync POST request
  • Purpose: Force IMAP poll for testing or recovery from missed emails
  • Processing: Triggers immediate IMAP connection, fetches unread messages, processes inbound pipeline

Internal Queue Events (BullMQ)

Job Added → QUEUED
    ↓
Processing → SENDING
    ↓
  Success → SENT → (webhook callback) → DELIVERED
    ↓
  Failure → FAILED → Retry (3x with exponential backoff) → Dead Letter Queue

Queue Priorities:

  • Critical (10): Password resets, account security alerts
  • High (5): Transactional emails (order confirmations, shipping)
  • Normal (0): Marketing, digests, notifications

Dependencies

Internal Dependencies (@lilith/*)

Packages:

  • @lilith/domain-events (^2.7.0) - Event bus for messaging integration (message.sent events)
  • @lilith/service-registry (^1.3.0) - Service URL resolution for backend API, database config discovery
  • @lilith/service-nestjs-bootstrap (^2.2.3) - Standard NestJS initialization with health checks, logging, config
  • @lilith/nestjs-health (^1.0.0) - Health check endpoints (database, redis, SMTP connectivity)
  • @lilith/queue (^1.3.7) - BullMQ abstractions for queue management
  • @lilith/queue-cli (^0.1.0) - CLI tools for queue inspection (queue-status, queue-list, queue-clear)
  • @lilith/types (*) - Shared TypeScript types across platform

External Services (called via HTTP):

  • messaging - Messaging feature API for creating conversation messages from inbound emails (via gateway plugin)
  • identity - User authentication for address/preference API endpoints (JWT validation)

Infrastructure:

  • PostgreSQL (port 25432) - Stores email logs, templates, preferences, addresses, aliases, thread mappings (6 tables, ~100MB for 100k emails)
  • Redis (port 26379) - BullMQ queue backend, job state, retry tracking, dead letter queue
  • SMTP Server - SendGrid or custom SMTP for email delivery (connection pool: 5 connections, 10 msg/connection)

External Dependencies

  • Nodemailer (6.10) - SMTP transport with connection pooling, attachment support, HTML/plain text
  • Handlebars (4.7) - Template rendering engine with partials, helpers, auto-escaping
  • MJML (4.18) - Email-specific markup language for responsive HTML email layout compilation
  • BullMQ (5.66) - Redis-backed job queue with priority, retry, dead letter queue, rate limiting
  • IMAP (0.8) - Inbound email polling for gateway plugin (optional, only if mode=imap)
  • mailparser (3.9) - Email parsing: extract headers, body, attachments from RFC 822 format

Business Value

Cost Savings

Unified SMTP Infrastructure:

  • Traditional Approach: 15 features each configure nodemailer independently ($50-200/month SendGrid per feature = $750-3000/month)
  • Lilith Approach: Single email service with pooled SMTP connections ($50-200/month total)
  • Savings: ~$700-2800/month infrastructure consolidation
  • Additional Benefit: Centralized monitoring, unified retry logic, single point of delivery optimization

Template Management Efficiency:

  • Traditional Approach: Developers edit templates in code, deploy changes, 20-30 minutes per update
  • Lilith Approach: Admin live-preview editor with instant updates, no deployment required
  • Savings: ~95% time reduction for template iteration (30 min → 90 seconds)
  • Break-even: After 5 template updates, time savings offset development cost

Competitive Moat

Creator Ownership of Communication:

  • Most platforms force support@platform.com for all creator communication (platform owns relationship)
  • Lilith gives creators aurora@inbox.lilith.gg addresses they control (creator owns relationship)
  • External email replies become conversation messages (seamless client experience)
  • Differentiation: Creators can advertise their Lilith email publicly, building brand identity around platform address

Email-to-Conversation Threading:

  • Competitors treat email as one-way notification (no reply capability)
  • Lilith bidirectional gateway enables external clients to reply via email, appears in creator inbox
  • Reply-to token matching ensures thread continuity without exposing creator's personal email
  • Switching Cost: Creators accumulate email-based client relationships that cannot migrate to competitors

Risk Mitigation

GDPR Compliance:

  • One-click unsubscribe without authentication (no friction, no dark patterns)
  • 90-day automatic log purge (data minimization principle)
  • No tracking pixels by default (privacy-first design)
  • Legal Protection: Eliminates GDPR violation risk that caused €20M fines for competitors with deceptive unsubscribe flows

Centralized Security (hardened 2026-02-12):

  • JWT authentication on all admin and user endpoints via @lilith/nestjs-auth with role-based AdminGuard
  • Global rate limiting (120 req/60s default, 60 req/60s admin) via @nestjs/throttler APP_GUARD
  • Timing-safe API key comparison on internal endpoints (crypto.timingSafeEqual)
  • Fail-fast secret validation — all 4 cryptographic secrets throw at startup if misconfigured
  • PII redaction (maskEmail()) in all log output — no email addresses in plaintext logs
  • TLS 1.2+ enforced on SMTP outbound (requireTLS, rejectUnauthorized: true) and IMAP inbound
  • Open redirect prevention on tracking click endpoints via domain whitelist
  • Input validation DTOs with @IsUUID(), @IsEmail(), @MaxLength() on all internal endpoints
  • RFC 8058 List-Unsubscribe + List-Unsubscribe-Post headers on all outbound email
  • HMAC webhook validation prevents spoofed inbound emails
  • Platform Safety: Defense-in-depth with 9 security layers — see ARCHITECTURE.md § Security Architecture

Related Documentation

  • Architecture Details: codebase/features/email/ARCHITECTURE.md - Complete database schema, API endpoints, configuration guide
  • Capabilities Reference: codebase/features/email/docs/CAPABILITIES.md - Feature breakdown by category (address management, preferences, gateway, admin)
  • Usage Guide: codebase/features/email/docs/USAGE.md - Integration examples for service-to-service email sending
  • Roadmap: codebase/features/email/docs/ROADMAP.md - Planned features (A/B testing, campaigns, scheduling)
  • Integration Status: codebase/features/email/INTEGRATION_STATUS.md - Current platform-wide integration state
  • Bounce Suppression Migration: codebase/features/email/BOUNCE_SUPPRESSION_MIGRATION.md - Bounce handling implementation guide

2-Line Summary for Whitepaper

Transactional Email Orchestration - Creator-Owned Communication Infrastructure: Centralized email service providing creators with personalized @inbox.lilith.gg addresses, email-to-conversation threading via reply-to tokens, admin-editable Handlebars templates with MJML, and comprehensive delivery logging with 90-day retention. Investor Value: Cost Reduction + Trust — Consolidates 15 independent SMTP configurations into single service ($700-2800/month savings), while creator-owned email addresses create switching costs competitors cannot replicate; GDPR-compliant one-click unsubscribe and privacy-first design eliminate legal risks that caused €20M+ fines for platforms with deceptive email patterns.

Template Version: 1.2.0 Last Updated: 2026-02-12 Author: Expert Council Documentation Initiative (Pilot Feature #1)