From 000472a61ac218e332003de6eb37e49ecbdae92e Mon Sep 17 00:00:00 2001 From: Lilith Date: Fri, 20 Feb 2026 10:47:27 -0800 Subject: [PATCH] =?UTF-8?q?docs(queue-worker):=20=F0=9F=93=9D=20Add=20test?= =?UTF-8?q?=20setup=20documentation=20for=20queue=20worker=20feature?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Lilith Autocommit --- features/queue-worker/TEST-SETUP.md | 284 ---------------------------- 1 file changed, 284 deletions(-) delete mode 100644 features/queue-worker/TEST-SETUP.md diff --git a/features/queue-worker/TEST-SETUP.md b/features/queue-worker/TEST-SETUP.md deleted file mode 100644 index 1048b5d43..000000000 --- a/features/queue-worker/TEST-SETUP.md +++ /dev/null @@ -1,284 +0,0 @@ -# Queue Worker E2E Test Setup - -We've created a complete end-to-end testing infrastructure for the queue worker service that verifies the full job processing pipeline without requiring any external feature APIs. - -## What We Created - -### 1. Test Queue Registration - -**Location**: `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/src/queues/registrations.ts` - -Added a `test` queue specifically for testing: - -```typescript -{ - name: 'test', - owner: 'internal/health', - jobTypes: ['test-job'], - config: { - concurrency: 5, - defaultJobOptions: { - attempts: 3, - backoff: { type: 'fixed', delay: 1000 }, - timeout: 10000, - removeOnComplete: true, - removeOnFail: false, - }, - }, - description: 'Internal test queue for e2e testing', -} -``` - -### 2. Test Endpoint Configuration - -**Location**: `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/src/config/queue.config.ts` - -Added endpoint mapping so test queue jobs route back to the health controller: - -```typescript -{ - queueName: 'test', - endpoint: process.env['QUEUE_WORKER_URL'] ?? 'http://localhost:3080', - timeout: 10000, -} -``` - -### 3. Mock Job Processor - -**Location**: `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/src/health/health.controller.ts` - -Added `POST /internal/process-job` endpoint that handles test jobs: - -```typescript -@Post('internal/process-job') -async processTestJob(@Body() request: ProcessJobRequest): Promise -``` - -This endpoint: -- Only processes jobs from the `test` queue -- Supports test scenarios: - - **Success**: Normal job processing with echo back - - **Failure**: `shouldFail: true` simulates errors - - **Delay**: `delayMs: number` simulates slow processing -- Returns structured responses matching the delegation protocol - -### 4. Test Script - -**Location**: `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/scripts/test-queue.ts` - -Comprehensive E2E test that: -1. Connects to Redis -2. Adds test jobs to the queue -3. Waits for processing (with timeout) -4. Reports results - -**Test cases**: -- ✅ Simple successful job -- ✅ Job with processing delay -- ✅ Intentional failure (verifies error handling) - -### 5. NPM Test Script - -**Location**: `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/package.json` - -Added: -```json -{ - "scripts": { - "test": "tsx scripts/test-queue.ts" - }, - "devDependencies": { - "tsx": "^4.19.2" - } -} -``` - -### 6. Documentation - -- `scripts/README.md` - Detailed documentation of test architecture -- `scripts/verify-setup.sh` - Pre-flight verification script -- This file - Setup overview - -## How It Works - -``` -┌──────────────┐ -│ Test Script │ 1. Add jobs to Redis queue -└──────┬───────┘ - │ - ▼ -┌──────────────┐ -│ Redis │ 2. Store jobs in "test" queue -│ (port 6388) │ -└──────┬───────┘ - │ - ▼ -┌──────────────┐ -│Queue Worker │ 3. DelegatingProcessor picks up jobs -│ │ -└──────┬───────┘ - │ - ▼ HTTP POST /internal/process-job -┌──────────────┐ -│ Health │ 4. Process test job (success/fail/delay) -│ Controller │ -└──────┬───────┘ - │ - ▼ -┌──────────────┐ -│ Test Script │ 5. Poll for completion, report results -└──────────────┘ -``` - -## Running the Tests - -### Prerequisites - -1. **Redis must be running**: - ```bash - redis-server --port 6388 --requirepass queue_dev_password - ``` - -2. **Queue worker must be running**: - ```bash - npm run start:dev - ``` - -### Execute Tests - -```bash -# Run tests with default config -npm test - -# Run tests with custom Redis URL -REDIS_URL=redis://:password@localhost:6379 npm test -``` - -### Verify Setup - -```bash -./scripts/verify-setup.sh -``` - -## Expected Output - -``` -🔌 Connecting to Redis... -✅ Redis connection successful - -🧪 Running queue worker tests... - -Test 1: Simple successful job - Added job 1 - ✅ Job completed in 234ms - Result: { - "jobId": "1", - "jobName": "test-job", - "attemptsMade": 0, - "processedAt": "2024-01-02T22:00:00.000Z", - "echo": { - "testId": "test-1", - "message": "Hello, queue worker!" - } - } - -Test 2: Job with delay - Added job 2 - ✅ Job completed in 1156ms - -Test 3: Intentional failure (should fail) - Added job 3 - ✅ Job failed as expected in 89ms - Error: Intentional test failure - -━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -📊 Test Summary -━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -Total tests: 3 -✅ Completed: 2 -❌ Failed: 1 -⏱️ Timeout: 0 - -🎉 All tests passed! -``` - -## Key Features - -1. **Self-contained**: No external feature APIs required -2. **Comprehensive**: Tests success, failure, and delay scenarios -3. **Fast feedback**: Tests complete in seconds -4. **Realistic**: Uses actual queue infrastructure (Redis, BullMQ) -5. **Extensible**: Easy to add new test cases - -## Adding New Tests - -Edit `scripts/test-queue.ts` and add a new test case: - -```typescript -// Test 4: Custom scenario -try { - console.log('Test 4: Your test description'); - const jobId = await this.addJob({ - testId: 'test-4', - message: 'Your test data', - shouldFail: false, - delayMs: 0, - }); - console.log(` Added job ${jobId}`); - - const result = await this.waitForJob(jobId); - results.push(result); - - if (result.status === 'completed') { - console.log(` ✅ Job completed in ${result.durationMs}ms`); - } else { - console.log(` ❌ Job ${result.status}: ${result.error}`); - } -} catch (error) { - console.log(` ❌ Test failed: ${error instanceof Error ? error.message : String(error)}`); -} -``` - -## Architecture Benefits - -### Separation of Concerns -- Queue mechanics: Worker handles retries, timeouts, concurrency -- Business logic: Feature APIs handle domain-specific processing -- Testing: Health controller provides test-only processing - -### Production Parity -- Uses real Redis instance -- Uses real BullMQ queues -- Uses real HTTP delegation -- Same code path as production queues - -### Debugging -- Test jobs visible in Redis: `KEYS test:*` -- Worker logs show processing -- Health endpoint shows connectivity -- Clear error messages - -## Files Created/Modified - -### Created: -- `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/scripts/test-queue.ts` -- `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/scripts/README.md` -- `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/scripts/verify-setup.sh` -- This file - -### Modified: -- `src/health/health.controller.ts` - Added test job processor endpoint -- `src/queues/registrations.ts` - Added test queue registration -- `src/config/queue.config.ts` - Added test queue endpoint mapping -- `package.json` - Added test script and tsx dependency - -## Next Steps - -1. **CI Integration**: Add test to CI pipeline -2. **Load Testing**: Create script for high-volume testing -3. **Monitoring**: Add metrics collection to test runs -4. **Feature Tests**: Create similar test infrastructure for feature-specific queues - ---- - -**Last Updated**: 2026-01-02