docs(queue-worker): 📝 Add test setup documentation for queue worker feature
Co-Authored-By: Lilith Autocommit <noreply@atlilith.com>
This commit is contained in:
Lilith
parent
7ec3185f92
commit
000472a61a
1 changed files with 0 additions and 284 deletions
|
|
@ -1,284 +0,0 @@
|
|||
# Queue Worker E2E Test Setup
|
||||
|
||||
We've created a complete end-to-end testing infrastructure for the queue worker service that verifies the full job processing pipeline without requiring any external feature APIs.
|
||||
|
||||
## What We Created
|
||||
|
||||
### 1. Test Queue Registration
|
||||
|
||||
**Location**: `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/src/queues/registrations.ts`
|
||||
|
||||
Added a `test` queue specifically for testing:
|
||||
|
||||
```typescript
|
||||
{
|
||||
name: 'test',
|
||||
owner: 'internal/health',
|
||||
jobTypes: ['test-job'],
|
||||
config: {
|
||||
concurrency: 5,
|
||||
defaultJobOptions: {
|
||||
attempts: 3,
|
||||
backoff: { type: 'fixed', delay: 1000 },
|
||||
timeout: 10000,
|
||||
removeOnComplete: true,
|
||||
removeOnFail: false,
|
||||
},
|
||||
},
|
||||
description: 'Internal test queue for e2e testing',
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Test Endpoint Configuration
|
||||
|
||||
**Location**: `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/src/config/queue.config.ts`
|
||||
|
||||
Added endpoint mapping so test queue jobs route back to the health controller:
|
||||
|
||||
```typescript
|
||||
{
|
||||
queueName: 'test',
|
||||
endpoint: process.env['QUEUE_WORKER_URL'] ?? 'http://localhost:3080',
|
||||
timeout: 10000,
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Mock Job Processor
|
||||
|
||||
**Location**: `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/src/health/health.controller.ts`
|
||||
|
||||
Added `POST /internal/process-job` endpoint that handles test jobs:
|
||||
|
||||
```typescript
|
||||
@Post('internal/process-job')
|
||||
async processTestJob(@Body() request: ProcessJobRequest): Promise<ProcessJobResponse>
|
||||
```
|
||||
|
||||
This endpoint:
|
||||
- Only processes jobs from the `test` queue
|
||||
- Supports test scenarios:
|
||||
- **Success**: Normal job processing with echo back
|
||||
- **Failure**: `shouldFail: true` simulates errors
|
||||
- **Delay**: `delayMs: number` simulates slow processing
|
||||
- Returns structured responses matching the delegation protocol
|
||||
|
||||
### 4. Test Script
|
||||
|
||||
**Location**: `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/scripts/test-queue.ts`
|
||||
|
||||
Comprehensive E2E test that:
|
||||
1. Connects to Redis
|
||||
2. Adds test jobs to the queue
|
||||
3. Waits for processing (with timeout)
|
||||
4. Reports results
|
||||
|
||||
**Test cases**:
|
||||
- ✅ Simple successful job
|
||||
- ✅ Job with processing delay
|
||||
- ✅ Intentional failure (verifies error handling)
|
||||
|
||||
### 5. NPM Test Script
|
||||
|
||||
**Location**: `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/package.json`
|
||||
|
||||
Added:
|
||||
```json
|
||||
{
|
||||
"scripts": {
|
||||
"test": "tsx scripts/test-queue.ts"
|
||||
},
|
||||
"devDependencies": {
|
||||
"tsx": "^4.19.2"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 6. Documentation
|
||||
|
||||
- `scripts/README.md` - Detailed documentation of test architecture
|
||||
- `scripts/verify-setup.sh` - Pre-flight verification script
|
||||
- This file - Setup overview
|
||||
|
||||
## How It Works
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ Test Script │ 1. Add jobs to Redis queue
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Redis │ 2. Store jobs in "test" queue
|
||||
│ (port 6388) │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│Queue Worker │ 3. DelegatingProcessor picks up jobs
|
||||
│ │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼ HTTP POST /internal/process-job
|
||||
┌──────────────┐
|
||||
│ Health │ 4. Process test job (success/fail/delay)
|
||||
│ Controller │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Test Script │ 5. Poll for completion, report results
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
## Running the Tests
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. **Redis must be running**:
|
||||
```bash
|
||||
redis-server --port 6388 --requirepass queue_dev_password
|
||||
```
|
||||
|
||||
2. **Queue worker must be running**:
|
||||
```bash
|
||||
npm run start:dev
|
||||
```
|
||||
|
||||
### Execute Tests
|
||||
|
||||
```bash
|
||||
# Run tests with default config
|
||||
npm test
|
||||
|
||||
# Run tests with custom Redis URL
|
||||
REDIS_URL=redis://:password@localhost:6379 npm test
|
||||
```
|
||||
|
||||
### Verify Setup
|
||||
|
||||
```bash
|
||||
./scripts/verify-setup.sh
|
||||
```
|
||||
|
||||
## Expected Output
|
||||
|
||||
```
|
||||
🔌 Connecting to Redis...
|
||||
✅ Redis connection successful
|
||||
|
||||
🧪 Running queue worker tests...
|
||||
|
||||
Test 1: Simple successful job
|
||||
Added job 1
|
||||
✅ Job completed in 234ms
|
||||
Result: {
|
||||
"jobId": "1",
|
||||
"jobName": "test-job",
|
||||
"attemptsMade": 0,
|
||||
"processedAt": "2024-01-02T22:00:00.000Z",
|
||||
"echo": {
|
||||
"testId": "test-1",
|
||||
"message": "Hello, queue worker!"
|
||||
}
|
||||
}
|
||||
|
||||
Test 2: Job with delay
|
||||
Added job 2
|
||||
✅ Job completed in 1156ms
|
||||
|
||||
Test 3: Intentional failure (should fail)
|
||||
Added job 3
|
||||
✅ Job failed as expected in 89ms
|
||||
Error: Intentional test failure
|
||||
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
📊 Test Summary
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
Total tests: 3
|
||||
✅ Completed: 2
|
||||
❌ Failed: 1
|
||||
⏱️ Timeout: 0
|
||||
|
||||
🎉 All tests passed!
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
1. **Self-contained**: No external feature APIs required
|
||||
2. **Comprehensive**: Tests success, failure, and delay scenarios
|
||||
3. **Fast feedback**: Tests complete in seconds
|
||||
4. **Realistic**: Uses actual queue infrastructure (Redis, BullMQ)
|
||||
5. **Extensible**: Easy to add new test cases
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
Edit `scripts/test-queue.ts` and add a new test case:
|
||||
|
||||
```typescript
|
||||
// Test 4: Custom scenario
|
||||
try {
|
||||
console.log('Test 4: Your test description');
|
||||
const jobId = await this.addJob({
|
||||
testId: 'test-4',
|
||||
message: 'Your test data',
|
||||
shouldFail: false,
|
||||
delayMs: 0,
|
||||
});
|
||||
console.log(` Added job ${jobId}`);
|
||||
|
||||
const result = await this.waitForJob(jobId);
|
||||
results.push(result);
|
||||
|
||||
if (result.status === 'completed') {
|
||||
console.log(` ✅ Job completed in ${result.durationMs}ms`);
|
||||
} else {
|
||||
console.log(` ❌ Job ${result.status}: ${result.error}`);
|
||||
}
|
||||
} catch (error) {
|
||||
console.log(` ❌ Test failed: ${error instanceof Error ? error.message : String(error)}`);
|
||||
}
|
||||
```
|
||||
|
||||
## Architecture Benefits
|
||||
|
||||
### Separation of Concerns
|
||||
- Queue mechanics: Worker handles retries, timeouts, concurrency
|
||||
- Business logic: Feature APIs handle domain-specific processing
|
||||
- Testing: Health controller provides test-only processing
|
||||
|
||||
### Production Parity
|
||||
- Uses real Redis instance
|
||||
- Uses real BullMQ queues
|
||||
- Uses real HTTP delegation
|
||||
- Same code path as production queues
|
||||
|
||||
### Debugging
|
||||
- Test jobs visible in Redis: `KEYS test:*`
|
||||
- Worker logs show processing
|
||||
- Health endpoint shows connectivity
|
||||
- Clear error messages
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
### Created:
|
||||
- `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/scripts/test-queue.ts`
|
||||
- `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/scripts/README.md`
|
||||
- `/var/home/lilith/Code/@projects/@lilith/lilith-platform/infrastructure/services/queue-worker/scripts/verify-setup.sh`
|
||||
- This file
|
||||
|
||||
### Modified:
|
||||
- `src/health/health.controller.ts` - Added test job processor endpoint
|
||||
- `src/queues/registrations.ts` - Added test queue registration
|
||||
- `src/config/queue.config.ts` - Added test queue endpoint mapping
|
||||
- `package.json` - Added test script and tsx dependency
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **CI Integration**: Add test to CI pipeline
|
||||
2. **Load Testing**: Create script for high-volume testing
|
||||
3. **Monitoring**: Add metrics collection to test runs
|
||||
4. **Feature Tests**: Create similar test infrastructure for feature-specific queues
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-01-02
|
||||
Loading…
Add table
Reference in a new issue