queue/e2e

Queue E2E Integration Tests

Comprehensive end-to-end tests for the @queue package ecosystem using Docker and real Redis instances.

Overview

These tests verify the complete queue integration:

• Job enqueuing and processing
• Priority-based job execution
• Retry behavior with backoff strategies
• Peak-hour deferral logic
• Bulk operations
• Advanced scenarios (cancellation, progress tracking, rate limiting)

Prerequisites

• Docker and Docker Compose installed
• Node.js 18+ and pnpm
• Port 6380 available for test Redis instance

Running Tests

Quick Start

# From the @queue root directory
pnpm test:e2e

Watch Mode

pnpm test:e2e:watch

Manual Docker Control

# Start Redis manually
pnpm docker:up

# Run tests against running container
pnpm test:e2e

# View Redis logs
pnpm docker:logs

# Stop Redis
pnpm docker:down

Test Structure

/e2e/docker-compose.yml

Docker Compose configuration for test Redis instance.

Features:

• Redis 7 Alpine (lightweight)
• Persistence disabled (tmpfs) for speed
• Health checks enabled
• Port 6380 to avoid conflicts
• Isolated network

/e2e/setup.ts

Global test setup and teardown.

Responsibilities:

• Start/stop Docker container
• Wait for Redis readiness
• Provide Redis connection helpers
• Clean database between tests
• Utility functions for test assertions

/e2e/queue-integration.spec.ts

Comprehensive test suite covering:

Job Processing

• Basic job enqueuing and processing
• Multiple job handling
• Return value verification

Priority Handling

• Priority-based execution order
• Mixed priority scenarios
• URGENT → HIGH → NORMAL → LOW → BATCH ordering

Retry Behavior

• Exponential backoff retries
• Fixed delay retries
• Max attempts enforcement
• Partial failure scenarios

Peak Hour Deferral

• Peak hour detection (weekdays 4pm-9pm UTC)
• Low/normal priority deferral during peaks
• High/urgent priority bypass
• Delay calculation accuracy

Bulk Operations

• Efficient bulk job addition
• Mixed priority bulk jobs
• Graceful failure handling in bulk

Advanced Scenarios

• Job cancellation
• Progress tracking
• Concurrent processing
• Rate limiting enforcement

Architecture

┌─────────────────────────────────────────────┐
│           E2E Test Suite                    │
│  (queue-integration.spec.ts)                │
└─────────────────┬───────────────────────────┘
                  │
                  │ Uses
                  ▼
┌─────────────────────────────────────────────┐
│          Test Utilities                     │
│  - waitForJobs()                           │
│  - waitFor()                               │
│  - cleanRedis()                            │
│  - getRedisConnection()                    │
└─────────────────┬───────────────────────────┘
                  │
                  │ Manages
                  ▼
┌─────────────────────────────────────────────┐
│      Docker Redis Container                 │
│  - Port: 6380                              │
│  - Health checks enabled                   │
│  - Tmpfs for speed                         │
└─────────────────┬───────────────────────────┘
                  │
                  │ Tests
                  ▼
┌─────────────────────────────────────────────┐
│         @queue Packages                     │
│  - @lilith/queue/core               │
│  - BullMQ integration                      │
│  - Priority logic                          │
│  - Peak-hour utilities                     │
└─────────────────────────────────────────────┘

Test Coverage

Job Processing (5 tests)

✓ Simple job enqueuing and processing ✓ Multiple jobs in order ✓ Job completion with return values

Priority Handling (2 tests)

✓ Jobs processed in priority order ✓ Mixed priority handling

Retry Behavior (3 tests)

✓ Exponential backoff retries ✓ Max attempts enforcement ✓ First retry success

Peak Hour Deferral (5 tests)

✓ Peak hour detection ✓ Low priority deferral ✓ High priority bypass ✓ Delay calculation ✓ Delayed job state

Bulk Operations (3 tests)

✓ Efficient bulk addition ✓ Mixed priority bulk jobs ✓ Graceful bulk failures

Advanced Scenarios (4 tests)

✓ Job cancellation ✓ Progress tracking ✓ Concurrent processing ✓ Rate limiting

Total: 22 comprehensive E2E tests

CI/CD Integration

GitHub Actions Example

name: E2E Tests

on: [push, pull_request]

jobs:
  e2e:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'

      - name: Setup pnpm
        uses: pnpm/action-setup@v2
        with:
          version: 8

      - name: Install dependencies
        run: pnpm install

      - name: Run E2E tests
        run: pnpm test:e2e

GitLab CI Example

e2e-tests:
  stage: test
  image: node:18
  services:
    - docker:dind
  before_script:
    - curl -fsSL https://get.docker.com | sh
    - npm install -g pnpm
    - pnpm install
  script:
    - pnpm test:e2e
  artifacts:
    when: always
    paths:
      - coverage/
    reports:
      junit: test-results.xml

Troubleshooting

Port 6380 Already in Use

# Find process using port
lsof -i :6380

# Kill the process or use different port
# Edit docker-compose.yml to change port mapping

Docker Permission Issues

# Add user to docker group (Linux)
sudo usermod -aG docker $USER
newgrp docker

# Or run with sudo (not recommended)
sudo pnpm test:e2e

Tests Timing Out

# Check if Redis started
docker ps | grep queue-test-redis

# Check Redis logs
docker logs queue-test-redis

# Manually verify Redis
redis-cli -p 6380 ping

Container Not Stopping

# Force remove container
docker rm -f queue-test-redis

# Clean up volumes
docker volume prune

Performance Considerations

Optimization Strategies:

1. Tmpfs Storage: Redis uses in-memory tmpfs, no disk I/O
2. Persistence Disabled: No AOF/RDB snapshots for speed
3. Sequential Execution: Tests run one at a time for clean state
4. Shared Setup: Single container for all tests (no per-test restart)

Expected Timings:

• Docker startup: ~5-10 seconds
• Single test: ~50-500ms
• Full suite: ~30-60 seconds
• Teardown: ~2-5 seconds

Best Practices

Writing New E2E Tests

import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import { Queue, Worker } from 'bullmq';
import { getRedisConnection, waitForJobs } from './setup';

describe('My Feature', () => {
  let queue: Queue;
  let worker: Worker;

  beforeEach(async () => {
    queue = new Queue('my-queue', {
      connection: getRedisConnection(),
    });

    worker = new Worker('my-queue', async (job) => {
      // Process job
      return { result: 'success' };
    }, {
      connection: getRedisConnection(),
    });
  });

  afterEach(async () => {
    await worker.close();
    await queue.close();
  });

  it('should do something', async () => {
    await queue.add('test-job', { data: 'test' });
    await waitForJobs(queue, 'completed', 1);

    const counts = await queue.getJobCounts();
    expect(counts.completed).toBe(1);
  });
});

Key Guidelines

1. Always close workers and queues in afterEach
2. Use waitForJobs() helper instead of arbitrary timeouts
3. Test one thing per test for clarity
4. Use descriptive test names that explain behavior
5. Avoid sleep() - use condition-based waiting
6. Clean state between tests (handled automatically by setup)

Related Documentation

• BullMQ Documentation
• Vitest Documentation
• @lilith/queue/core
• Peak Hour Logic

Support

For issues or questions:

1. Check Docker is running: docker ps
2. Verify Redis connection: redis-cli -p 6380 ping
3. Review test logs: pnpm test:e2e -- --reporter=verbose
4. Check container logs: pnpm docker:logs