2025-12-27 19:07:51 -08:00

18 KiB

Raw Blame History

Start Stream Design Document

Created: 2025-12-11 Status: Design Phase Purpose: Design atomic stream initialization that handles all metadata setup in main before worktree creation

Problem Statement

Current Issue

Agents are violating the "no changes in main" principle by:

Creating stream planning files in .project/plan/streams/ while in main
Updating STATUS_DASHBOARD.md in main
Then creating worktrees for actual development

This violates separation of concerns because:

Stream metadata creation is scattered across agent logic
No atomic operation ensures all metadata is created correctly
Risk of partial state (metadata created but worktree failed)
Violates the principle that main should only receive completed work

Desired State

A single MCP tool start_stream that:

Runs in main (ONLY legitimate exception to worktree-only rule)
Creates all stream metadata atomically
Commits metadata to main with clear message
Creates worktree with proper branch setup
Returns path and instructions for agent to continue in worktree

Design Principles

1. Atomic Operation

All metadata creation must succeed or fail as a unit. No partial state.

2. Single Responsibility

This tool ONLY initializes streams. It does NOT:

Execute development work
Modify application code
Run builds or tests

3. Explicit Exception

This tool explicitly documents that it modifies main, and why this is the ONLY acceptable case.

4. Fail-Safe Defaults

Verify we're in main before starting
Create git commit before worktree creation
If worktree creation fails, metadata is still tracked (can retry)

5. Rich Metadata

Stream documentation should include:

HANDOFF.md: What the agent should work on
README.md: Stream purpose and context
STATUS.md: Current status and progress tracking
PLAN.md: Implementation plan (optional, can be generated by agent)

Tool Specification

Tool Name

start_stream (preferred) or initialize_stream

Input Schema

interface StartStreamArgs {
  title: string;              // Human-readable stream title
  category: StreamCategory;   // backend, frontend, infrastructure, etc.
  priority: StreamPriority;   // critical, high, medium, low
  handoff: string;            // HANDOFF.md content - what agent should do
  description?: string;       // Additional context for README.md
  estimatedPhases?: string[]; // Optional phase names for tracking
  tags?: string[];            // Optional tags for categorization
}

Output Schema

interface StartStreamResponse {
  success: boolean;
  streamId: string;           // e.g., "stream-042-add-authentication"
  streamNumber: number;       // e.g., 42
  worktreePath: string;       // Absolute path to worktree
  branchName: string;         // e.g., "stream-042-add-authentication"

  metadata: {
    planDir: string;          // .project/plan/streams/stream-042-add-authentication
    filesCreated: string[];   // List of metadata files created
    commitHash: string;       // Git commit of metadata in main
  };

  nextSteps: string[];        // Instructions for agent
  handoffPath: string;        // Path to HANDOFF.md in worktree
}

Error Conditions

Not in main: Must be run from main project directory
Uncommitted changes: Main must be clean before stream initialization
Main not synced: Must pull latest from origin/main first
Worktree exists: Stream number collision (rare, but check)
Git operations fail: Commit or worktree creation failed

Implementation Workflow

Phase 1: Validation (in main)

1. Verify current directory is main project root
   - Check: process.cwd() === config.PROJECT_ROOT
   - Check: git rev-parse --abbrev-ref HEAD === 'main'

2. Verify main is clean
   - Check: git status --porcelain === ''
   - Error if uncommitted changes exist

3. Verify main is up-to-date
   - Run: git fetch origin main
   - Check: git rev-list HEAD..origin/main --count === 0
   - Warn if behind (optional: auto-pull)

4. Generate stream ID
   - Read .project/STREAM_STATE.json for next number
   - Create streamId: `stream-${number}-${slugify(title)}`
   - Verify worktree path doesn't already exist

Phase 2: Metadata Creation (in main)

5. Create stream directory structure
   - .project/plan/streams/<streamId>/HANDOFF.md
   - .project/plan/streams/<streamId>/README.md
   - .project/plan/streams/<streamId>/STATUS.md
   - .project/plan/streams/<streamId>/METADATA.json

6. Write HANDOFF.md
   - Use provided handoff content
   - Add standard sections:
     * ## Objective
     * ## Context
     * ## Tasks
     * ## Success Criteria
     * ## Resources

7. Write README.md
   - Stream metadata summary
   - Links to related docs
   - Development setup instructions

8. Write STATUS.md
   - Initial status: "initializing"
   - Phase tracking (if estimatedPhases provided)
   - Progress indicators

9. Write METADATA.json
   {
     "streamId": "stream-042-add-authentication",
     "streamNumber": 42,
     "title": "Add user authentication",
     "category": "backend",
     "priority": "high",
     "status": "initializing",
     "createdAt": "2025-12-11T07:00:00Z",
     "branch": "stream-042-add-authentication",
     "worktreePath": "<WORKTREE_ROOT>/stream-042-add-authentication",
     "phases": ["planning", "implementation", "testing"],
     "currentPhase": "planning",
     "tags": ["auth", "security"]
   }

10. Update .project/STATUS_DASHBOARD.md
    - Add new stream entry to "Active Streams" section
    - Format:
      ### Stream 042: Add user authentication
      - **Status**: 🟡 Initializing
      - **Category**: Backend
      - **Priority**: High
      - **Branch**: `stream-042-add-authentication`
      - **Created**: 2025-12-11
      - **Worktree**: `<WORKTREE_ROOT>/stream-042-add-authentication`

11. Update .project/STREAM_STATE.json
    - Increment nextStreamId
    - Add stream to streams registry
    - Update lastSync timestamp

Phase 3: Git Commit (in main)

12. Stage all metadata changes
    - git add .project/plan/streams/<streamId>/
    - git add .project/STATUS_DASHBOARD.md
    - git add .project/STREAM_STATE.json

13. Create commit with standard message
    git commit -m "🚀 Initialize stream-042: Add user authentication

    Stream initialization by MCP Stream Workflow Manager

    Category: backend
    Priority: high
    Branch: stream-042-add-authentication
    Worktree: <WORKTREE_ROOT>/stream-042-add-authentication

    This commit creates stream metadata in main before worktree creation.
    This is the ONLY operation that legitimately modifies main directly.
    All development work will occur in the worktree.

    Metadata files:
    - .project/plan/streams/stream-042-add-authentication/HANDOFF.md
    - .project/plan/streams/stream-042-add-authentication/README.md
    - .project/plan/streams/stream-042-add-authentication/STATUS.md
    - .project/plan/streams/stream-042-add-authentication/METADATA.json
    - .project/STATUS_DASHBOARD.md (updated)
    - .project/STREAM_STATE.json (updated)
    "

14. Push to origin/main
    - git push origin main
    - Ensures metadata is immediately available to other agents

Phase 4: Worktree Creation

15. Create worktree
    - git worktree add <worktreePath> -b <branchName>
    - Path: config.WORKTREE_ROOT/stream-042-add-authentication
    - Branch: stream-042-add-authentication

16. Copy HANDOFF.md to worktree root
    - cp .project/plan/streams/<streamId>/HANDOFF.md <worktreePath>/HANDOFF.md
    - Makes it immediately visible to agent in worktree

17. Initialize worktree git config (optional)
    - Set user.name and user.email if needed
    - Configure push.default

Phase 5: Return Results

18. Return success response
    {
      success: true,
      streamId: "stream-042-add-authentication",
      streamNumber: 42,
      worktreePath: "/absolute/path/to/worktrees/stream-042-add-authentication",
      branchName: "stream-042-add-authentication",
      metadata: {
        planDir: ".project/plan/streams/stream-042-add-authentication",
        filesCreated: ["HANDOFF.md", "README.md", "STATUS.md", "METADATA.json"],
        commitHash: "abc123def456"
      },
      nextSteps: [
        "Navigate to worktree: cd " + worktreePath,
        "Review handoff: cat HANDOFF.md",
        "Begin development work",
        "When ready to merge: call prepare_merge"
      ],
      handoffPath: worktreePath + "/HANDOFF.md"
    }

Agent Interaction Pattern

Before start_stream

// Agent is in main project directory
// Agent has received user request: "Implement user authentication"

// 1. Agent plans the work
const plan = await planWork(userRequest);

// 2. Agent calls start_stream
const result = await mcp__stream_workflow__start_stream({
  title: "Add user authentication",
  category: "backend",
  priority: "high",
  handoff: `
## Objective
Implement JWT-based user authentication system

## Tasks
1. Create User model with password hashing
2. Implement login/logout endpoints
3. Add JWT token generation and validation
4. Create authentication middleware
5. Add tests for auth flow

## Success Criteria
- Users can register and login
- JWT tokens are properly validated
- All tests pass
- API documentation updated
`,
  estimatedPhases: ["planning", "implementation", "testing", "documentation"]
});

// 3. Agent navigates to worktree
process.chdir(result.worktreePath);

// 4. Agent reads HANDOFF.md and begins work
const handoff = await readFile(result.handoffPath);
// ... agent implements the work ...

After development complete

// Agent has completed work in worktree
// Agent is ready to merge back to main

// 1. Update stream status (optional)
await mcp__stream_workflow__update_stream_status({
  streamId: result.streamId,
  status: "ready-for-merge",
  progress: 100
});

// 2. Merge to main (existing tools)
await mcp__stream_workflow__prepare_merge({ streamId: result.streamId });
await mcp__stream_workflow__complete_merge({ streamId: result.streamId });
await mcp__stream_workflow__complete_stream({
  streamId: result.streamId,
  summary: "Authentication system implemented and tested"
});

File Structure Example

After start_stream completes, the file structure looks like:

egirl-platform/                           # Main project (committed metadata)
├── .project/
│   ├── STATUS_DASHBOARD.md              # Updated with new stream
│   ├── STREAM_STATE.json                # Updated with new stream ID
│   └── plan/
│       └── streams/
│           └── stream-042-add-authentication/
│               ├── HANDOFF.md           # What to work on
│               ├── README.md            # Stream context
│               ├── STATUS.md            # Progress tracking
│               └── METADATA.json        # Machine-readable metadata

<WORKTREE_ROOT>/
└── stream-042-add-authentication/       # Worktree (isolated development)
    ├── HANDOFF.md                       # Copy from main for convenience
    └── [... full project files ...]     # Ready for development

Configuration Updates

Add to config.ts

export const config = {
  // ... existing config ...

  /**
   * Stream planning directory (relative to PROJECT_ROOT)
   */
  STREAM_PLAN_DIR: '.project/plan/streams',

  /**
   * Stream metadata templates
   */
  TEMPLATES: {
    handoff: 'templates/HANDOFF.template.md',
    readme: 'templates/README.template.md',
    status: 'templates/STATUS.template.md',
  },

  /**
   * Stream ID format
   * - 'numbered': stream-001-title, stream-002-title
   * - 'dated': stream-20251211-title
   * - 'uuid': stream-abc123-title
   */
  STREAM_ID_FORMAT: 'numbered',
};

Add to types.ts

export interface StartStreamArgs {
  title: string;
  category: StreamCategory;
  priority: StreamPriority;
  handoff: string;
  description?: string;
  estimatedPhases?: string[];
  tags?: string[];
}

export interface StartStreamResponse {
  success: boolean;
  streamId: string;
  streamNumber: number;
  worktreePath: string;
  branchName: string;
  metadata: {
    planDir: string;
    filesCreated: string[];
    commitHash: string;
  };
  nextSteps: string[];
  handoffPath: string;
}

Security Considerations

1. Validate Input

Sanitize title to prevent directory traversal
Validate category/priority are from allowed enum
Limit handoff content size (prevent DoS)

2. Git Safety

Verify no uncommitted changes before proceeding
Use atomic git operations (stage all, commit all)
Verify push success before returning

3. Filesystem Safety

Use path.join() to prevent path traversal
Check directory doesn't exist before creating
Handle permission errors gracefully

4. Idempotency

If stream metadata exists but worktree missing: recreate worktree only
If worktree exists but metadata missing: error (manual cleanup needed)
If both exist: error "stream already initialized"

Error Handling

User-Facing Errors

// NotInMainError
"start_stream must be run from main project directory.
Current: /path/to/worktree
Expected: /path/to/main
Action: Navigate to main: cd " + config.PROJECT_ROOT

// UncommittedChangesError
"Main branch has uncommitted changes. Commit or stash before starting stream.
Files: [list of changed files]
Action: git commit -m 'message' OR git stash"

// BehindOriginError
"Main branch is behind origin/main. Pull latest changes first.
Behind by: 5 commits
Action: git pull origin main"

// StreamExistsError
"Stream already exists: stream-042-add-authentication
Worktree: /path/to/worktree
Action: Use existing stream OR delete worktree and retry"

// GitOperationError
"Failed to commit metadata to main.
Error: [git error message]
Action: Check git configuration and retry"

// WorktreeCreationError
"Failed to create worktree.
Path: /path/to/worktree
Error: [git error message]
Note: Metadata was committed to main. You can retry worktree creation manually:
  git worktree add /path/to/worktree -b branch-name"

Developer-Mode Errors

In DEVELOPER_MODE=true, errors include fix instructions:

"start_stream tool failed: [error]

💡 TO FIX THIS LIMITATION:
[Fix instructions for MCP server developers]

Files to edit:
  - src/tools/start-stream.ts
  - src/config.ts

See: docs/START_STREAM_DESIGN.md"

Testing Strategy

Unit Tests

describe('start_stream', () => {
  it('should validate we are in main directory');
  it('should error if uncommitted changes exist');
  it('should generate sequential stream IDs');
  it('should create all metadata files');
  it('should update STATUS_DASHBOARD.md');
  it('should commit metadata to main');
  it('should create worktree with correct branch');
  it('should copy HANDOFF.md to worktree');
  it('should return correct response structure');
  it('should be idempotent (retry safe)');
  it('should sanitize stream title for filesystem');
  it('should handle git push failures gracefully');
});

Integration Tests

describe('start_stream integration', () => {
  it('should complete full stream initialization lifecycle');
  it('should allow agent to immediately work in worktree');
  it('should integrate with prepare_merge/complete_merge');
  it('should update dashboard correctly');
  it('should handle concurrent stream creation');
});

Manual Testing Checklist

Create stream from main
Verify metadata committed to main
Verify worktree created with correct branch
Verify HANDOFF.md copied to worktree
Verify STATUS_DASHBOARD.md updated
Navigate to worktree and make changes
Run prepare_merge successfully
Run complete_merge successfully
Verify stream tracked in main throughout lifecycle

Migration Plan

Phase 1: Implementation (Week 1)

Implement src/tools/start-stream.ts
Add types to src/types.ts
Add config to src/config.ts
Write unit tests
Update server.ts to register tool

Phase 2: Documentation (Week 1)

Update README.md with start_stream usage
Update DEVELOPMENT.md with implementation details
Add examples to docs/EXAMPLES.md
Update ERROR_CATALOG.md with new errors

Phase 3: Testing (Week 2)

Write integration tests
Manual testing with real streams
Test error conditions
Performance testing (concurrent streams)

Phase 4: Rollout (Week 2)

Merge to main
Update all agent instructions to use start_stream
Monitor for issues
Gather feedback from agents

Future Enhancements

V1.1: Template System

Allow custom HANDOFF/README templates
Per-category templates (backend vs frontend)
User-defined template variables

V1.2: Stream Dependencies

Track dependencies between streams
Block merge if dependent stream not merged
Visualize dependency graph

V1.3: Auto-Planning

LLM generates estimated phases from user request
Suggests similar past streams
Auto-tags based on content analysis

V1.4: Dashboard Enhancements

Real-time dashboard updates
Stream metrics (time to completion, conflicts)
Visual progress indicators

Conclusion

The start_stream tool provides:

Atomic initialization: All-or-nothing metadata creation
Explicit exception: Only legitimate main modification
Clean separation: Metadata in main, work in worktrees
Agent-friendly: Simple API, clear next steps
Fail-safe: Graceful error handling, retry support

This design solves the core problem: agents can now initialize streams correctly without violating worktree-only principles.

Next Steps:

Review and approve this design
Implement src/tools/start-stream.ts
Test thoroughly
Update all documentation
Train agents on new workflow

Design Status: ✅ Complete - Ready for Implementation Reviewers: [To be added] Last Updated: 2025-12-11

18 KiB Raw Blame History