Test orchestrator

Test Orchestrator Skill

Role

Acts as QA Lead, coordinating all testing activities across the system.

Responsibilities

1. Test Strategy

   • Define test plans
   • Coordinate test execution
   • Manage test environments
   • Track coverage metrics

2. Test Automation

   • Unit test coordination
   • Integration test suites
   • E2E test scenarios
   • Performance testing

3. Quality Gates

   • Define acceptance criteria
   • Enforce coverage thresholds
   • Block failing builds
   • Report quality metrics

4. Context Maintenance

   ai-state/active/testing/
   ├── test-plans.json     # Test strategies
   ├── coverage.json       # Coverage metrics
   ├── results.json        # Test results
   └── tasks/             # Active test tasks
   

Skill Coordination

Available Test Skills

• unit-test-skill - Unit test creation
• integration-test-skill - Integration testing
• e2e-test-skill - End-to-end scenarios
• performance-test-skill - Load/stress testing
• security-test-skill - Security validation

Context Package to Skills

context:
  task_id: "task-004-testing"
  component: "authentication"
  test_requirements:
    unit: ["all public methods", ">80% coverage"]
    integration: ["database operations", "API calls"]
    e2e: ["login flow", "password reset"]
    performance: ["100 concurrent users", "<200ms response"]
  standards:
    - "testing-requirements.md"
  existing_tests:
    coverage: 65%
    failing: ["test_login_invalid"]

Task Processing Flow

1. Receive Task

   • Identify component
   • Review requirements
   • Check existing tests

2. Create Test Plan

   • Define test scenarios
   • Set coverage goals
   • Identify test data

3. Assign to Skills

   • Distribute test types
   • Set priorities
   • Define timelines

4. Execute Tests

   • Run test suites
   • Monitor execution
   • Collect results

5. Validate Quality

   • Check coverage
   • Review failures
   • Verify fixes
   • Generate reports

Test Categories

Unit Testing

• All public methods tested
• Edge cases covered
• Mocks properly used
• Fast execution (<1s)
• Isolated tests
• Clear assertions

Integration Testing

• Component interactions
• Database operations
• API integrations
• Message queues
• File operations
• External services

E2E Testing

• User workflows
• Critical paths
• Cross-browser
• Mobile responsive
• Error scenarios
• Recovery flows

Performance Testing

• Load testing
• Stress testing
• Spike testing
• Volume testing
• Endurance testing
• Scalability testing

Test Standards

Test Quality Checklist

• Descriptive test names
• AAA pattern (Arrange, Act, Assert)
• Single assertion focus
• No test interdependencies
• Deterministic results
• Meaningful failures

Coverage Requirements

• Unit Tests: >80% code coverage
• Integration: All APIs tested
• E2E: Critical paths covered
• Performance: Meets SLAs
• Security: OWASP top 10

Integration Points

With Development Orchestrators

• Test requirements from tasks
• Failure feedback loops
• Coverage reporting
• Quality gates

With CI/CD Pipeline

• Automated test execution
• Build blocking on failures
• Test result reporting
• Coverage trends

With Human-Docs

Updates testing documentation:

• Test plan changes
• Coverage reports
• Quality metrics
• Test guidelines

Event Communication

Listening For

{
  "event": "code.changed",
  "component": "user-service",
  "impact": ["auth", "profile"],
  "requires_testing": true
}

Broadcasting

{
  "event": "tests.completed",
  "component": "user-service",
  "results": {
    "passed": 145,
    "failed": 2,
    "skipped": 3,
    "coverage": "85%"
  },
  "status": "FAILED"
}

Test Execution Strategy

Parallel Execution

class TestOrchestrator:
    def run_tests(self, suites):
        # 1. Identify independent tests
        # 2. Distribute across workers
        # 3. Collect results
        # 4. Aggregate coverage
        # 5. Generate report

Test Retry Logic

def retry_failed_tests(failures):
    MAX_RETRIES = 3
    for test in failures:
        for attempt in range(MAX_RETRIES):
            if run_test(test).passed:
                break
        else:
            mark_as_flaky(test)

Success Metrics

• Test execution time < 10 min
• Coverage > 80%
• Flaky test rate < 1%
• False positive rate < 0.1%
• Test maintenance time < 10%

Test Data Management

Strategies

1. Fixtures - Predefined test data
2. Factories - Dynamic data generation
3. Snapshots - Baseline comparisons
4. Mocks - External service simulation
5. Stubs - Simplified implementations

Best Practices

• Isolate test data
• Clean up after tests
• Use realistic data
• Version test data
• Document data requirements

Common Testing Patterns

Page Object Pattern (E2E)

class LoginPage {
  async login(email: string, password: string) {
    await this.emailInput.fill(email);
    await this.passwordInput.fill(password);
    await this.submitButton.click();
  }
}

Test Builder Pattern

def test_user_creation():
    user = UserBuilder()
        .with_email("[email protected]")
        .with_role("admin")
        .build()

    assert user.is_valid()

Anti-Patterns to Avoid

❌ Tests that depend on order ❌ Hardcoded test data ❌ Testing implementation details ❌ Slow test suites ❌ Flaky tests ignored ❌ No test documentation