Pair capability execute manual tests

/pair-capability-execute-manual-tests — Manual Test Suite Executor

Execute a manual test suite against released or deployed artifacts (website, CLI packages, registries). Produces a structured report following the manual-test-report-template (resolve override-first — template resolution).

Reads test case files from the project's manual test suite directory; each follows the manual-test-case-template. Organizational context (who, when, which areas): manual-verification.md; test case design principles: manual-testing.md.

Arguments

Argument	Required	Description
$suite	No	Path to the test suite directory. Default: auto-detect qa/ at project root.
$version	No	Version under test. If omitted, derived from the artifact (e.g. pair-cli --version or release tag).
$base-url	No	Production website URL. If omitted, derived from deployment config or adoption files.
$scope	No	Limit execution to specific critical paths: CP1, CP2, ..., all (default: all). Comma-separated for multiple.
$priority	No	Minimum priority to execute: P0, P1, P2 (default: P2 — run all). Set P0 to run only blockers.

Algorithm

Execute in sequence. For every step, follow the check → skip → act → verify pattern.

Step 1: Locate Test Suite

1. Check: Does $suite point to a directory containing test case files (CP*.md)?
2. Skip: If no suite found and no $suite argument → HALT: "No manual test suite found. Run /pair-capability-design-manual-tests to generate one from your project's artifacts, then re-invoke /pair-capability-execute-manual-tests."
3. Check: If $suite directory exists but contains zero CP*.md files → HALT: "Suite directory exists but contains no critical path files. Run /pair-capability-design-manual-tests --output {$suite} to populate it."
4. Act: Read the suite README.md for variable definitions and execution order. List all CP*.md files.
5. Verify: Suite loaded. Report: N critical paths found, N total test cases.

Step 2: Resolve Variables

1. Check: Are all variables declared in the suite README.md Variables table resolvable?
2. Skip: Variables already provided via arguments.
3. Act: For each unresolved variable, follow the "How to resolve" column in the suite README. Common patterns:
   • $VERSION: extract from artifact (--version flag) or release tag.
   • $BASE_URL: read from deployment config, adoption files, or ask the user.
   • $WORKDIR: create isolated temp directory: mktemp -d /tmp/manual-test.XXXXX.
   • $RELEASE_URL: derive from $VERSION and repo URL.
   • Project-specific variables: resolve per the suite README instructions (e.g., auth tokens, registry URLs, API keys).
4. Verify: All variables resolved. Present to user for confirmation:

VARIABLES RESOLVED:
├── [var1]:      [value]
├── [var2]:      [value]
└── [varN]:      [value]

Ask: "Proceed with these values?"

Step 3: Select Tool Strategy

1. Check: Which tools are available in the current environment?
2. Act: Build tool mapping based on availability:

Action	Preferred Tool	Fallback
Website page load + interaction	agent-browser skill	Playwright MCP (browser_navigate, browser_snapshot)
Bulk HTTP status checks	WebFetch or curl -sI via Bash	agent-browser (slower)
CLI command execution	Bash	—
File existence / content	Read tool or Bash test -f	—
Checksum verification	Bash sha256sum / shasum -a 256	—
Search UI interaction	agent-browser (fill form, click, screenshot)	Playwright MCP (browser_press_key, browser_fill_form)
Responsive viewport	agent-browser (resize, screenshot)	Playwright MCP (browser_resize, browser_take_screenshot)
Report generation	Write tool	—

1. Verify: Tool strategy set. Tests requiring unavailable tools are pre-marked BLOCKED.

Step 4: Execute Critical Paths

For each critical path file (in order: CP1, CP2, ..., CPN):

1. Check: Is this CP in $scope? Are there test cases at or above $priority?
2. Skip: If CP not in scope or all tests below priority threshold → skip entire CP.
3. Act: For each test case MT-{CPNN} in the file: a. Check preconditions: if a required precondition test failed → mark BLOCKED. b. Execute steps: run each step using the selected tool. Capture output/evidence. Follow any setup instructions in the test case (e.g. config files to copy, auth to configure). c. Evaluate expected result: compare actual vs expected. Determine PASS or FAIL. d. Record result: store test ID, status, evidence (command output, HTTP status, screenshot path).
4. Verify: All tests in CP executed or marked SKIP/BLOCKED. Log progress:

CP{N} COMPLETE: [X pass | Y fail | Z skip | W blocked] of N total

Step 5: Generate Report

1. Check: Is the manual-test-report-template available?
2. Skip: If not found → generate a minimal plain-text report.
3. Act: Fill the report template with:
   • Test suite name, version, date, tester (AI agent ID), environment details
   • Summary table (per CP: total, pass, fail, skip, blocked)
   • Overall result: PASS (all P0 pass, no Critical/Major failures) or FAIL
   • Failure details with evidence
   • Skipped/Blocked tests with reasons
4. Verify: Report written. Output path.

Step 6: Store Report

1. Act: Write report to .tmp/manual-test-reports/{suite-name}-{VERSION}-{YYYY-MM-DD}.md.
   • Create directory if it doesn't exist.
   • If file already exists, append timestamp to filename to avoid overwrite.
2. Verify: File written. Report path.

Step 7: Cleanup

1. Act: Remove $WORKDIR temp directory.
2. Verify: Cleaned up. Present final summary to user.

Output Format

MANUAL TEST EXECUTION:
├── Suite:    [{suite path}]
├── Version:  [{VERSION}]
├── Scope:    [{scope}]
├── Priority: [≥{priority}]
├── Results:
│   ├── CP1: [X/Y pass]
│   ├── CP2: [X/Y pass]
│   └── ...
├── Total:    [N pass | N fail | N skip | N blocked] of N
├── Result:   [PASS | FAIL]
└── Report:   [{report path}]

Composition Interface

When composed by /pair-process-review (Phase 6, post-merge):

• Input: /pair-process-review invokes /pair-capability-execute-manual-tests after merge as optional post-release validation.
• Output: Returns the overall result (PASS/FAIL) and report path.
  • FAIL does NOT revert the merge — it creates an issue for the findings.
  • Results are appended to the review report as an addendum.
• Scope: /pair-process-review passes $scope = P0 (blockers only) for fast post-merge validation. Full suite runs standalone.

When composed by /pair-capability-verify-done (Step 11, optional):

• Input: /pair-capability-verify-done checks if a manual test report exists for the current version with PASS result.
• Output: Returns PASS (report exists and passing) or SKIPPED (no suite or no report).
• Note: /pair-capability-verify-done does NOT invoke /pair-capability-execute-manual-tests — it only checks for an existing report.

When invoked independently:

• Run all CPs (or scoped CPs) at full priority.
• Generate and store the report.
• Present summary to user.

Determinism Strategies

To maximize reliability when executed by AI agents:

• Never rely on timing: use explicit waits (Playwright waitUntil: networkidle), not sleep.
• Assert on stable selectors: use text content, data-testid, semantic HTML — not CSS classes.
• Checksum comparison: always compute and compare — never hardcode expected hashes.
• Version from artifact: extract from the artifact itself, never from source code.
• Isolated filesystem: $WORKDIR must be outside the repo to avoid workspace interference.
• Clean npm environment: use --no-workspaces, ensure no .npmrc inheritance from parent dirs.
• Idempotent — see idempotency convention. This skill's check: re-executing produces the same result without manual cleanup.

Graceful Degradation

See graceful degradation (guideline/template missing → use a minimal structure directly) for the standard scenarios. Additional cases:

• No test suite found: HALT with guidance to run /pair-capability-design-manual-tests first (a required dependency, not optional).
• agent-browser not available: Fall back to Playwright MCP. If Playwright MCP also unavailable, fall back to WebFetch/curl for HTTP checks. Mark interactive tests (search, responsive) as BLOCKED.
• No internet: Skip tests requiring network (website, registry, auto-download). Mark as BLOCKED with reason.
• Suite-specific variable not resolvable: Mark dependent tests as BLOCKED with the resolution hint from the suite README.
• $VERSION not resolvable: Ask user to provide explicitly.
• Report template not found: Generate minimal plain-text report.
• Partial failure: Complete all CPs even if some fail. Never abort early on FAIL.

Notes

• This skill executes tests and writes reports — it does not modify application code.
• Test execution order follows CP numbering. Within a CP, tests execute in document order.
• The $WORKDIR is created once and shared across all CPs, then cleaned up at the end.
• Evidence (command output, screenshots) is captured inline in the report, not as separate files.
• This capability is designed to evolve into a standalone skill that AI agents invoke autonomously after each release.