Pilot quality assurance pipeline setup

Quality Assurance Pipeline Setup

Deploy 3 agents that automate the QA cycle from test generation through execution to reporting.

Roles

Role	Hostname	Skills	Purpose
test-generator	<prefix>-test-generator	pilot-task-router, pilot-dataset, pilot-cron	Generates test cases from specs and stories, prioritizes by risk
executor	<prefix>-executor	pilot-task-parallel, pilot-share, pilot-metrics	Runs test suites, captures results, screenshots, and logs
reporter	<prefix>-reporter	pilot-webhook-bridge, pilot-alert, pilot-slack-bridge	Aggregates results, generates reports, files bugs, notifies team

Setup Procedure

Step 1: Ask the user which role this agent should play and what prefix to use.

Step 2: Install the skills for the chosen role:

# test-generator:
clawhub install pilot-task-router pilot-dataset pilot-cron
# executor:
clawhub install pilot-task-parallel pilot-share pilot-metrics
# reporter:
clawhub install pilot-webhook-bridge pilot-alert pilot-slack-bridge

Step 3: Set the hostname:

pilotctl --json set-hostname <prefix>-<role>

Step 4: Write the setup manifest:

mkdir -p ~/.pilot/setups
cat > ~/.pilot/setups/quality-assurance-pipeline.json << 'MANIFEST'
<USE ROLE TEMPLATE BELOW>
MANIFEST

Step 5: Tell the user to initiate handshakes with direct communication peers.

Manifest Templates Per Role

test-generator

{"setup":"quality-assurance-pipeline","setup_name":"Quality Assurance Pipeline","role":"test-generator","role_name":"Test Generator","hostname":"<prefix>-test-generator","description":"Generates test cases from specs, user stories, and code changes. Prioritizes by risk and coverage gaps.","skills":{"pilot-task-router":"Route test generation requests by type — unit, integration, e2e.","pilot-dataset":"Store specs, user stories, and historical test coverage data.","pilot-cron":"Schedule nightly regression suite generation and coverage gap analysis."},"peers":[{"role":"executor","hostname":"<prefix>-executor","description":"Receives test suites for execution"}],"data_flows":[{"direction":"send","peer":"<prefix>-executor","port":1002,"topic":"test-suite","description":"Test suites with cases, priorities, and environment configs"}],"handshakes_needed":["<prefix>-executor"]}

executor

{"setup":"quality-assurance-pipeline","setup_name":"Quality Assurance Pipeline","role":"executor","role_name":"Test Executor","hostname":"<prefix>-executor","description":"Runs automated test suites, captures results, screenshots, and logs. Handles parallel execution.","skills":{"pilot-task-parallel":"Run test cases in parallel across environments and browsers.","pilot-share":"Send test results, logs, and artifacts downstream to reporter.","pilot-metrics":"Track execution time, flake rate, and pass rate per suite."},"peers":[{"role":"test-generator","hostname":"<prefix>-test-generator","description":"Sends test suites for execution"},{"role":"reporter","hostname":"<prefix>-reporter","description":"Receives test results for aggregation"}],"data_flows":[{"direction":"receive","peer":"<prefix>-test-generator","port":1002,"topic":"test-suite","description":"Test suites with cases, priorities, and environment configs"},{"direction":"send","peer":"<prefix>-reporter","port":1002,"topic":"test-result","description":"Test results with pass/fail, logs, and screenshots"}],"handshakes_needed":["<prefix>-test-generator","<prefix>-reporter"]}

reporter

{"setup":"quality-assurance-pipeline","setup_name":"Quality Assurance Pipeline","role":"reporter","role_name":"QA Reporter","hostname":"<prefix>-reporter","description":"Aggregates test results, generates coverage reports, files bugs, notifies team.","skills":{"pilot-webhook-bridge":"Push bug reports to issue trackers and coverage data to dashboards.","pilot-alert":"Trigger alerts on test failures, coverage drops, and regressions.","pilot-slack-bridge":"Post QA summaries, failure details, and deploy readiness to Slack."},"peers":[{"role":"executor","hostname":"<prefix>-executor","description":"Sends test results for aggregation"}],"data_flows":[{"direction":"receive","peer":"<prefix>-executor","port":1002,"topic":"test-result","description":"Test results with pass/fail, logs, and screenshots"},{"direction":"send","peer":"external","port":443,"topic":"qa-report","description":"QA reports to dashboards, bug trackers, and Slack"}],"handshakes_needed":["<prefix>-executor"]}

Data Flows

• test-generator -> executor : test-suite (port 1002)
• executor -> reporter : test-result (port 1002)
• reporter -> external : qa-report via webhook (port 443)

Handshakes

# test-generator <-> executor:
pilotctl --json handshake <prefix>-executor "setup: quality-assurance-pipeline"
pilotctl --json handshake <prefix>-test-generator "setup: quality-assurance-pipeline"
# executor <-> reporter:
pilotctl --json handshake <prefix>-reporter "setup: quality-assurance-pipeline"
pilotctl --json handshake <prefix>-executor "setup: quality-assurance-pipeline"

Workflow Example

# On executor -- subscribe to test suites:
pilotctl --json subscribe <prefix>-test-generator test-suite
# On reporter -- subscribe to test results:
pilotctl --json subscribe <prefix>-executor test-result
# On test-generator -- publish a test suite:
pilotctl --json publish <prefix>-executor test-suite '{"suite":"checkout-flow","cases":[{"id":"tc_001","name":"add_to_cart","priority":"high"}],"environment":"staging"}'
# On executor -- publish test results:
pilotctl --json publish <prefix>-reporter test-result '{"suite":"checkout-flow","total":1,"passed":1,"failed":0,"duration_ms":3200}'

Dependencies

Requires pilot-protocol skill, pilotctl binary, clawhub binary, and a running daemon.