agentsclimarketplace
Sign in

Generate adhoc tests

Skill Autonoma-AI/test-planner-plugin/skills/generate-adhoc-tests

Generates focused E2E test cases for a user-defined topic through a validated multi-step pipeline. Each step runs in an isolated subagent and must pass deterministic validation before the next step begins. When scenarios already exist in Autonoma, fetches context from the API and runs only Step 3 scoped to the topic. On a first run, executes the full 4-step pipeline with Step 3 focused. Use when you want targeted test coverage for a specific feature or domain.From its SKILL.md

Install
npx -y skills add Autonoma-AI/test-planner-plugin --skill generate-adhoc-tests

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

27.6 KB, as published. Nobody here has run it

Autonoma Focused E2E Test Generation Pipeline

You are orchestrating a focused test generation pipeline. Each step runs as an isolated subagent. Every step MUST complete successfully and pass validation before the next step begins. Do NOT skip steps. Do NOT proceed if validation fails.

User Confirmation Between Steps

By default, after each step (1, 2, and 3), you MUST present the summary and then ask the user for confirmation using the AskUserQuestion tool. This creates an interactive UI prompt that makes it clear the user needs to respond before the pipeline continues.

After calling AskUserQuestion, wait for the user's response. Only proceed to the next step after they confirm.

Auto-advance mode: If the environment variable AUTONOMA_AUTO_ADVANCE is set to true, skip the AskUserQuestion calls and automatically proceed to the next step after presenting the summary. The summaries are still displayed — only the confirmation prompt is skipped.

Before Starting

Resolve the focus prompt from the user's input (the text after the command name):

FOCUS_PROMPT="<the user's focus description>"
FOCUS_SLUG=$(echo "$FOCUS_PROMPT" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-\|-$//g')
echo "Focus: $FOCUS_PROMPT"
echo "Slug:  $FOCUS_SLUG"

If no focus description was provided, list top-level route/feature directories in the codebase, call AskUserQuestion with 3–4 suggested focus areas plus an "Other" option, wait for the user's response, then derive FOCUS_SLUG from their answer.

Create the output directory and save the project root (subagents change working directory, so we need an absolute path reference):

AUTONOMA_ROOT="$(pwd)"
echo "$AUTONOMA_ROOT" > /tmp/autonoma-project-root
mkdir -p autonoma/skills autonoma/qa-tests

The plugin root path (where hooks, validators, and helper scripts live) is persisted to /tmp/autonoma-plugin-root automatically by the PostToolUse validation hook on the first Write. All bash snippets that need plugin-local files read it back:

PLUGIN_ROOT=$(cat /tmp/autonoma-plugin-root 2>/dev/null || echo '')

Read the environment variables. These are required for reporting progress back to Autonoma:

  • AUTONOMA_API_KEY — your Autonoma API key
  • AUTONOMA_PROJECT_ID — your Autonoma project ID
  • AUTONOMA_API_URL — Autonoma API base URL
  • AUTONOMA_AUTO_ADVANCE — (optional) set to true to skip user confirmation prompts between steps

Before creating the record, derive a clean human-readable application name from the repository. Look at the git remote URL, the directory name, and any package.json / pyproject.toml / README.md to infer what the product is actually called. Prefer the product name over the repo slug (e.g. "My App" not "my-app-v2-final"). Store it in APP_NAME.

Create the generation record so the dashboard can track progress in real time:

RESPONSE=$(curl -s -w "\nHTTP_STATUS:%{http_code}" -X POST "${AUTONOMA_API_URL}/v1/setup/setups" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"applicationId\":\"${AUTONOMA_PROJECT_ID}\",\"repoName\":\"${APP_NAME}\"}")
HTTP_STATUS=$(echo "$RESPONSE" | grep -o "HTTP_STATUS:[0-9]*" | cut -d: -f2)
BODY=$(echo "$RESPONSE" | sed '/HTTP_STATUS:/d')
echo "Setup API response (HTTP $HTTP_STATUS): $BODY"
GENERATION_ID=$(echo "$BODY" | python3 -c "import json,sys; print(json.load(sys.stdin).get('id',''))" 2>/dev/null || echo '')
mkdir -p autonoma
echo "$GENERATION_ID" > "autonoma/.generation-id-${FOCUS_SLUG}"
echo "Generation ID: $GENERATION_ID"

If GENERATION_ID is empty, log the HTTP status and response body above for debugging, then continue anyway — reporting is best-effort and must never block test generation.

Checking Existing Setup

Check whether scenarios with active recipes already exist in Autonoma for this application:

AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
GENERATION_ID=$(cat "$AUTONOMA_ROOT/autonoma/.generation-id-${FOCUS_SLUG}" 2>/dev/null || echo '')
HAS_SCENARIOS="no"
SCENARIOS_RESPONSE=""
if [ -n "$GENERATION_ID" ]; then
  SCENARIOS_RESPONSE=$(curl -s "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/scenarios" \
    -H "Authorization: Bearer ${AUTONOMA_API_KEY}")
  HAS_SCENARIOS=$(echo "$SCENARIOS_RESPONSE" | python3 -c "
import json, sys
data = json.loads(sys.stdin.read())
active = [s for s in data.get('scenarios', []) if s.get('hasActiveRecipe')]
print('yes' if active else 'no')
" 2>/dev/null || echo "no")
fi
echo "Has active scenarios: $HAS_SCENARIOS"

If HAS_SCENARIOS=yes — scenarios and tests already exist. Fetch context from the API and run only Step 3:

AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
GENERATION_ID=$(cat "$AUTONOMA_ROOT/autonoma/.generation-id-${FOCUS_SLUG}" 2>/dev/null || echo '')

EXISTING_CONTEXT=$(curl -s "${AUTONOMA_API_URL}/v1/setup/applications/${AUTONOMA_PROJECT_ID}/test-suite" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}")

SCENARIOS_CONTEXT=$(echo "$SCENARIOS_RESPONSE" | python3 -c "
import json, sys
data = json.loads(sys.stdin.read())
lines = ['## Available Scenarios', '']
for s in data.get('scenarios', []):
    status = 'active' if s.get('hasActiveRecipe') else 'no recipe'
    lines.append(f\"- **{s['name']}** ({status})\")
print('\n'.join(lines))
" 2>/dev/null || echo "")

TESTS_CONTEXT=$(echo "$EXISTING_CONTEXT" | python3 -c "
import json, sys
data = json.loads(sys.stdin.read())
tests = data.get('tests', [])
lines = [f'## Existing Tests ({len(tests)} total)', '']
for t in tests:
    lines.append(f\"- {t['name']} (slug: {t['slug']})\")
print('\n'.join(lines))
" 2>/dev/null || echo "")

SKILLS_CONTEXT=$(echo "$EXISTING_CONTEXT" | python3 -c "
import json, sys
data = json.loads(sys.stdin.read())
skills = data.get('skills', [])
lines = [f'## Available Skills ({len(skills)} total)', '']
for s in skills:
    lines.append(f\"- {s['name']}: {s['description']}\")
print('\n'.join(lines))
" 2>/dev/null || echo "")

Skip to Step 3: Generate Focused E2E Test Cases and pass the fetched context inline in the subagent task — do not run Steps 1, 2, or 4.

If HAS_SCENARIOS=no — this is a first run. Continue with the full pipeline below (Steps 1 through 4).

Step 1: Generate Knowledge Base

Report step start:

AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
GENERATION_ID=$(cat "$AUTONOMA_ROOT/autonoma/.generation-id-${FOCUS_SLUG}" 2>/dev/null || echo '')
echo "GENERATION_ID=${GENERATION_ID:-<empty>}"
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"step.started","data":{"step":0,"name":"Knowledge Base"}}' || true
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"log","data":{"message":"Analyzing codebase structure and identifying features..."}}' || true

Spawn the kb-generator subagent with the following task:

Analyze the codebase and generate the knowledge base. Write the output to autonoma/AUTONOMA.md and create skill files in autonoma/skills/. The file MUST have YAML frontmatter with app_name, app_description, core_flows (feature/description/core table), feature_count, and skill_count. You MUST also write autonoma/features.json — a machine-readable inventory of every feature discovered. It must have: features array (each with name, type, path, core), total_features, total_routes, total_api_routes. Fetch the latest instructions from https://docs.agent.autonoma.app/llms/test-planner/step-1-knowledge-base.txt first.

After the subagent completes:

  1. Verify autonoma/AUTONOMA.md and autonoma/features.json exist and are non-empty
  2. The PostToolUse hook will have validated the frontmatter and features.json schema automatically
  3. Read the file and present the frontmatter to the user — specifically the core_flows table

Report step complete and upload skills:

AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
GENERATION_ID=$(cat "$AUTONOMA_ROOT/autonoma/.generation-id-${FOCUS_SLUG}" 2>/dev/null || echo '')
echo "GENERATION_ID=${GENERATION_ID:-<empty>}"
SKILL_COUNT=$(ls "$AUTONOMA_ROOT/autonoma/skills/"*.md 2>/dev/null | wc -l | tr -d ' ')
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"type\":\"log\",\"data\":{\"message\":\"Knowledge base complete. Generated ${SKILL_COUNT} skills. Uploading to dashboard...\"}}" || true

[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"step.completed","data":{"step":0,"name":"Knowledge Base"}}' || true

[ -n "$GENERATION_ID" ] && python3 -c "
import os, json, sys
root = open('/tmp/autonoma-project-root').read().strip() if os.path.exists('/tmp/autonoma-project-root') else '.'
skills = []
d = os.path.join(root, 'autonoma/skills')
if os.path.isdir(d):
    for f in os.listdir(d):
        if f.endswith('.md'):
            with open(os.path.join(d, f)) as fh:
                skills.append({'name': f, 'content': fh.read()})
print(json.dumps({'skills': skills}))
" | curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/artifacts" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d @- || true
  1. If AUTONOMA_AUTO_ADVANCE is not true: Call AskUserQuestion with:
    • question: "Does this core flows table look correct? These flows determine how the test budget is distributed."
    • options: ["Yes, proceed to Step 2", "I want to suggest changes"] Wait for the user's response before proceeding. If AUTONOMA_AUTO_ADVANCE=true: Skip the prompt and proceed directly to Step 2.

Step 2: Generate Scenarios

Report step start:

AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
GENERATION_ID=$(cat "$AUTONOMA_ROOT/autonoma/.generation-id-${FOCUS_SLUG}" 2>/dev/null || echo '')
echo "GENERATION_ID=${GENERATION_ID:-<empty>}"
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"step.started","data":{"step":1,"name":"Scenarios"}}' || true
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"log","data":{"message":"Mapping data model and designing test data environments..."}}' || true

Before spawning the Step 2 subagent, fetch the SDK discover artifact and save it to autonoma/discover.json. This step requires these environment variables:

  • AUTONOMA_SDK_ENDPOINT — full URL of the customer's SDK endpoint
  • AUTONOMA_SHARED_SECRET — the HMAC shared secret used by the SDK endpoint

If either variable is missing, stop and tell the user that Step 2 now requires SDK discover access. Do not suggest skipping ahead, reordering the pipeline, or continuing without a working Environment Factory endpoint. State plainly that the endpoint and both environment variables are mandatory prerequisites for Step 2.

Fetch and validate the artifact:

AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
mkdir -p "$AUTONOMA_ROOT/autonoma"
BODY='{"action":"discover"}'
SIG=$(echo -n "$BODY" | openssl dgst -sha256 -hmac "$AUTONOMA_SHARED_SECRET" | sed 's/.*= //')
RESPONSE=$(curl -sS -w "\nHTTP_STATUS:%{http_code}" -X POST "$AUTONOMA_SDK_ENDPOINT" \
  -H "Content-Type: application/json" \
  -H "x-signature: $SIG" \
  -d "$BODY")
HTTP_STATUS=$(echo "$RESPONSE" | grep -o "HTTP_STATUS:[0-9]*" | cut -d: -f2)
DISCOVER_BODY=$(echo "$RESPONSE" | sed '/HTTP_STATUS:/d')
if [ "$HTTP_STATUS" != "200" ]; then
  echo "SDK discover failed (HTTP $HTTP_STATUS): $DISCOVER_BODY"
  exit 1
fi
printf '%s\n' "$DISCOVER_BODY" > "$AUTONOMA_ROOT/autonoma/discover.json"
python3 "$(cat /tmp/autonoma-plugin-root)/hooks/validators/validate_discover.py" "$AUTONOMA_ROOT/autonoma/discover.json"

If the fetch fails or validation fails, stop the pipeline at Step 2. Do not suggest skipping ahead. Tell the user to provide a working SDK endpoint and correct shared secret, then rerun the command.

Spawn the scenario-generator subagent with the following task:

Read the knowledge base from autonoma/AUTONOMA.md, autonoma/skills/, and the SDK discover artifact from autonoma/discover.json. Generate test data scenarios. Write the output to autonoma/scenarios.md. The file MUST have YAML frontmatter with scenario_count, scenarios summary, entity_types, discover metadata, and variable_fields. Prefer fixed, reviewable seed values by default. If a field needs uniqueness, prefer a planner-chosen hardcoded literal plus a discriminator before introducing a variable placeholder. Use variable fields only for truly dynamic values such as backend-generated or time-based fields. generator is optional and must not default to faker. Fetch the latest instructions from https://docs.agent.autonoma.app/llms/test-planner/step-2-scenarios.txt first.

After the subagent completes:

  1. Verify autonoma/discover.json and autonoma/scenarios.md exist and are non-empty
  2. Validate autonoma/discover.json using the plugin's validator (path saved in /tmp/autonoma-plugin-root)
  3. The PostToolUse hook will have validated the scenarios.md frontmatter format automatically
  4. Read the file and present the summary to the user — scenario names, entity counts, entity types, discover schema counts, and the minimal variable field tokens that remain dynamic

Report step complete:

AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
GENERATION_ID=$(cat "$AUTONOMA_ROOT/autonoma/.generation-id-${FOCUS_SLUG}" 2>/dev/null || echo '')
echo "GENERATION_ID=${GENERATION_ID:-<empty>}"
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"log","data":{"message":"Scenarios generated from SDK discover. Preserved standard/empty/large plus schema metadata, keeping variable fields minimal and intentional."}}' || true
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"step.completed","data":{"step":1,"name":"Scenarios"}}' || true
  1. If AUTONOMA_AUTO_ADVANCE is not true: Call AskUserQuestion with:
    • question: "Do these scenarios look correct? Most seed values should stay concrete, ideally as planner-chosen literals with discriminators, and only truly dynamic values should remain variable for later tests."
    • options: ["Yes, proceed to Step 3", "I want to suggest changes"] Wait for the user's response before proceeding. If AUTONOMA_AUTO_ADVANCE=true: Skip the prompt and proceed directly to Step 3.

Step 3: Generate Focused E2E Test Cases

Report step start:

AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
GENERATION_ID=$(cat "$AUTONOMA_ROOT/autonoma/.generation-id-${FOCUS_SLUG}" 2>/dev/null || echo '')
echo "GENERATION_ID=${GENERATION_ID:-<empty>}"
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"step.started","data":{"step":2,"name":"Focused E2E Tests"}}' || true
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"log","data":{"message":"Generating focused E2E test cases..."}}' || true

Spawn the focused-test-case-generator subagent with the following task (substitute the actual values for FOCUS_PROMPT, FOCUS_SLUG, and — when coming from the API-fetch path — the context variables SCENARIOS_CONTEXT, TESTS_CONTEXT, and SKILLS_CONTEXT before spawning):

FOCUS_PROMPT: <the user's focus description> FOCUS_SLUG: <kebab-case slug>

(API-fetch path only — omit this block when running the full pipeline) Context fetched from the Autonoma API (use this instead of reading local files): <SCENARIOS_CONTEXT> <TESTS_CONTEXT> <SKILLS_CONTEXT>

Read the knowledge base from autonoma/AUTONOMA.md, skills from autonoma/skills/, and scenarios from autonoma/scenarios.md (if they exist and no inline context was provided above). Generate E2E test cases focused exclusively on the topic described in FOCUS_PROMPT. Write tests to autonoma/qa-tests/{FOCUS_SLUG}/. You MUST create autonoma/qa-tests/{FOCUS_SLUG}/INDEX.md with frontmatter containing total_tests, total_folders, folder breakdown, and coverage_correlation. Each test file MUST have frontmatter with title, description, criticality, scenario, and flow. Treat scenario data as fixture input only. Do not generate tests whose purpose is to verify scenario counts, seeded inventories, or Environment Factory correctness. Only reference scenario data when it is needed to test a real user-facing app behavior within the focus area. Fetch the latest instructions from https://docs.agent.autonoma.app/llms/test-planner/step-3-e2e-tests.txt first.

After the subagent completes:

  1. Verify autonoma/qa-tests/${FOCUS_SLUG}/INDEX.md exists and is non-empty
  2. The PostToolUse hook will have validated the INDEX frontmatter and individual test file frontmatter
  3. Read the INDEX.md and present the summary to the user — total tests, folder breakdown, coverage correlation

Report step complete and upload test cases:

AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
GENERATION_ID=$(cat "$AUTONOMA_ROOT/autonoma/.generation-id-${FOCUS_SLUG}" 2>/dev/null || echo '')
echo "GENERATION_ID=${GENERATION_ID:-<empty>}"
TEST_COUNT=$(find "$AUTONOMA_ROOT/autonoma/qa-tests/${FOCUS_SLUG}" -name '*.md' ! -name 'INDEX.md' 2>/dev/null | wc -l | tr -d ' ')
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"type\":\"log\",\"data\":{\"message\":\"Generated ${TEST_COUNT} focused test cases. Uploading to dashboard...\"}}" || true

[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"step.completed","data":{"step":2,"name":"Focused E2E Tests"}}' || true

[ -n "$GENERATION_ID" ] && python3 -c "
import os, json
proj_root = open('/tmp/autonoma-project-root').read().strip() if os.path.exists('/tmp/autonoma-project-root') else '.'
qa_dir = os.path.join(proj_root, 'autonoma/qa-tests/${FOCUS_SLUG}')
test_cases = []
for root, dirs, files in os.walk(qa_dir):
    for f in files:
        if f.endswith('.md') and f != 'INDEX.md':
            path = os.path.join(root, f)
            folder = os.path.relpath(root, qa_dir)
            with open(path) as fh:
                content = fh.read()
            entry = {'name': f, 'content': content}
            if folder != '.':
                entry['folder'] = '${FOCUS_SLUG}/' + folder
            test_cases.append(entry)
print(json.dumps({'testCases': test_cases}))
" | curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/artifacts" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d @- || true
  1. If AUTONOMA_AUTO_ADVANCE is not true: Call AskUserQuestion with:
    • question: "Does this focused test distribution look correct? The tests should cover the requested topic thoroughly."
    • options: ["Yes, proceed to Step 4", "I want to suggest changes", "Done — skip Step 4 (scenarios already exist)"] Wait for the user's response before proceeding. If AUTONOMA_AUTO_ADVANCE=true: Skip the prompt and proceed directly to Step 4 (or stop here if coming from the API-fetch path).

If coming from the API-fetch path (scenarios already existed), stop here after uploading. Step 4 is not needed.

Step 4: Environment Factory

Report step start:

AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
GENERATION_ID=$(cat "$AUTONOMA_ROOT/autonoma/.generation-id-${FOCUS_SLUG}" 2>/dev/null || echo '')
echo "GENERATION_ID=${GENERATION_ID:-<empty>}"
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"step.started","data":{"step":3,"name":"Environment Factory"}}' || true
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"log","data":{"message":"Implementing or completing the Environment Factory and validating planned scenarios..."}}' || true

This step requires these environment variables:

  • AUTONOMA_SDK_ENDPOINT — full URL of the customer's SDK endpoint
  • AUTONOMA_SHARED_SECRET — the HMAC shared secret used by the SDK endpoint

If either variable is missing, stop and tell the user that Step 4 requires SDK endpoint access for preflight validation. State plainly that both environment variables are mandatory.

Spawn the env-factory-generator subagent with the following task:

Read autonoma/discover.json and autonoma/scenarios.md. Implement or complete the Autonoma Environment Factory in the project's backend so it can support the planned scenarios with the current SDK contract, then validate the planned scenarios against that implementation. Fetch the latest instructions from https://docs.agent.autonoma.app/llms/test-planner/step-4-implement-scenarios.txt and https://docs.agent.autonoma.app/llms/guides/environment-factory.txt first. Preserve the existing discover integration if it already works, and finish up / down behavior using AUTONOMA_SHARED_SECRET and AUTONOMA_SIGNING_SECRET. Smoke-test the discover -> up -> down lifecycle in-session after implementing. Then validate standard, empty, and large, and write approved recipes to autonoma/scenario-recipes.json. The recipe file must match the current setup API schema: top-level version: 1, source, validationMode, recipes; each recipe must use name, description, create, and validation with status: "validated", a valid method, phase: "ok", and optional up_ms / down_ms. Do not use the old shape with top-level scenarios, generatedAt, or per-recipe validated / timing. When create uses {{token}} placeholders, include a variables field per recipe that defines how each token is resolved. Allowed strategies: literal, derived, faker. Persisted create must remain tokenized — never store resolved concrete values. After writing the recipe file, run the preflight helper to validate all recipes against the live SDK endpoint before uploading: python3 "$(cat /tmp/autonoma-plugin-root)/hooks/preflight_scenario_recipes.py" autonoma/scenario-recipes.json The preflight must pass for all three scenarios before Step 4 is considered complete.

After the subagent completes:

  1. Verify the backend implementation or integration changes were made
  2. Verify autonoma/scenario-recipes.json exists and is non-empty
  3. Run the preflight helper if the subagent did not already do so:
AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
python3 "$(cat /tmp/autonoma-plugin-root)/hooks/preflight_scenario_recipes.py" "$AUTONOMA_ROOT/autonoma/scenario-recipes.json"

If preflight fails, do NOT proceed to upload. Report the failure to the user and stop. 4. Present the results to the user — endpoint location, what was implemented or fixed, smoke-test results, per-scenario preflight results 5. Report which environment variables the backend now requires 6. Report any backend issues that still need manual attention

Report step complete:

AUTONOMA_ROOT=$(cat /tmp/autonoma-project-root 2>/dev/null || echo '.')
GENERATION_ID=$(cat "$AUTONOMA_ROOT/autonoma/.generation-id-${FOCUS_SLUG}" 2>/dev/null || echo '')
echo "GENERATION_ID=${GENERATION_ID:-<empty>}"
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"log","data":{"message":"Uploading validated scenario recipes to setup..."}}' || true
if [ -n "$GENERATION_ID" ]; then
  RECIPE_PATH="$AUTONOMA_ROOT/autonoma/scenario-recipes.json"
  if ! python3 -c "import json; json.load(open('$RECIPE_PATH'))" 2>/dev/null; then
    echo "ERROR: scenario-recipes.json is not valid JSON. Step 4 cannot complete."
    exit 1
  fi
  UPLOAD_RESPONSE=$(curl -s -w "\nHTTP_STATUS:%{http_code}" -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/scenario-recipe-versions" \
    -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
    -H "Content-Type: application/json" \
    -d @"$RECIPE_PATH")
  UPLOAD_STATUS=$(echo "$UPLOAD_RESPONSE" | grep -o "HTTP_STATUS:[0-9]*" | cut -d: -f2)
  UPLOAD_BODY=$(echo "$UPLOAD_RESPONSE" | sed '/HTTP_STATUS:/d')
  echo "Scenario recipe upload response (HTTP $UPLOAD_STATUS): $UPLOAD_BODY"
  if [ "$UPLOAD_STATUS" != "200" ] && [ "$UPLOAD_STATUS" != "201" ]; then
    echo "ERROR: Recipe upload failed (HTTP $UPLOAD_STATUS). Step 4 cannot complete."
    exit 1
  fi
fi
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"log","data":{"message":"Environment Factory implementation and scenario validation completed."}}' || true
[ -n "$GENERATION_ID" ] && curl -f -X POST "${AUTONOMA_API_URL}/v1/setup/setups/${GENERATION_ID}/events" \
  -H "Authorization: Bearer ${AUTONOMA_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"step.completed","data":{"step":3,"name":"Environment Factory"}}' || true

Completion

After all steps complete, summarize:

  • Focus: The user-defined topic and output location (autonoma/qa-tests/{FOCUS_SLUG}/)
  • Step 1: Knowledge base location and core flow count (full pipeline only)
  • Step 2: Scenario count and entity types covered (full pipeline only)
  • Step 3: Total focused test count, folder breakdown, coverage correlation
  • Step 4: Environment Factory location, backend changes, smoke-test results, required secrets, and per-scenario lifecycle results (full pipeline only)

Keep looking

Skills are one crate of 628,853. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.