Red team

Red Team

An adversarial loop-until-dry. The artifact is a target system; the feedback signal is the count of distinct failure classes you can surface. Each round you craft adversarial inputs aimed at new weaknesses and run them through the target and a ground-truth oracle via tools/harness.py, which records every disagreement as a failure and de-dupes by the class (technique) you label each input with. You loop until fresh rounds stop finding anything new. This is only the find half of a find→fix setup: it catalogues failures and never patches the target (see Pairing).

When to use

Use to harden a guardrail, classifier, content filter, prompt, or API that the user owns or is explicitly authorized to test — when the goal is a catalogue of distinct, reproducible failures, each an objective target-vs-oracle disagreement. A failure is a bypass (target allows what the oracle would block) or an over-block (target blocks what the oracle would allow).

Default: drive the loop with a runnable oracle so the signal is objective. Escape hatch: if the user has no runnable oracle, the oracle is your judgment against a written policy — apply it consistently and record the intended verdict per input. Not for fixing the target, and not for testing systems outside the user's authorization.

Setup

Resolve bindings interactively. If loop.run.yaml exists, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value per binding and recommend it; on other hosts ask each as a quoted prompt. Then write loop.run.yaml (format: examples/run.example.yaml) and confirm before creating any other files.

binding	meaning	default	how to infer
<target_cmd>	system under test: reads one input on stdin, prints a verdict (BLOCK/ALLOW, a label, a score). Never edited.	—	the guardrail/classifier/API entrypoint the user names
<oracle_cmd>	ground-truth verdict for the same input. A failure is target != oracle.	—	a reference checker / policy impl; else your judgment vs a written policy
<candidates_file>	each round's candidates, JSONL {id, text, class}; class is the technique the harness de-dupes on	<sandbox_root>/candidates.jsonl	—
<failures_log>	append-only log of confirmed failures	<sandbox_root>/failures.jsonl	—
<sandbox_root>	where candidates, failures, and the ledger live	./sandbox	—
<budget>	max rounds	8	—
<patience>	stop after N consecutive rounds with no new failure class	2	—

The signal comes from tools/harness.py. Run it each round:

python3 <skill_dir>/tools/harness.py --target "<target_cmd>" --oracle "<oracle_cmd>" \
  --inputs <candidates_file> --log <failures_log>

It runs both commands on every candidate and prints one JSON object: {tested, failures_this_run, new_classes, total_classes, examples}.

The loop

Copy this checklist and tick items off each round:

• Round 0 — probe: read the target's intended contract, run a small mixed batch through the harness to confirm wiring, note any failures it already reveals.
• Pick a fresh angle — a failure mode not yet surfaced (see the attack toolkit below).
• Generate a batch of candidates for that angle, each labeled with a root-cause class; write to <candidates_file>.
• Run tools/harness.py; read new_classes and the example failures.
• Record: note which new classes this round added (the harness already appended the failures and de-duped). Append one ledger row.
• Stop when <patience> consecutive rounds add no new class, or at <budget>. Else steer the next round at an untried angle.

Attack toolkit (draw from these, and invent your own):

• Obfuscation — case changes, spacing/punctuation, leetspeak, unicode homoglyphs, encoding.
• Paraphrase / synonyms — say the forbidden thing a different way; expand abbreviations.
• Boundary & context — embed the payload in benign text; multi-step or indirect phrasing.
• Over-block probes — benign inputs containing a trigger substring, to find false positives.

Labeling classes. A class is the root-cause technique — the single fixable weakness — not one label per payload. Capitalizing password, apikey, and ssn are all the same class (case-bypass), because one fix closes all of them; do not split them into case-password, case-apikey, … That inflates the count so the loop never goes dry. Aim for a handful of root-cause classes (e.g. case-bypass, leetspeak, spacing, missing-synonym, overblock), each demonstrated by several payloads. Use a fresh class only for a genuinely new root cause; reuse a class to add more evidence for one already found.

On stop, report: the catalogue of distinct failure classes with one reproducible example each, the bypass/over-block split, and — since the goal is a more robust target — a short suggested fix per class.

Ledger

<sandbox_root>/ledger.tsv, tab-separated, never commas in the text. Header round angle tested new_classes total_classes:

round	angle	tested	new_classes	total_classes
0	probe mixed batch	6	case,spacing	2
1	leetspeak + unicode	8	leetspeak	3
2	synonyms + expansions	8	synonym,expansion	5
3	benign trigger substrings	6	overblock	6
4	multi-step phrasing	8	(none)	6

Report the catalogue at the round with the most distinct classes (the cumulative total_classes), not whichever round ran last.

Constraints

• Never edit the target, the oracle, or tools/harness.py. They define the system and the ground truth; changing them manufactures or hides failures.
• A failure is an objective target-vs-oracle disagreement, not a hunch — every recorded failure is reproducible from its input.
• Label classes honestly and pursue new angles — the signal is distinct failure modes, so do not pad counts by relabeling the same technique, and do not stop at the first bypass when others remain.
• Keep findings oriented toward fixing the target; this is robustness testing of an authorized system, and the catalogue exists to be handed to a fixer.
• Stay inside <sandbox_root>; no path escapes outside it.
• Do not pause to ask whether to continue; run until the target goes dry or hits <budget>.

Pairing

This skill is the attacker — half of a find→fix loop. By itself it tells you how the system fails but leaves it unfixed. The intended full loop pairs it with a separate coding agent that patches the target, in three strictly separated phases:

1. Find (this loop) — run against the frozen target → a catalogue of distinct failure classes, each with a reproducible example and a suggested fix.
2. Fix (a separate coding agent) — apply patches to the target to close those classes, between runs, never inside one: the target is read-only ground truth for the duration of a run, so mutating it mid-loop would break reproducibility and the class accounting.
3. Re-verify (a fresh find run) — start a new run against the patched target. Confirm each prior class is closed and watch for regressions — especially new over-blocks an over-eager fix may introduce (this loop already hunts that direction).

Repeat find → fix → re-verify until a fresh run stays dry. Keep the two agents independent: the attacker that wrote the catalogue should not also grade its own patch. This skill deliberately stops at the end of phase 1; the fix/re-verify orchestration lives outside it.