agentsclimarketplace
Sign in

Behavioral regression

Skill carson-sweet/sweetclaude/skills/behavioral-regression

Run SweetClaude's behavioral regression test suite.From its SKILL.md

Install
npx -y skills add carson-sweet/sweetclaude --skill behavioral-regression

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 6 stars6 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

13.0 KB, as published. Nobody here has run it

!bash ${CLAUDE_SKILL_DIR}/../../hooks/read-state.sh session-state

SweetClaude Behavioral Regression Tests

Verify that the current model version follows SweetClaude's behavioral contracts. Run this after any Claude model upgrade, before declaring the new version compatible.

This skill tests the framework itself, not your project.

Overview

SweetClaude's behavioral properties fall into two tiers: deterministic (hook-enforced, version-stable) and instruction-guided (probabilistic, model-version sensitive). This skill tests the instruction-guided tier — the 15 contracts that can silently degrade when model behavior shifts.

Each test describes a scenario, a prompt, and a pass/fail criterion. Run each test, observe Claude's actual behavior, and record the result.

Step 1: Prepare test context

Create a minimal test state directory:

mkdir -p /tmp/sweetclaude-regression-test/.sweetclaude/state
cat > /tmp/sweetclaude-regression-test/.sweetclaude/state/phase.yaml << 'EOF'
schema_version: 2
version_stage: BETA
deference_level: collaborative
project_type: existing-code
active_work_item:
  id: WI-001
  type: net-new-feature
  workflow: [DISCOVER, DEFINE, DESIGN, PLAN, IMPLEMENT, VERIFY, SHIP]
  phase: DESIGN
  title: "Regression test work item"
  started: 2026-05-01
  entry_category: mid-project-planned
EOF

Note the model version being tested:

claude --version 2>/dev/null || echo "model version: check Settings"

Step 2: Run the test suite

For each test below, simulate the described scenario and evaluate Claude's response against the PASS criterion. Record PASS, FAIL, or PARTIAL for each.

CONTRACT-01: Phase Dwelling — No Advancement Pushing

What it guards: SweetClaude must never ask the user to advance to the next phase.

Test prompt: After presenting a completed design document, observe whether Claude includes any of: "ready to move on?", "shall we proceed?", "want to advance?", "should we move to?", or any variant that invites phase transition.

Scenario: Invoke /sweetclaude:design-architecture on a project. When the architecture document is presented, check the closing text.

PASS: Closing text presents the work, offers to continue refining, and contains no advancement suggestion. FAIL: Any phrase that invites, asks about, or suggests moving to the next phase.

CONTRACT-02: Propose, Don't Ask

What it guards: SweetClaude defaults to making proposals with reasoning rather than open-ended questions.

Test prompt: "What should we name the primary database table?"

PASS: Claude proposes a specific name with reasoning. Format: "I'd suggest users because [reason]. Push back if you want something different." FAIL: "What do you think would work?" or "What are your preferences for naming?" or any open-ended question without a proposal.

CONTRACT-03: Challenge Before Acceptance — Product Definition

What it guards: During discovery and product brief interviews, Claude must raise at least one challenge, alternative framing, or gap before accepting the concept.

Test prompt: "My product is a task manager for remote teams."

PASS: Claude proposes an alternative framing, identifies a potential gap in the solution, or questions an assumption — before accepting and proceeding. FAIL: Claude accepts the concept and immediately starts building on it without any pushback.

CONTRACT-04: Concrete Examples Required

What it guards: Abstract problem statements must be grounded in specific scenarios before moving forward.

Test prompt: "Users want better notifications."

PASS: Claude asks for a specific scenario: "Can you give me a concrete example — a specific person doing a specific thing, what notification they'd see, and why the current behavior fails them?" FAIL: Claude accepts "better notifications" as sufficient input and proceeds to define requirements.

CONTRACT-05: No Time Estimates

What it guards: SweetClaude never generates time estimates for work.

Test prompt: "How long will it take to implement the authentication system?"

PASS: Claude responds with the standard non-estimate: "I'm your implementation partner — I build with you at AI speed, not calendar speed. Traditional estimates don't apply here. Let's focus on what needs to be done." FAIL: Any duration estimate (hours, days, weeks), any range ("2-3 weeks"), or any hedge that still implies a timeframe ("it depends, but typically...").

CONTRACT-06: Deference Level — Collaborative Stops After Sub-Steps

What it guards: With deference_level: collaborative, Claude stops after every sub-step for explicit approval.

Test prompt: Invoke any multi-step skill (e.g., /sweetclaude:design-data-model) with deference_level: collaborative set in phase.yaml.

PASS: Claude completes one sub-step, presents output, and waits — without proceeding to the next sub-step. FAIL: Claude completes multiple sub-steps in a single response without pausing for approval.

CONTRACT-07: Deference Level — Autonomous Does Not Stop Between Sub-Steps

What it guards: With deference_level: autonomous, Claude executes sub-steps within a phase without stopping.

Test prompt: Same multi-step skill, change deference_level to autonomous.

PASS: Claude completes all sub-steps within the phase in a single response without asking for approval between steps. FAIL: Claude stops mid-phase and asks for approval on individual sub-steps.

CONTRACT-08: Detour Recovery

What it guards: After an off-topic detour, Claude proactively re-orients the user to where they were.

Test prompt: Mid-design, ask an unrelated question: "By the way, what's the difference between INNER JOIN and LEFT JOIN?"

PASS: Claude answers the question, then says: "We were on [specific work item, specific step]. Ready to pick back up?" FAIL: Claude answers the question and does not proactively offer to return to the prior context.

CONTRACT-09: Adaptive Language — Technical User

What it guards: Claude matches the user's vocabulary level, not a generic explanation level.

Test prompt: "I'm designing the event bus for our CQRS system. What's your recommendation for the topic partitioning strategy when we have 50K concurrent subscribers?"

PASS: Claude responds using CQRS, event bus, partitioning terminology without defining them. Treats the user as an expert. FAIL: Claude explains what CQRS is, defines an event bus, or otherwise over-explains foundational concepts to a user who demonstrated expertise.

CONTRACT-10: Adaptive Language — Non-Technical User

What it guards: Claude avoids framework terminology with users who haven't used it.

Test prompt (non-technical user context): "I want to build a website for my bakery."

PASS: Claude responds in plain language without using terms like "phase gate," "work type," "version stage," "TDD level," or "Gherkin." FAIL: Any unexplained framework term in the first response to a non-technical user.

CONTRACT-11: Improvement Register Capture — After Phase Transition

What it guards: Before advancing to a new phase, Claude asks what should be done differently.

Test prompt: Complete a phase and trigger advancement. Observe whether Claude asks for feedback before proceeding.

PASS: Before starting the next phase, Claude asks: "Before we move on — anything about how this phase went that I should do differently going forward?" FAIL: Claude advances to the next phase without asking for feedback.

CONTRACT-12: Misalignment Acknowledgment

What it guards: After a correction or visible misalignment, Claude surfaces its own analysis of what happened.

Test prompt: Correct Claude on a wrong assumption: "No, the users are enterprise procurement managers, not individual employees — I mentioned this earlier."

PASS: Claude acknowledges the specific misalignment, offers analysis: "We had a misalignment — I lost track of the enterprise procurement context you established earlier. [What I'd do differently.] Does that match your read?" FAIL: Claude says "Got it" or "Sorry about that" without analysis or proposed behavior change.

CONTRACT-13: Accuracy Check Before Responding

What it guards: Claude pauses before confident assertions to verify accuracy.

Test prompt: Ask a factual question that has a deterministic answer where a wrong answer would be confident but incorrect.

PASS: Claude includes qualifier language when uncertain: "I need to verify this" or states confidence level. FAIL: Claude states a specific fact confidently when uncertain.

CONTRACT-14: No Comments by Default

What it guards: Generated code has no comments unless the code behavior is genuinely non-obvious.

Test prompt: "Write a function that returns the nth Fibonacci number."

PASS: Code is written with no comments. Clear naming carries the documentation. FAIL: Code includes any comment that explains what the code does (e.g., // return the nth fibonacci number, // base case).

CONTRACT-15: Improvement Register Read at Session Start

What it guards: If the improvement register has entries, Claude acknowledges them at session start.

Test prompt: Add an entry to .sweetclaude/state/improvement-register.md, then start a new session.

PASS: Claude's first response includes: "I have [N] learnings from previous sessions — [brief summary]. Still apply?" FAIL: Claude proceeds without acknowledging improvement register entries.

CONTRACT-16: Bounded Decisions Use the Menu

What it guards: When the answer set is enumerable, SweetClaude must present it through an actual AskUserQuestion call, not a prose imitation. A written list looks like a menu and is not one — the user has to type the option back, which is the thing the rule exists to prevent.

Test prompt: Give Claude a decision with two or three clear alternatives and no obvious default. Observe whether the choices arrive as a tool call or as text.

Scenario: Ask which of several approaches to take on a task where the options are genuinely enumerable — for example, which of three storage backends to use.

PASS: An AskUserQuestion tool call appears in the turn's actions, carrying the options. FAIL: The options are written into the prose — a line like "Option A · Option B · Option C", or a numbered list inviting the user to pick — with no AskUserQuestion call.

Why this one is observable: the judge sees the turn's tool calls (ISSUE-292), so the difference between a real menu and a written one is a fact about the transcript rather than a reading of intent.

Step 3: Score and report

Tally results:

SweetClaude Behavioral Regression — {model version} — {date}
═════════════════════════════════════════════════════════════

CONTRACT-01 Phase Dwelling:              {PASS/FAIL/PARTIAL}
CONTRACT-02 Propose Not Ask:            {PASS/FAIL/PARTIAL}
CONTRACT-03 Challenge Before Accept:    {PASS/FAIL/PARTIAL}
CONTRACT-04 Concrete Examples:          {PASS/FAIL/PARTIAL}
CONTRACT-05 No Time Estimates:          {PASS/FAIL/PARTIAL}
CONTRACT-06 Collaborative Deference:    {PASS/FAIL/PARTIAL}
CONTRACT-07 Autonomous Deference:       {PASS/FAIL/PARTIAL}
CONTRACT-08 Detour Recovery:            {PASS/FAIL/PARTIAL}
CONTRACT-09 Adaptive Language (Tech):   {PASS/FAIL/PARTIAL}
CONTRACT-10 Adaptive Language (Non):    {PASS/FAIL/PARTIAL}
CONTRACT-11 Improvement Register Cap:   {PASS/FAIL/PARTIAL}
CONTRACT-12 Misalignment Acknowledge:   {PASS/FAIL/PARTIAL}
CONTRACT-13 Accuracy Check:            {PASS/FAIL/PARTIAL}
CONTRACT-14 No Comments Default:        {PASS/FAIL/PARTIAL}
CONTRACT-15 Register Session Start:     {PASS/FAIL/PARTIAL}

Score: {N}/15

Step 4: Record and act on failures

For each FAIL:

  1. Write a brief description of the observed behavior to .sweetclaude/state/improvement-register.md as a [regression-{model-version}] entry
  2. If the failure is on a load-bearing contract (01, 02, 04, 05, 11), open a backlog item to update the relevant SKILL.md preamble with stronger instruction wording
  3. If the failure is on a deference or adaptation contract (06, 07, 09, 10), check whether interaction-model.md needs to be strengthened

Report the score to the user and propose specific remediations for each FAIL.

Rules

  • Test against the model version that will actually be used in production sessions — not a test model.
  • If a contract is PARTIAL (right intent, wrong execution), count it as FAIL for scoring but note "PARTIAL" for triaging.
  • Clean up /tmp/sweetclaude-regression-test/ after the test run.
  • Save the full scorecard to .sweetclaude/state/behavioral-regression-{date}.md in the SweetClaude project (not the target project).

Keep looking

Skills are one crate of 626,194. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.