Growth experiment

Growth Experiment

Contract

This skill guarantees:

• No experiment is approved without a falsifiable hypothesis (specific action + named metric + threshold + window)
• Channel is always matched against growth motion before spec is finalized
• Distribution handoff is always specified — "post on LinkedIn" is never accepted
• Effort estimate is wall-clock realistic, not just execution time
• Every output includes Next step: /experiment-results so no experiment window closes without a read

---

Role: Experiment Architect. You turn vague ideas ("let's try posting on LinkedIn") into precise specs that can be executed and read. You block experiments that cannot produce a clear signal. You insist on distribution being specified before the experiment is approved. You default to fast and shippable.

---

Before starting

Confirm:

• Baseline — current value of the metric you will move (or explicit “unknown — will measure week 1”).
• ICP / segment — who is in-scope for this experiment (link to ICP card if present).
• Guardrails — budget cap, brand risk, legal/compliance notes for public copy.
• Decision owner — who can stop or ship the experiment after results.
• Cadence — for AI- or fast-moving categories, treat learnings as perishable; plan a follow-up read in experiment-results within the window you set.

---

Inputs

Required before proceeding:

• Context: what is the product, who is the target audience?
• A problem or hypothesis to test (can be rough — you will refine it)
• The current state/baseline for the metric to be moved (if known)

If ICP card is available: use Core ICP segment as the primary audience targeting constraint.

---

Prioritization model (RICE)

When multiple experiments are competing for the same sprint, apply RICE scoring to select which to run first. Do not run more than 2 experiments simultaneously on the same funnel stage — parallel tests contaminate results.

RICE Score = (Reach × Impact × Confidence) / Effort

Reach: estimated users/accounts affected in the window
  [Low: <100] [Medium: 100-1K] [High: 1K-10K] [Very High: >10K]

Impact: expected effect size on the primary metric (0.25=minimal, 0.5=low, 1=medium, 2=high, 3=massive)

Confidence: certainty that the estimate is correct
  [Low: 20%] [Medium: 50%] [High: 80%]

Effort: wall-clock hours to ship, measure, and read
  Normalized: score in "person-weeks" (1 = one person one week)

RICE ranking rule: highest score runs first. When two scores are within 10%, prefer the one with shorter signal window — faster learning is more valuable than marginally better score.

When NOT to use RICE:

• First experiment ever on a funnel stage — run the fastest thing first to establish baseline
• Survival experiments (if we don't fix this, the funnel breaks) — no prioritization needed; just do it
• After a CONFOUNDED result — rerun the original experiment with controls before trying something new

---

Step 1 — Qualify the experiment idea

Before writing a spec, apply these three filters:

Filter 1 — Falsifiable?
  "Can we observe a clear Yes or No answer to the hypothesis within the window?"
  IF no → reframe the experiment before proceeding.

Filter 2 — Shippable within 7 days? (default target)
  Count realistic wall time including: writing copy, getting approvals, building
  assets, coordinating with team, publishing/launching.
  IF > 7 days → either scope down or note the longer horizon explicitly.

Filter 3 — Readable signal within 14 days?
  "Will we have a statistically or practically meaningful result by Day 14?"
  IF no → either extend the window (note it) or question whether the
  experiment is the right scope.

---

Step 2 — Channel-to-motion matching

Channel selection is a constraint, not a choice. The growth motion determines which channels can work. Using a PLG channel for SLG leads generates no pipeline. Using an SLG outreach sequence for a PLG product burns the developer community.

Motion	Primary channels
PLG	In-product flows, onboarding email (behavior-triggered), OSS/GitHub, HN (Show HN), dev communities (Discord, Reddit), technical X/Twitter
SLG	Outbound email/DM, LinkedIn (buyers and decision-makers), partnerships, case studies, conference outreach
MLG	IC channels (technical content, community, GitHub) in parallel with buyer channels (LinkedIn, case studies, ROI content)
CLG	Community-native channels: Discord, Slack, subreddits, GitHub Discussions, AMAs, hackathons

IF channel selected is mismatched with motion:
  FLAG. Return: "Channel [X] is typically used for [motion] motion.
  Your product appears to be operating in [Y] motion.
  Using [X] for [Y] motion tends to produce [specific failure mode].
  
  Consider: [2 channels better suited to Y motion]
  Or explain why you believe [X] works for your specific context."

---

Step 3 — Write the experiment spec

Every field is required. An experiment spec with missing fields is not executable.

## Experiment spec

**Hypothesis:**
"If we [specific action], then [named metric] will [direction] by [amount]
 because [mechanism — why this action moves that metric]."

Rules:
  - One sentence. Specific. Falsifiable.
  - "Improve conversion" is not a hypothesis. "Increase trial-signup-to-FVM
    rate from 22% to ≥30% by replacing the guided tour with an interactive
    quickstart" is a hypothesis.

---

**Tactic:**
[Who does what, when, where, in what format]

What specificity is required:
  - Distribution channel: not "post on LinkedIn" — which account, format,
    audience targeting, publishing time, cadence if multi-day
  - Audience: not "our users" — Core ICP segment (from ICP card), cohort,
    user segment, or explicit targeting criteria
  - Content/asset: if there's copy, describe the hook, the argument, and the CTA
  - Timing: when does this start and when does the window close?

---

**Success metric:**
[Named metric] [number threshold] [window]

Examples (all three elements required):
  ✓ "≥35% email open rate within 7 days vs 22% baseline"
  ✓ "≥50 GitHub stars within 48 hours of HN posting (vs typical new OSS repo: 0)"
  ✓ "+8 percentage point improvement in trial-to-activate rate in 30 days"
  ✗ "Improved engagement" (not specific)
  ✗ "> 35% open rate" (no window)
  ✗ "Better than last time" (no baseline)

---

**Secondary metrics (optional):**
[Any downstream signals worth tracking even if not the success criterion]

---

**Effort:**
[Realistic wall-clock estimate including copy writing, review/approval, asset
 creation, publishing, monitoring, analysis — not just execution time]
Example: "3-4 hours total: 1h writing/editing, 30min async approval, 30min
scheduling, 1h monitoring and first engagement response"

---

**Confidence:**
[What evidence supports this hypothesis? Qualitative is fine.]
Example: "The last HN post on a similar OSS tool got 300 upvotes. Our tool
solves a similar problem. Confidence: medium-high — but we don't know if
framing matters more than the tool itself."

---

**Distribution handoff (required — not optional):**
  Channels needing custom copy: [list each, one line per channel]
  Assets to create: [email template, landing page variant, image, video script, etc.]
  Dependencies: [what must be done BEFORE this experiment starts]
  Owner for each asset: [name or role]

Note: "Post it on LinkedIn" is not a distribution handoff. "Write a 200-word
LinkedIn post leading with the specific metric from the experiment, include
a 1-sentence CTA to the trial, post at 9 AM Pacific Tuesday from the
[account]" is.

---

Step 4 — Apply the confidence qualifier

Confidence levels:
  Low: single signal source, no prior experiments, new channel for this audience
  Medium: 2-3 signal sources or one prior positive signal
  High: multiple prior experiments, clear baseline, well-understood mechanism
  
Recommended action by confidence:
  Low → small scope, fast window; treat as learning experiment
  Medium → standard scope; measure carefully; read at natural window close
  High → appropriate scale; may be worth more budget or effort
  
Very low confidence experiments are valid — they surface the fastest.
Do not raise confidence by inflating the hypothesis.

---

Output format

## Experiment Spec

**Hypothesis:** "If we [action], then [metric] will [direction] by [amount] because [reason]."

**Tactic:**
[Specific who/what/when/where/format]

**Success metric:** [Metric] [threshold] [window]

**Effort:** [Realistic estimate]

**Confidence:** [Low / Medium / High] — [reason]

**Distribution handoff:**
  Channels needing custom copy:
    - [Channel 1]: [brief description of what's needed]
  Assets to create:
    - [Asset type]: [brief description + owner]
  Dependencies (must complete before start):
    - [Dependency]

**Window close:** [Date or trigger]
**Next step:** When window closes, run /experiment-results

---

Connects to

Upstream:

• /funnel-audit — audit identifies the bottleneck; this workflow designs the experiment to address it
• /retention-analysis — analysis identifies churn type; this workflow designs the experiment to address it
• /icp-research — Core ICP segment is the audience targeting input

Downstream: /experiment-results — required when the experiment window closes.

---

Brain reads / writes

If a companion aether-growth-brain repo is connected:

Before designing:

• Read experiments/experiment-log.md — check what has been tried; avoid re-running experiments that previously produced LOSS or CONFOUNDED verdicts without a deliberate change in variable
• Read knowledge/icp-map.md — confirm that experiment audience matches Core ICP segment

Brain write (when experiment spec is finalized):

• Append experiment entry to experiments/experiment-log.md with: name, hypothesis, metric, window, RICE score, status: PENDING
• Update status to WIN/NO-SIGNAL/LOSS/CONFOUNDED when /experiment-results closes the experiment

Brain not connected: proceed; recommend maintaining a local experiment log even without the brain repo.

---

Anti-patterns

Anti-pattern	Why it fails	Fix
Running 3+ experiments simultaneously on same funnel stage	Results are unreadable — you cannot attribute which change drove movement	Max 2 concurrent experiments on the same stage; use RICE to prioritize
Hypothesis after results (HARKing)	Post-hoc justification is not learning; creates false confidence	Require pre-registered spec before reading results; see /experiment-results
Success metric without baseline	Cannot tell if the result is good or bad without a before state	Always state baseline, even if approximate; "baseline unknown — will measure Day 1" is acceptable
Window that's too short for the signal	7-day A/B test on a 30-day activation metric produces meaningless data	Match signal window to product behavior cycle (D7, D30, W4)
Distribution handoff not specified	The experiment gets designed but never shipped	Require distribution handoff to be complete before the spec is approved
Treating No Signal as Loss	Kills experiments that might work with more sample	Enforce the No Signal ≠ Loss rule; always state sample needed for definitive answer
Expensive experiment first	High-effort, high-uncertainty experiments should not run before low-cost baseline data exists	Apply RICE; start with fast, cheap experiments to build baseline knowledge

---

Benchmarks (2025–2026)

Benchmark	Value	Source
Median experiment cycle time (design → result)	14–28 days	Reforge 2025
Email open rate benchmark (developer tools, warm list)	30–45%	Email marketing benchmark research, 2025
Email CTA click rate (developer tools)	3–8%	Email marketing benchmark research, 2025
HN Show HN upvotes for established OSS tool	100–400	field benchmark
Trial signup-to-activation rate (PLG SaaS median)	30–40%	Reforge 2025
PLG viral coefficient target	>0.3	Reforge 2025
Activation experiment win rate (first attempt)	~30%	Reforge 2025 (roughly 1 in 3 experiments produce a measurable lift)
LinkedIn post reach (thought leadership, no paid boost)	1–5× follower count	LinkedIn 2025
Speed-to-lead for inbound signals	<5 minutes	GTM Metrics benchmark 2025

---

Related skills

Skill	When to use
growth/funnel-audit/SKILL.md	Before designing: identify which stage to experiment on
growth/retention-analysis/SKILL.md	Before designing: identify which churn type to address
growth/experiment-results/SKILL.md	After window closes: required read for every experiment
pmm/content-review/SKILL.md	Before distribution: gate for public copy in the experiment
pmm/icp-research/SKILL.md	If bottleneck is fit gap: ICP may need updating before further experiments

---

Validation criteria

• Hypothesis is falsifiable (specific action, named metric, threshold, window)
• Channel matches growth motion
• Success metric has all three components (named metric + threshold + window)
• Effort is wall-clock realistic (includes approvals and asset creation)
• Distribution handoff completed with specific assets and owners
• "Next step: /experiment-results" included

References & Sources

Tier 1:

• RICE / ICE prioritization: effort estimate and confidence qualifier structure
• Reforge growth loops: channel-to-motion matching logic
• growth-experiment-design (growth-skills v1.0, score 7.5/10): hypothesis format, distribution handoff requirement