Metric design experimentation

Skill Avyayalaya/agent-prime/shared/toolkits/skills/metric-design-experimentation

A persistent AI operating system. 13 specialized agents + a runtime-portable 5-agent quality gate (Agent Council), 12 skills, 8 guardrails, 4 workflows. Built entirely from markdown files. Zero dependencies.

Install

npx -y skills add Avyayalaya/agent-prime --skill metric-design-experimentation

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use when designing a metric framework, selecting a North Star metric, building a metric decomposition tree, designing A/B experiments, setting up retention cohort analysis, or diagnosing whether a metric is being gamed. Encodes NSM rubrics, Goodhart's Law countermeasures, statistical validity for PMs, and retention curve methodology.

The file declares its own license as MIT. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

91.7 KB, as published. Nobody here has run it

Purpose

Produce a complete Measurement Framework — metric hierarchy (North Star → L1 → L2 → input), leading/lagging indicator pairs with temporal lag classification, counter-metric design that resists Goodhart's Law, experiment plans with statistical validity, and retention cohort methodology. The output is not a dashboard mockup or a list of KPIs — it is a metric engineering system: instrumented to detect problems early, paired to resist gaming, and validated causally. The artifact a PM cannot produce unaided.

When to Use / When NOT to Use

Use this skill when:

Launching a new product or feature and need to define what success looks like before building
Designing an A/B test or experiment plan with proper statistical rigor
An existing metric feels "off" — you suspect proxy divergence, gaming, or Simpson's paradox
Building a metric hierarchy for a team or org (North Star → team-level → input metrics)
Setting up retention cohort analysis to detect PMF erosion early
Evaluating whether a metric improvement is real or an artifact of denominator shift

Do NOT use this skill when:

You need SaaS finance metric definitions (MRR, ARR, CAC, LTV formulas → use a finance metrics reference)
You need dashboard layout or visualization design (that's a BI/design task)
You need to analyze experiment results that already exist (use the computation scripts directly)
You need customer research methodology (→ Discovery & Research skill — that's primary research, this is measurement design)

Anti-inputs (what this skill does NOT handle):

Finance metric calculation formulas (→ SaaS finance reference skills)
Data pipeline architecture (→ engineering)
Dashboard UI design (→ BI tooling)
Customer interview design (→ Discovery & Research skill)

Format Rules

These rules apply to every output from this skill. They are mandatory, not optional.

Rule 1: Take Positions with Calibrated Confidence

Never use weasel words in conclusions. Replace "likely," "may," "could," "seems" with explicit confidence levels:

H (>70%) — Strong evidence (validated in your own data)
M (40-70%) — Mixed or moderate evidence; direction is probable
L (<40%) — Evidence is thin or conflicting; treat as hypothesis

Why: A high-confidence NSM selection based on validated correlation requires different action than a low-confidence hypothesis about an activation metric. Undifferentiated confidence obscures this distinction.

Rule 2: Per-Cell Evidence Tier Annotation in Comparison Tables

Every table cell making a substantive claim carries an inline evidence tier:

Strong (T2) — validated in your own experiment or cohort data
Adequate (T4) — industry benchmark or comparable product data
Unknown (T6: inferred) — theoretical reasoning or first principles only

Why: A retention target of 40% from your own validated cohorts (T2) carries different weight than a general SaaS benchmark (T4). Conflating them leads to miscalibrated goals.

Rule 3: Metric Intervention Cascade for All Recommendations

Every metric design recommendation or triggered intervention follows:

OBSERVATION [evidence tier] → IMPLICATION [mechanism] → RESPONSE [specific action + owner] → CONFIDENCE [H/M/L + key assumption] → WATCH INDICATOR [observable signal]

Why: The most common failure in metric frameworks is disconnected recommendations — "increase activation rate" without specifying the mechanism, owner, or the observable signal that confirms the intervention is working.

Rule 4: Framework Selection Before Application (Step 0)

Different metric design questions require different framework subsets. Always apply the Step 0 routing table before selecting which of the 9 frameworks to use.

Why: Applying all 9 frameworks to every question inflates output 2-3x without improving decision quality. An experiment design question doesn't need PMF assessment; a PMF diagnosis doesn't need MAB algorithm selection.

Rule 5: Surface Contradictions Between Frameworks

When NSM selection criteria conflict with experiment design constraints, or retention curve analysis contradicts PMF signals, surface the contradiction explicitly. Do not resolve artificially.

Why: Contradictions are the most valuable signals. A product showing strong NSM growth alongside a degrading cohort retention curve has a structural problem that "reconciling" the two metrics would hide.

Rule 6: Staleness Flags

Any benchmark, threshold, or normative value based on data older than 6 months must carry [POTENTIALLY STALE — verify before using].

Rule 7: Evidence-Limited Flags

If a metric design recommendation rests only on theoretical reasoning (T6) or general SaaS benchmarks without validation in your own data, prepend: [EVIDENCE-LIMITED: validate with your own data before acting].

Rule 8: Framework References Get One-Line Context

Not "Goodhart's Law (Regressional variant)" but "Goodhart's Law — when optimizing a proxy metric causes it to diverge from the outcome you actually care about. The 'regressional' variant means the metric was a good proxy initially but optimization pushed it past the point where it predicts the real outcome." The reader needs to understand why a framework matters for their decision, not just its academic name.

Rule 9: The Document Must Be Navigable by Non-Creators

Include a reading guide (by time and by role), a notation key, and layered depth. A VP should be able to read only the Executive Summary and know whether the measurement plan is sound. A PM lead should be able to read through the metric hierarchy and experiment plans. A data scientist should be able to dive into statistical design. No reader should encounter unexplained notation.

Output Template (Mandatory Document Skeleton)

Every Measurement Framework MUST follow this exact structure. Copy this skeleton and fill it in. Do not reorder sections, skip sections, or invent new top-level sections. If a framework was skipped in Step 0, note "Skipped — not load-bearing for this question type" in that section.

# Measurement Framework: [Subject — e.g., "Co-Editing Feature Launch Metrics"]

> **Date:** [YYYY-MM-DD] | **Confidence band:** [Overall H/M/L] | **Staleness window:** [Date after which benchmarks and thresholds need revalidation]

---

## Executive Summary

[5 sentences max. A VP reads only this and decides whether the measurement plan is sound. No framework names, no statistical jargon, no evidence tier tags. Plain language. Final sentence = the single most important metric to watch in bold.]

---

## How to Read This Document

**What this is:** A measurement engineering system — not a KPI list. It defines what to measure, how to know if it's working, what could go wrong, and how to detect problems early.

**Reading by time available:**

| Time | Read | You'll get |
|---|---|---|
| **5 min** | Executive Summary only | Whether the measurement plan is sound + the key metric to watch |
| **15 min** | Executive Summary + Metric Hierarchy (section 2) + Experiment Plan (section 5) | What we're measuring, how we're testing, and the decision rules |
| **30 min** | Full document through Recommendations | Complete metric system with counter-metrics, retention design, and interventions |
| **Deep dive** | Everything including Appendix | Statistical design details, gaming detection, assumption stress-testing |

**Reading by role:**

| Role | Start with | Then read | Skip unless curious |
|---|---|---|---|
| VP / Exec | Executive Summary | Metric Hierarchy (section 2), Review Cadence | Statistical design, Goodhart analysis |
| PM Lead | Executive Summary | Sections 1-5 (NSM through Experiments), Recommendations | MAB algorithm selection, Statistical Validity details |
| Data Scientist / Analyst | Full document in order | Statistical Validity (section 5), Retention Cohorts (section 8), Adversarial Self-Critique | Outcome methodology (they know this) |
| Engineering Lead | Executive Summary | Instrumentation Feasibility, Experiment Plan (duration/sample/unit) | Framework theory sections |

---

## Notation Key

**Confidence levels** — applied to every metric design conclusion:
- **H (>70% confident)** — Validated in your own data. Act on it.
- **M (40-70%)** — Based on comparable products or reasonable inference. Validate before committing.
- **L (<40%)** — Hypothesis only. Must be tested before treating as a metric target.

**Evidence tiers** — how we know what we claim to know (tagged inline as T1-T6):
- **T1** — Direct internal behavioral data: your own usage analytics, experiment results (strongest)
- **T2** — Primary research on your users: well-sampled surveys, structured interviews
- **T3** — Expert analysis with methodology: published case studies, Reforge frameworks
- **T4** — Industry benchmarks: SaaS averages, Gartner norms (useful for direction, not targets)
- **T5** — Comparable product claims: competitor announcements, case studies without methodology
- **T6** — First-principles reasoning or general best practices (weakest — treat as starting hypothesis)

**Metric status indicators:**
- ✅ — Criterion met or metric healthy
- ❌ — Criterion failed or metric degrading
- ⚠️ — Warning signal; investigate

**Recommendation format** (O→I→R→C→W):
- **O**bservation — What the data shows (with evidence tier)
- **I**mplication — Why it matters (the mechanism)
- **R**esponse — What to do (specific action + owner + timeline)
- **C**onfidence — How sure we are (H/M/L + key assumption)
- **W**atch — How to know if we're wrong (observable signal)

**Flags:**
- `[POTENTIALLY STALE]` — Benchmark data is >6 months old; verify before using as a target
- `[EVIDENCE-LIMITED]` — Recommendation based on T4-T6 only; validate with your own data before acting

---

## Step 0: Context Fitness Check

Before selecting frameworks, verify that a Measurement Framework is the right artifact and that you have the data access to produce one.

| Question | If Yes | If No |
|---|---|---|
| **Do you have access to the product's usage data?** | Analysis can set validated targets (T1-T2 evidence) | All targets are benchmarks or hypotheses. Flag prominently: "Targets below are industry benchmarks (T4) — replace with your own validated thresholds before operationalizing." |
| **Has the product been live long enough for retention data?** | Retention cohort design can use real curves | Retention targets are hypothetical. Design the instrumentation to collect this data; don't set targets you can't yet validate. |
| **Is the metric system for a new product or an existing one?** | New: focus on F1 (NSM), F4 (Experiments), F9 (PMF). Existing: focus on F3 (Goodhart), F6 (Cohorts), F2 (Leading/Lagging). | — |
| **Who will operationalize this framework?** | If data team: full statistical depth. If PM without data support: simplify experiment design, focus on hierarchy + counter-metrics. | Match the framework's complexity to the team that will maintain it. A beautiful statistical design that nobody monitors is a planning artifact. |

---

## Step 0b: Framework Selection

| Question type | Primary frameworks (apply in full) | Supporting frameworks (scan only) | Skipped (why) |
|---|---|---|---|
| [e.g., "New feature launch"] | [e.g., F1 NSM + Decomposition, F2 Leading/Lagging, F3 Counter-Metrics, F4 Experiment Design] | [e.g., F8 HEART] | [e.g., "F7 MAB — insufficient traffic for multi-arm. F9 PMF — PMF already validated."] |

---

## 1. Value Moment & North Star Metric

**Value moment:** [The specific instant the user receives core value.]

**NSM Candidate Evaluation:**

| Candidate NSM | Value Reflection | Leading Nature | Influenceability | Simplicity | Non-Gameability | Score |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| [Candidate 1] | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | X/5 |
| [Candidate 2] | | | | | | X/5 |
| [Candidate 3] | | | | | | X/5 |

**Selected NSM:** [Winner + one-sentence explanation a new hire could repeat.]

**GSM validation:** Goal → [what outcome?] | Signal → [what user behavior?] | Metric → [how measured?]

---

## 2. Metric Decomposition Tree

| Level | Metric | Owner | Cadence | Target | Counter-Metric |
|---|---|---|---|---|---|
| **NSM** | [metric] | [exec] | Monthly | [target] (TX) | [counter-metric] |
| **L1** | [metric] | [PM] | Weekly | [target] (TX) | [counter-metric] |
| **L1** | [metric] | [PM] | Weekly | [target] (TX) | [counter-metric] |
| **L1** | [metric] | [PM] | Weekly | [target] (TX) | [counter-metric] |
| **L2** | [metric] | [feature team] | Daily | [target] (TX) | [counter-metric] |
| **L2** | [metric] | [feature team] | Daily | [target] (TX) | [counter-metric] |
| **Input** | [metric] | [eng lead] | Per-deploy | [target] (TX) | — |

**Causal chain check:** [Trace from each input metric to the NSM in ≤3 steps. Flag any broken branch.]

---

## 3. Leading / Lagging Indicator Pairs

| Lagging Metric | Leading Indicator | Temporal Lag | Correlation (est.) | Causal? | Alert Threshold |
|---|---|---|---|---|---|
| [metric] | [leading indicator] | Immediate/Short/Medium | r ≈ X.XX (TX) | Yes/Hypothesis | [threshold = action] |
| [metric] | [leading indicator] | | | | |

**Activation Metric (Aha Moment Protocol):**
1. **Aha moment hypothesis:** [What action predicts retention?]
2. **Correlation check:** [r value between action and retention] (TX)
3. **Threshold:** [X actions within Y days]
4. **Causal validation plan:** [Experiment to confirm causation, not just correlation]

⚠️ [Flag whether activation metric is validated or hypothesized. If hypothesized, mark as `[EVIDENCE-LIMITED]`.]

---

## 4. Counter-Metric Design & Goodhart Vulnerability

| Primary Metric | Most Likely Goodhart Variant | What Goes Wrong | Counter-Metric | Threshold | Gaming Detection Pattern |
|---|---|---|---|---|---|
| [metric] | Regressional/Extremal/Causal/Adversarial | [specific gaming scenario] | [counter-metric] | [failure threshold] | [observable signal of gaming] |
| [metric] | | | | | |
| [metric] | | | | | |

**Quarterly Health Review Protocol:**
- **Cadence:** [e.g., Every 13 weeks]
- **Owner:** [name/role]
- **Decision framework:** Keep (proxy-outcome r > 0.5) / Recalibrate (r 0.3-0.5) / Replace (r < 0.3)

---

## 5. Experiment Plan

**Experiment 1: [Title]**

| Field | Value |
|---|---|
| Hypothesis | [If we do X, metric Y will improve by ≥ Z] |
| Primary metric | [single metric] |
| Secondary metrics | [exploratory, Bonferroni-corrected at α = X] |
| Guardrail metrics | [thresholds that must hold] |
| MDE | [Xpp — business rationale for this threshold] |
| α / Power | 0.05 / 0.80 |
| Sample size | [computed via `sample_size_calculator.py` — state result or flag "script not available"] |
| Duration | [X days — rationale: covers Y cycles] |
| Randomization unit | [user/session/cluster — rationale] |
| Exclusions | [who is excluded and why] |
| Decision rule | [Ship if... Do NOT ship if...] |

**Experiment Quality Score:** X/6 (pre-registered hypothesis, single primary metric, guardrails declared, duration committed, sample size computed, segmentation planned)

**Experiment 2: [Title]**
[Same structure]

**"When NOT to Experiment" Check:**
- [ ] Is the change reversible? If not, consider staged rollout instead.
- [ ] Is there enough traffic? If sample size > 4 weeks of traffic, the experiment is infeasible.
- [ ] Is the ethical bar met? If the control group is harmed by withholding, use quasi-experimental.

---

## 6. HEART Framework (if applicable)

| Dimension | Goal | Signal | Metric | Target | Counter-Metric |
|---|---|---|---|---|---|
| Happiness | [goal] | [signal] | [metric] (TX) | [target] | [counter] |
| Engagement | | | | | |
| Adoption | | | | | |
| Retention | | | | | |
| Task Success | | | | | |

[Select 2-3 relevant dimensions. Do NOT force all five. Note "Skipped — [reason]" for unused dimensions.]

---

## 7. PMF Assessment (if applicable)

**Ellis Test:** [% who would be "very disappointed" if product disappeared] = X% (TX)
- ≥40% = PMF signal present | 25-40% = weak signal | <25% = PMF not established

**Behavioral PMF Check:**

| Signal | Value | Assessment |
|---|---|---|
| Retention curve shape | Smile / Flat / Frown | [interpretation] |
| Organic referral rate | X% (TX) | [benchmark comparison] |
| Activation-to-retention correlation | r = X.XX (TX) | [strong/weak/unvalidated] |

**Segmentation:** [Where is PMF strongest? Segment by channel, use case, activation behavior.]

---

## 8. Retention Cohort Design

**Primary cohort type:** [Time-based / Behavior-based / Channel-based]

**Retention Windows:**

| Window | Definition | Benchmark | Degradation Threshold |
|---|---|---|---|
| Day 1 | [definition] | X% (TX) | >Xpp decline vs. prior cohort |
| Day 7 | | | |
| Day 14 | | | |
| Day 30 | | | |
| Day 60 | | | |
| Day 90 | | | |

**Cohort Cuts:**
- **Time-based:** [weekly/monthly signup cohorts]
- **Behavior-based:** [activated vs. not activated — validates activation metric]
- **Channel-based:** [organic vs. paid — detects acquisition quality shifts]

**Retention Curve Shape:** [Smile (recovering) / Flat (stable) / Frown (decaying)] — interpretation and action.

**Revenue vs. Logo:** [Track both. Note if they diverge — expansion revenue can mask logo churn.]

---

## 9. Metric Intervention Recommendations (O→I→R→C→W Cascade)

**Intervention 1: [Title]**
- **Observation** [TX]: [What the data shows]
- **Implication**: [Why it matters — the mechanism]
- **Response**: [Specific action + owner + timeline]
- **Confidence**: [H/M/L] — assumes [key assumption]
- **Watch**: [Observable signal]; if [threshold], re-assess

**Intervention 2: [Title]**
- **Observation** [TX]: ...
- **Implication**: ...
- **Response**: ...
- **Confidence**: ...
- **Watch**: ...

---

## Cross-Framework Contradictions

| Contradiction | Framework A says | Framework B says | Resolution / Which to weight |
|---|---|---|---|
| [e.g., "NSM growth vs. cohort degradation"] | [NSM trending up] | [Retention cohorts degrading] | [Which matters more and why — e.g., "Cohort signal is more structural; NSM growth is masking mix shift"] |

---

## Instrumentation Feasibility

| Metric | Data exists? | Clean & reliable? | Timely (within cadence)? | ≥30 days history? | Status |
|---|:---:|:---:|:---:|:---:|---|
| [metric] | ✅/❌ | ✅/❌ | ✅/❌ | ✅/❌ | Ready / Needs work / Blocked |

[Flag any metric that fails ≥1 check. Propose measurable proxy or instrumentation investment.]

---

## Review Cadence & Ownership

| Metric Level | Review Cadence | Owner | Escalation Trigger |
|---|---|---|---|
| NSM | Monthly | [exec] | [e.g., ">10% decline sustained 2 months"] |
| L1 | Weekly | [PM] | [e.g., "Misses target 3 consecutive weeks"] |
| L2 | Daily | [feature team] | [e.g., "Alert threshold crossed"] |
| Experiments | Per-experiment | [PM + data] | [e.g., "Guardrail violated"] |
| Quarterly health review | Every 13 weeks | [owner] | [e.g., "Any proxy-outcome r < 0.5"] |

---

## Assumption Registry

| # | Assumption | Framework it underpins | Confidence | Evidence | What would invalidate this |
|---|---|---|---|---|---|
| 1 | | | H/M/L | (TX) | |
| 2 | | | H/M/L | (TX) | |
| 3 | | | H/M/L | (TX) | |

---

## Adversarial Self-Critique

**Weakness 1: [Title]**
[What assumption is being made? What evidence would disprove it? Is there a scenario where this metric system is catastrophically wrong? Link to a Watch Indicator.]

**Weakness 2: [Title]**
[Same depth]

**Weakness 3: [Title]**
[Same depth]

---

## Revision Triggers

| Trigger | What to re-assess | Timeline |
|---|---|---|
| NSM-outcome correlation drops below r = 0.5 | NSM selection | Next quarterly review |
| Leading indicator no longer predicts lagging | Leading/lagging pairs | Immediate investigation |
| Assumption Registry item invalidated | Dependent framework sections | Within 1 week |
| Retention curve shape changes | PMF assessment + cohort design | Next quarterly review |
| Counter-metric threshold crossed 2+ consecutive periods | Primary metric + gaming detection | Immediate investigation |

---

## Sources

[All sources cited in the framework, with evidence tier and date.]

Rules for using this template:

Do not skip sections. If a section isn't applicable (e.g., HEART skipped in Step 0), write "Skipped — [reason]" and move on.
Every table cell with a target, benchmark, or claim must have an evidence tier tag — (T1) through (T6).
Section headers are conclusions, not labels. Replace generic headers (e.g., "Retention Analysis") with insight headers (e.g., "Day-30 Retention Is Masking a Cohort Degradation Problem") after completing the section.
The Executive Summary is written last but appears first. Do not write it until all sections are complete.
Instrumentation Feasibility is not optional. A beautiful metric that can't be measured is a planning artifact, not a measurement framework.

Domain Frameworks

This section IS the knowledge weapon. Each framework is encoded with its scoring rubrics, decision tables, and output format templates — not merely referenced. A PM using this skill designs metric systems that resist gaming, detect problems early, and validate causally; without these frameworks, the output degrades to a KPI list.

Framework 1: North Star Framework + Metric Decomposition Tree

The foundational architecture for metric systems. A North Star Metric (NSM) captures the core value your product delivers. The decomposition tree breaks it into actionable layers.

NSM Selection Rubric:

Score every candidate NSM against these 5 criteria. A viable NSM scores ≥4/5:

Criterion	Question	Score 1 (Fail)	Score 0 (Pass)
Value reflection	Does this metric move when customers get value?	Metric can rise while customers are unhappy (e.g., pageviews from rage-clicks)	Metric movement = value delivered
Leading nature	Does this predict future outcomes, not just record past ones?	Revenue (lagging — already happened)	Weekly active usage (leads to retention and revenue)
Influenceability	Can the product team move this metric through their work?	Total market size (can't influence)	Activation rate (directly influenceable)
Simplicity	Can you explain it in one sentence to a new hire?	"Revenue-weighted DAU adjusted for seasonal variance"	"Weekly users who complete a core action"
Non-gameability	Is it hard to inflate without delivering real value?	Registered accounts (create bots)	Weekly retained users who complete value action

Decision point: If your best candidate scores 3/5, strengthen it before proceeding. If multiple candidates score 5/5, pick the one closest to the value moment — the instant the user gets what they came for.

Metric Decomposition Tree — 4 Levels:

North Star Metric (NSM)
├── L1 Metrics (3-5) — Exec/PM-owned, reviewed weekly
│   ├── L2 Metrics (2-4 per L1) — Feature-team-owned, reviewed daily
│   │   └── Input Metrics — Directly manipulable by shipping code
│   ├── L2 Metrics
│   │   └── Input Metrics
│   └── L2 Metrics
│       └── Input Metrics
├── L1 Metrics
│   └── ...
└── L1 Metrics
    └── ...

Decomposition Method:

Start with the NSM. Ask: "What 3-5 things must ALL be true for this metric to grow?"
For each L1, ask: "What 2-4 sub-components drive this L1?" These are L2 metrics.
For each L2, ask: "What can an engineer ship this sprint to move this?" Those are input metrics.
Validation check: Can you trace from every input metric back up to the NSM with a clear causal story? If not, the tree has a broken branch.

Metric-to-Team Ownership Mapping:

Level	Owner	Review Cadence	Example
NSM	CEO / CPO	Monthly board review	Weekly active teams completing a project
L1	VP / Senior PM	Weekly leadership review	Activation rate, retention rate, expansion rate
L2	PM / Feature lead	Daily standup / weekly sprint	Onboarding completion, feature adoption, invite sent
Input	Engineering lead	Sprint-level, per-deploy	Page load time, error rate, funnel step conversion

GSM (Goals-Signals-Metrics) Integration:

Before decomposing, validate the NSM using Google's GSM framework:

Component	Question	Example
Goal	What user outcome are we trying to achieve?	Users successfully collaborate on documents in real time
Signal	What user behavior would tell us the goal is being achieved?	Multiple users editing the same document within a 5-minute window
Metric	How do we measure that signal at scale?	Weekly collaborative editing sessions (≥2 users, same doc, <5 min gap)

Key insight: The signal is the hardest part. Most PMs jump from Goal to Metric and skip Signal entirely — which is how you end up measuring pageviews when you meant to measure collaboration.

Output format — Metric Hierarchy Table:

| Level | Metric | Owner | Cadence | Target | Counter-Metric |
|-------|--------|-------|---------|--------|----------------|
| NSM   | Weekly active teams completing a project | CPO | Monthly | 50K | Quality score ≥ 4.0 |
| L1    | Activation rate (day-7) | PM, Growth | Weekly | 40% | Time-to-activate ≤ 3 days |
| L1    | Week-4 retention | PM, Core | Weekly | 65% | Session depth ≥ 3 actions |
| L2    | Onboarding completion | PM, Onboarding | Daily | 70% | Support tickets from new users |
| L2    | Invite-sent rate | PM, Virality | Daily | 25% | Invite-spam complaints |
| Input | Onboarding step-3 conversion | Eng | Per-deploy | 85% | Error rate < 1% |

Framework 2: Leading vs. Lagging Indicator Design

Every metric has a temporal character. Lagging metrics tell you what already happened. Leading metrics tell you what's about to happen. A measurement framework without leading indicators is a rearview mirror — you'll see the cliff after you've driven off it.

Temporal Lag Classification:

Category	Lag Time	Monitoring Cadence	Examples
Immediate	Minutes to hours	Real-time alerts	Error rate, page load, API latency, crash rate
Short	1-3 days	Daily dashboard	Daily active usage, signup rate, onboarding starts
Medium	1-4 weeks	Weekly review	Activation rate, feature adoption, NPS responses
Long	1-6 months	Monthly/quarterly	Retention rate, revenue, LTV, churn
Structural	6-24 months	Quarterly strategy review	Market share, brand perception, platform adoption

The Lag-to-Cadence Rule: Your monitoring cadence must match the metric's lag category. Checking a structural metric daily creates noise anxiety. Checking an immediate metric monthly means you miss fires.

Activation Metric Design (the "Aha Moment" Protocol):

The activation metric is the single most valuable leading indicator — it predicts long-term retention from early behavior.

Step	Action	Example
1	Define the lagging outcome you want to predict	90-day retention
2	List early user behaviors (first 7-14 days)	Created project, invited teammate, completed task, viewed report, connected integration
3	Correlate each behavior with the lagging outcome	Users who invited ≥1 teammate in week 1 → 3.2x more likely to retain at day 90
4	Identify the top 2-3 most predictive behaviors	Invite teammate (3.2x), complete first project (2.8x), connect integration (2.1x)
5	Set threshold and timeframe	"Invited ≥1 teammate within 7 days" = activated
6	Validate: does causing the behavior cause retention?	Run experiment: prompt invite flow → does retention actually improve?

Critical warning — Step 6 is where most teams fail. Correlation (people who invite retain better) ≠ causation (making people invite causes retention). The user who invites may simply be more engaged. You MUST run the causal experiment before declaring this your activation metric.

Leading Indicator Discovery Protocol:

For every lagging metric in your hierarchy, identify 2-3 leading indicators using this template:

| Lagging Metric | Leading Indicator | Lag Time | Correlation | Causal? | Threshold |
|---------------|-------------------|----------|-------------|---------|-----------|
| Monthly churn  | Support ticket vol | 2-3 weeks | r=0.72 | Likely (test) | >2x baseline = alert |
| Monthly churn  | Login frequency drop | 1-2 weeks | r=0.68 | Yes (validated) | <50% of avg = alert |
| Revenue growth | Pipeline velocity | 4-6 weeks | r=0.81 | Yes | <80% of target = alert |

Framework 3: Counter-Metric Design & Goodhart's Law

"When a measure becomes a target, it ceases to be a good measure." — Charles Goodhart

Every PM knows the quote. Almost none can name the four mechanisms by which it happens, or design metric systems that resist them.

Goodhart's Law — The 4 Variants (Manheim & Garrabrant):

Variant	Mechanism	Example	Counter-Strategy
Regressional	Metric was a proxy. Optimizing proxy diverges from true outcome.	Optimizing click-through rate → clickbait proliferates → user satisfaction drops	Pair with downstream quality metric (e.g., time-on-page, return rate)
Extremal	At extreme optimization, the metric-outcome relationship breaks down.	Maximizing "features shipped" → tiny features, no impact	Pair with outcome metric (feature adoption rate, not count)
Causal	Intervening on the metric destroys the causal link to the outcome.	Paying users to invite friends → invites up, quality of invitees down	Measure downstream behavior of acquired users, not just acquisition
Adversarial	Agents actively game the metric for reward.	Support team closes tickets without resolution to hit "resolution time" target	Measure reopened tickets, customer satisfaction post-resolution

Counter-Metric Pairing Template:

For every primary metric, complete this row:

| Primary Metric | What Could Go Wrong | Counter-Metric | Threshold | Review |
|---------------|--------------------|--------------------|-----------|--------|
| Signup rate | Fake accounts, low-quality signups | Day-7 activation rate | Must stay ≥ 35% | Weekly |
| Feature adoption | Forced adoption, dark patterns | User satisfaction (CSAT) | Must stay ≥ 4.0/5 | Bi-weekly |
| Resolution time | Premature ticket closure | Reopen rate | Must stay ≤ 8% | Weekly |
| Revenue per user | Price increases erode retention | 90-day retention | Must stay ≥ 70% | Monthly |
| DAU | Notification spam drives opens | Session duration, uninstall rate | Duration ≥ 3min, uninstall ≤ 2% | Weekly |

Guardrail Metric Protocol:

Guardrails are the non-negotiables — metrics that must NOT degrade when you optimize a primary metric. Every experiment must declare guardrails before launch.

Step	Action
1	List what could degrade if your primary metric succeeds
2	Assign a guardrail metric to each risk
3	Set a threshold: "experiment fails if guardrail crosses X"
4	Pre-register guardrails in the experiment plan
5	If primary wins but guardrail fails → do NOT ship

Gaming Detection Patterns:

Signal	What It Suggests	Investigation
Sudden spike without product change	External manipulation or instrumentation error	Check by segment, channel, device
Metric improves only at reporting boundaries	Reporting-cycle gaming (end-of-quarter, end-of-sprint)	Plot daily, look for sawtooth pattern
Primary metric up, counter-metric down	Goodhart's Law is active	Activate quarterly metric health review
Metric improves for one segment, degrades for others	Simpson's paradox or targeted gaming	Segment decomposition, check overall vs. cohort
Leading and lagging indicators moving in opposite directions	Proxy divergence — leading indicator no longer predicts lagging	Re-validate correlation, consider metric rotation

Quarterly Metric Health Review:

Every quarter, review each metric in the hierarchy:

Question	If "No" → Action
Does this metric still correlate with the outcome we care about?	Re-validate. If r < 0.5 → replace or recalibrate
Has anyone changed behavior specifically to hit this metric?	Assess Goodhart variant. Add counter-metric or rotate
Is the proxy still a good proxy?	Check proxy-outcome correlation trend. Declining = divergence
Are counter-metrics stable?	If counter-metric degraded → primary metric gains are illusory
Decision	Keep / Recalibrate (adjust threshold or definition) / Replace (metric has been gamed out)

Framework 4: Experiment Design

Not every question needs an experiment. Not every experiment should be an A/B test. This framework encodes when to experiment, what type to use, and how to design it properly.

Experiment Type Decision Table:

Question	If Yes →	If No →
Can you randomly assign individual users?	A/B test (gold standard)	↓
Are there marketplace/network effects between users?	Switchback experiment (randomize by time window, not user)	↓
Can you find a natural comparison group?	Quasi-experimental (diff-in-diff, propensity matching)	↓
Do you need to stop early if results are clear?	Sequential testing (controls for peeking)	↓
Is traffic too low for any of the above?	Pre/post with caution or qualitative validation	Document why experiment wasn't feasible

A/B Test Design Protocol:

Complete every field before launching. Incomplete fields = incomplete experiment.

| Field | Value | Notes |
|-------|-------|-------|
| Hypothesis | [If we change X, metric Y will improve by Z%] | Must be falsifiable |
| Primary metric | [ONE metric] | Pre-declared. This is what decides ship/no-ship |
| Secondary metrics | [2-3 metrics] | Exploratory. Bonferroni-corrected if used for decisions |
| Guardrail metrics | [1-3 metrics] | Must NOT degrade. Pre-set thresholds |
| MDE | [X%] | Smallest effect worth detecting (business decision, not stats) |
| Significance level (α) | [0.05 typical] | False positive tolerance |
| Power (1-β) | [0.80 minimum] | False negative tolerance |
| Sample size (N) | [computed] | Use sample_size_calculator.py |
| Duration | [X days/weeks] | Must cover full weekly cycle (min 7 days) |
| Randomization unit | [user / session / org / region] | Must match analysis unit |
| Exclusions | [who is excluded and why] | New users only? Existing only? |
| Decision rule | [Ship if primary ≥ MDE AND guardrails hold] | Written before launch |

When NOT to Experiment — Checklist:

Condition	Why	Alternative
Ethical concern	Can't withhold a safety feature from control group	Ship to 100%, monitor pre/post
Insufficient traffic	Need 50K users but have 5K	Qualitative research, pre/post, or longer duration
Irreversible change	Database migration, pricing change with contracts	Staged rollout with monitoring, not true A/B
Legal/regulatory	GDPR, healthcare, financial regulations	Consult legal. Document decision
Network/spillover effects	Treatment users affect control users (social product)	Cluster randomization or switchback
The answer is obvious	500ms → 50ms page load; broken flow fix	Just ship it. Experimentation has opportunity cost

Experiment Quality Rubric:

Score each experiment 0-6. Below 4 = do not trust results.

Criterion	1 point if...
Pre-registration	Hypothesis, primary metric, and sample size documented before launch
Single primary metric	ONE metric decides ship/no-ship (not "whichever metric moves")
Guardrails declared	At least 1 guardrail metric with pre-set threshold
Duration committed	Full duration run (no early stopping without sequential testing framework)
Sample size committed	Computed via power analysis, not guessed
Segmentation planned	Key segments identified for heterogeneous treatment effect analysis

Framework 5: Statistical Validity for PMs

This section explains statistical concepts in plain English. No formulas. No textbook definitions. The computation scripts handle the math — this section builds the intuition to use them correctly.

Significance (α) — "How sure are you this isn't a fluke?"

Imagine flipping a coin 10 times and getting 7 heads. Suspicious? Maybe. Now imagine 100 flips and 70 heads. Very suspicious. Statistical significance measures whether your experiment result is suspicious enough to believe.

α = 0.05 means: "I'm accepting a 5% chance this result is a coincidence." One in twenty experiments will show a "win" that's actually noise.
Lower α (0.01) = more cautious. Fewer false alarms, but you'll miss some real effects.
Higher α (0.10) = more aggressive. Catch more real effects, but more false alarms too.
PM rule of thumb: Use 0.05 for most tests. Use 0.01 for high-stakes, irreversible decisions (pricing, major redesigns). Use 0.10 for quick directional learnings you'll validate later.

Power (1-β) — "Would you even notice if something real happened?"

Power is like a metal detector's sensitivity. A cheap detector (low power) misses gold coins buried 6 inches deep. A good detector (high power) finds them.

Power = 0.80 means: "If there IS a real effect, I have an 80% chance of detecting it." Twenty percent of the time, a real improvement goes undetected.
Low power is the silent killer. You run a test, see "no significant result," and conclude "the feature doesn't work." But maybe it DOES work — your test just wasn't powerful enough to detect it.
What kills power: Small sample size, small effect, high variance in the metric.
PM rule of thumb: Never run a test with power below 0.80. If your calculator says you need 100K users and you have 10K, don't run the test — you'll learn nothing.

MDE (Minimum Detectable Effect) — "What's the smallest win worth knowing about?"

Before every test, decide: "What's the smallest improvement that would change my decision?"

If a 0.5% conversion lift isn't worth the eng effort to maintain, don't design a test to detect 0.5%. Set MDE at 2% or 5%.
Smaller MDE = bigger sample size = longer test. This is a business tradeoff, not a statistics one.
The conversation to have: "Are we willing to run this test for 6 weeks to detect a 1% lift? Or would a 3% lift be the threshold for shipping, in which case we need only 2 weeks?"

Confidence Intervals — "Where does the truth probably live?"

A p-value tells you "significant or not" — binary. A confidence interval tells you "the real effect is probably between X% and Y%."

Result: "+3.2% conversion, 95% CI [0.8%, 5.6%]" means the true effect is probably between 0.8% and 5.6%. The point estimate is 3.2%, but it could be as low as 0.8%.
PM decision rule: Look at the bottom of the confidence interval. If the worst case (0.8%) is still worth shipping, ship confidently. If the worst case crosses zero or goes negative, the result is fragile.
Always report CIs, not just p-values. A "significant" result with CI [0.01%, 6.0%] is very different from [2.5%, 3.5%] — same p-value, wildly different confidence.

Sample Size — "How long do we need to run this?"

Sample size is determined by three inputs: your baseline rate, your MDE, and your α/power choices. It is not a guess.

Use sample_size_calculator.py with your inputs. It outputs the required N per variant and estimated duration given your daily traffic.
The most common mistake: "Let's run it for two weeks and see." Two weeks might give you 50% power — equivalent to flipping a coin to decide.
Duration must cover full cycles. Weekday vs. weekend behavior differs. Minimum 7 days. Ideally 14+ days for B2B (Monday-Friday cycle bias).

Multiple Testing Correction — "Testing 10 things and calling the winner is cheating."

If you test 10 metrics at α = 0.05, there's a ~40% chance at least one shows "significant" by pure chance. This is the multiple testing trap.

Pre-declare one primary metric. This is your ship/no-ship decision. No correction needed for one metric.
Bonferroni for secondaries. If you're testing 5 secondary metrics, use α = 0.05/5 = 0.01 for each. Simple and conservative.
Confirmatory vs. exploratory. Primary metric = confirmatory (strict). Secondary metrics = exploratory (generate hypotheses for next experiment, don't ship on them).

Framework 6: Retention Cohort Methodology

Retention is the metric that separates real products from leaky buckets. But "retention" is not one number — it's a family of curves that tell different stories depending on how you slice them.

Cohort Construction Types:

Type	Grouping Logic	When to Use	What It Reveals
Time-based	Users who signed up in the same week/month	Default. Always start here	Overall retention trends over time
Behavior-based	Users who performed a specific action (e.g., "completed onboarding")	When testing activation hypotheses	Whether specific behaviors predict retention
Acquisition channel	Users from the same source (organic, paid, referral)	When evaluating channel quality	Which channels produce sticky users vs. churny ones
Plan/tier	Users on the same pricing tier	When evaluating monetization impact	Whether pricing correlates with engagement

Retention Curve Analysis:

Curve Shape	Name	What It Means	Action
Drops then flattens	Smile curve ✅	Users who survive the initial drop are retained. Product has PMF for a segment.	Optimize activation to move more users past the drop point
Drops and keeps dropping	Frown curve ❌	No stable retained base. Even "retained" users are slowly leaving.	Product-market fit issue. Investigate core value proposition
Drops then rises	Resurrection curve	Users return after absence (seasonal, event-driven)	Understand re-engagement triggers. Design nudges around them
Flat high from start	Habitual curve	Very strong PMF. Users adopt and stay immediately.	Rare and wonderful. Focus on acquisition, not retention

Cohort Degradation Detection:

The most dangerous signal in a metric system is invisible in blended metrics: newer cohorts retaining worse than older ones.

Step	Action	Red Flag
1	Plot day-30 retention for each monthly signup cohort
2	Compare across cohorts chronologically	Each new cohort's day-30 is worse than the prior
3	If degrading → assess rate of degradation	>2% absolute decline per cohort = urgent
4	Segment by channel — is degradation channel-specific?	If paid-only degradation → channel quality issue, not product
5	Segment by user type — is degradation segment-specific?	If one segment only → PMF narrowing

Why blended metrics hide this: If you grow 20% MoM, each month's new users dilute the denominator. Blended retention looks stable while per-cohort retention erodes. By the time blended retention drops, you're 3-6 months behind.

Revenue vs. Logo Retention:

Metric	What It Measures	When They Diverge
Logo retention	% of customers retained	Lose 10 small accounts, keep all large ones → logo retention drops, revenue retention stable
Revenue retention (gross)	% of revenue retained from existing customers	Lose $1K accounts, retain $100K accounts → revenue stable, logo dropping
Revenue retention (net)	Gross + expansion revenue from existing customers	Can exceed 100% if upsells > churn — the holy grail (NRR > 100%)

Critical insight: Logo retention dropping while revenue retention holds is a ticking bomb — you're relying on fewer, larger customers. Concentration risk rises. Always track both.

Output format — Retention Cohort Table:

| Cohort     | Week 0 | Week 1 | Week 2 | Week 4 | Week 8 | Week 12 | Trend |
|------------|--------|--------|--------|--------|--------|---------|-------|
| Jan 2026   | 100%   | 62%    | 48%    | 38%    | 32%    | 30%     | ✅ Flattening |
| Feb 2026   | 100%   | 58%    | 44%    | 34%    | 28%    | 25%     | ⚠️ Degrading |
| Mar 2026   | 100%   | 55%    | 40%    | 30%    | —      | —       | 🔴 Accelerating decline |

Framework 7: Multi-Armed Bandit

A multi-armed bandit (MAB) is an experiment that continuously reallocates traffic toward better-performing variants in real time, rather than pre-committing to a fixed split for a fixed duration. It is not a replacement for A/B testing — it is a different tool for a different goal.

The Core Tradeoff — Exploration vs. Exploitation:

A/B Test	Multi-Armed Bandit
Explores fixed proportion, then exploits winner	Continuously balances exploration and exploitation
Goal: statistical certainty about a specific hypothesis	Goal: minimize regret (maximize cumulative reward across the experiment)
Fixed split (50/50 or equal across variants)	Dynamic allocation — winner gets more traffic over time
Clean ship/no-ship decision	Ongoing optimization — there may be no single "decision"
Formal frequentist statistical guarantees	No fixed p-value guarantee (depends on algorithm and stopping rule)
Best for: irreversible decisions, regulatory contexts	Best for: continuous optimization, many variants, when opportunity cost of the losing variant is high

Experiment Type Decision Extension (add after Framework 4 decision table):

Question	If Yes →
Are you making a one-time ship/no-ship decision?	A/B test
Are you continuously optimizing (no single decision point)?	Multi-Armed Bandit
Do you have 5+ variants to compare simultaneously?	MAB (A/B at this scale is underpowered)
Does regulatory or legal context require formal statistical guarantees?	A/B test only
Is traffic limited and exposure to the losing variant costly?	MAB (reduces regret vs. fixed 50/50)
Do variants affect each other (network/spillover effects)?	Neither — use switchback

MAB Algorithm Selection:

Algorithm	Mechanism	When to Use	Avoid When
Epsilon-greedy	Explore randomly ε% of time; exploit best arm (1−ε)% of time	Simple, interpretable; ε is tunable	You don't want to tune ε; non-stationary rewards
UCB (Upper Confidence Bound)	Choose arm with highest upper confidence bound on reward	No hyperparameter; good theoretical guarantees	Rewards change over time (UCB assumes stationarity)
Thompson Sampling	Sample from Bayesian posterior for each arm; take action with highest sample	Best empirical performance; naturally handles uncertainty; recommended default	When you need classical statistical interpretability

When NOT to Use MAB:

The decision is irreversible (pricing change, permanent feature removal) — you need a formal A/B with declared α
Your metric has delayed feedback (e.g., 90-day retention) — MAB requires fast signal to reallocate
You need to report a p-value to stakeholders or regulators
The experiment will run fewer than ~500 exposures — not enough data for meaningful reallocation

Failure mode to watch: Reporting a MAB "winner" as if it were an A/B result. MAB does not produce a p-value. Saying "the variant won at p=0.03" after a Thompson Sampling run is methodologically incorrect. State the posterior probability that the variant is best (e.g., "95% probability variant B has higher conversion than control").

Framework 8: HEART Framework

Google's HEART framework (Kerry Rodden, 2010) provides structured UX metric design when the North Star metric and decomposition tree capture growth and retention but miss experience quality.

The Five Dimensions:

Dimension	Measures	Signal Type	Example Metrics
Happiness	User attitudes — satisfaction, perceived ease, emotional response	Survey-based (can't be inferred from logs)	CSAT, NPS, SUS (System Usability Scale), "How easy was it?" rating
Engagement	Depth and frequency of interaction	Behavioral (log-based)	DAU/MAU ratio, sessions per user per week, features used per session, actions per session
Adoption	New users or new feature uptake over a period	Behavioral	New user activation rate, % of existing users who tried new feature in 30 days, time-to-first-use
Retention	Whether users return over time	Behavioral	Day-7/Day-30 retention, MAU stability, voluntary churn rate
Task Success	Efficiency and effectiveness completing core tasks	Behavioral + sometimes survey	Task completion rate, time-on-task, error rate, help-seeking rate, abandon rate

HEART vs. NSM Decomposition — When to Use Each:

Goal	Use
Measuring whether the product is growing and delivering value at scale	NSM decomposition tree
Measuring whether the product experience is working for users	HEART
Product redesign or UX overhaul where experience quality is primary	HEART
Feature launch where the question is "did behavior change?"	NSM + L2 metrics
Feature launch where the question is "did users find it useful and easy?"	HEART (Happiness + Task Success)
Tracking that growth isn't degrading experience	Both in parallel

Critical insight: A product can have strong NSM metrics and poor HEART scores simultaneously — when users are retained by switching costs or habit, not satisfaction. HEART Happiness is the leading indicator of eventual voluntary churn that NSM retention misses.

GSM Application to HEART:

For each HEART dimension relevant to your context, apply Goals-Signals-Metrics:

Dimension: Task Success
Goal: Users complete their core workflow without confusion or error
Signal: User completes workflow in one session without triggering help or abandoning midway
Metric: Single-session workflow completion rate (no help trigger, no abandon)

Practical scoping rule: Don't measure all five dimensions for every feature. Choose the 2-3 most relevant to the feature's goals. HEART is a menu, not a mandatory checklist.

Framework 9: Product-Market Fit Measurement

PMF is not binary and it can erode. This framework provides structured methodology for detecting whether you have PMF, where it lives, and whether it's strengthening or weakening — before retention curves tell the story months later.

The Sean Ellis Test:

Survey question: "How would you feel if you could no longer use [product]?"

Very disappointed
Somewhat disappointed
Not disappointed (it isn't that useful)
N/A — I no longer use it

% "Very Disappointed"	PMF Signal	Action
≥ 40%	Strong PMF — safe to scale acquisition	Scale channels; protect the core experience that drives this score
25–39%	Partial PMF — segment exists, but isn't the majority	Find the "very disappointed" segment; make them the ICP; don't scale broadly yet
< 25%	Weak PMF — don't scale	Iterate on core value proposition; rescope the target user

Retention Curve PMF Detection (behavioral — no survey required):

Curve Shape	PMF Signal	Immediate Action
Flattens ≥20% by Day 30 ✅	PMF exists — there is a retained base	Optimize activation to move more users past the drop-off
Keeps declining toward 0% ❌	No PMF — even long-term users eventually leave	Fix core value proposition before any scaling
Flattens but at <10% ⚠️	Narrow PMF — very small retained base	Find who the retained 10% are; they are your real ICP
High initial, steep drop, then flat	Segment PMF — strong for a specific group, weak for others	Disaggregate by segment; the retained group is your starting point

PMF Segmentation Protocol:

When overall PMF is weak, aggregate metrics hide where it's strong. Disaggregate in this order:

By acquisition channel — does one channel produce meaningfully stickier users?
By use case / entry job — do users who arrive for job X retain at 3x the rate of users arriving for job Y?
By company size or customer type — do specific profiles retain while others churn?
By activation behavior — do users who completed a specific action in week 1 retain at 5x the rate of those who didn't?

The segment with the highest retention rate and highest "very disappointed" score is your actual ICP, regardless of who you thought you were building for.

Leading PMF Indicators (useful before retention data matures at 30–90 days):

Indicator	Strong PMF Signal	Weak PMF Signal
Time to value	Users hit value moment in <10 min unprompted	Users require extensive onboarding or hand-holding
Organic referrals	≥20% of new users from unprompted word-of-mouth	All acquisition is paid or driven
Return frequency	Users return unprompted, outside of notification triggers	Users only return when prompted/nudged
Support ticket type	"How do I do more of X?" (capability-seeking)	"Why doesn't X work?" (failure-recovery)
User-built extensions	Users build workarounds, integrations, or uses you didn't design	Users do exactly what you designed, no more
Disappointment proxy	Users express concern when features are removed or changed	Users don't notice feature changes

PMF vs. Retention vs. NSM — the relationship:

NSM measures whether value delivery is scaling
Retention measures whether users come back
PMF measures whether users would be upset if they couldn't come back — the emotional bond, not just the behavior

A product can have high retention (from switching costs) and low PMF (users stay because leaving is painful, not because staying is valuable). These are different states with different strategic implications.

Quarterly PMF Health Check:

Question	If No →
Is the "very disappointed" % stable or rising?	PMF may be eroding — investigate cohort-level NPS trend
Is the retained base (flattening point on retention curve) stable or rising?	Run segmentation protocol — check if a segment is pulling the average down
Is the time-to-value improving?	Onboarding or activation is degrading — check funnel step by step
Are organic referrals holding?	Core value prop may be weakening even if retention is sticky

Computation Requirements

⚠️ STRONGLY RECOMMENDED: For sample size estimation, run:

python3 scripts/sample_size_calculator.py --baseline <rate> --mde <percent> --alpha 0.05 --power 0.80 --daily-traffic <N>

⚠️ STRONGLY RECOMMENDED: For significance testing on completed experiments, run:

python3 scripts/significance_test.py --control-n <N> --control-conv <N> --treatment-n <N> --treatment-conv <N>

⚠️ STRONGLY RECOMMENDED: For retention cohort analysis, run:

python3 scripts/retention_cohort.py --input <csv> --signup-col <col> --activity-col <col>

⚠️ STRONGLY RECOMMENDED: For metric decomposition (numerator vs. denominator attribution), run:

python3 scripts/metric_decomposition.py --before-num <N> --before-den <N> --after-num <N> --after-den <N>

If scripts are unavailable, state "Script not available — estimates below are approximate" and flag which numbers should be verified.

Do NOT estimate sample sizes by reasoning alone when the script can provide precise computation. Do NOT hand-calculate significance when the test script can provide p-values, confidence intervals, and achieved power.

Application Method

Step 0b: Framework Selection

After passing the Context Fitness Check, use this routing table to select the 3-5 load-bearing frameworks. Not all 9 frameworks are needed for every question.

Question Type	Load-Bearing Frameworks	Skip Unless...
New product/feature launch	F1 (NSM + Decomposition), F2 (Leading/Lagging), F3 (Counter-Metrics), F4 (Experiment Design)	F7 (MAB): only if >50K daily users and multiple variants. F8 (HEART): only if UX quality is the primary launch risk. F9 (PMF): only if PMF is unvalidated.
Existing metric feels off / gaming suspected	F3 (Counter-Metrics + Goodhart), F2 (Leading/Lagging), F6 (Retention Cohorts)	F4/F5: skip if no experiment is planned. F1: only if NSM itself is suspect.
A/B experiment design	F4 (Experiment Design), F5 (Statistical Validity), F3 (Guardrails)	F7 (MAB): only for continuous optimization. F1/F2: reference existing hierarchy; don't rebuild it.
PMF assessment	F9 (PMF), F6 (Retention Cohorts), F2 (Leading Indicators)	F4/F5: skip unless designing a PMF-validation experiment. F8: add if satisfaction signal is the question.
Retention diagnosis	F6 (Retention Cohorts), F2 (Leading/Lagging), F9 (PMF), F3 (Counter-Metrics)	F4/F5: skip unless experiment is needed. F1: only if NSM redefinition is on the table.
Metric hierarchy redesign	F1 (NSM + Decomposition), F2 (Leading/Lagging), F3 (Counter-Metrics)	F6: add if retention is a redesign priority. F8: add if UX metrics are absent from current hierarchy.
Experiment results interpretation	F5 (Statistical Validity), F3 (Counter-Metrics/guardrails), F2 (denominator effects)	F4: reference original design only. F6: only if cohort effects are suspected.

Default rule: For a new product/feature launch, start with F1–F4. Add frameworks only if the question specifically requires them.

Quick Version (9 steps for experienced practitioners)

Define value moment and NSM — Identify the instant the user gets value. Score NSM candidates against the 5-criterion rubric.
Build decomposition tree — NSM → L1 → L2 → input metrics with ownership and cadence.
Design leading indicators — For every lagging metric, identify 2-3 leading indicators with temporal lag classification.
Pair counter-metrics — Every primary metric gets a counter-metric and threshold. Complete the pairing template.
Select experiment type — Use the type decision table (A/B, MAB, switchback, quasi-experimental). Complete the protocol for top 2-3 hypotheses.
Apply HEART where experience quality matters — For UX-heavy launches or redesigns, select the 2-3 relevant HEART dimensions and define GSM for each.
Assess PMF signal — Run Ellis test or behavioral PMF check. Segment by channel, use case, and activation behavior to find where PMF is strongest.
Set up cohort tracking — Define cohort construction type, retention windows, degradation thresholds.
Establish review cadence — Quarterly metric health review protocol. Assign owners.

Full Version (detailed steps with quality checkpoints)

Step 1: Define Value Moment and North Star Metric

Articulate the value moment — the specific instant a user receives the core value your product promises.
List 3-5 candidate NSMs that reflect that value moment.
Score each against the 5-criterion NSM rubric (value reflection, leading nature, influenceability, simplicity, non-gameability).
Select the highest-scoring candidate. If tied, pick the one closest to the value moment.
Validate with GSM: Goal → Signal → Metric. Does your NSM map to a clear signal of user behavior?

Quality checkpoint: Can you explain the NSM to a new hire in one sentence? Can you describe what a team would DO if this metric dropped 20%? If either answer is no, iterate.

Step 2: Build the Decomposition Tree

Decompose NSM into 3-5 L1 metrics. Each L1 must be necessary — if any L1 collapses, the NSM suffers.
Decompose each L1 into 2-4 L2 metrics.
For each L2, identify input metrics that engineers can directly move.
Assign ownership: NSM → exec, L1 → PM, L2 → feature team, input → eng lead.
Assign cadence: NSM → monthly, L1 → weekly, L2 → daily, input → per-deploy.
Produce the hierarchy table (see Framework 1 output format).

Quality checkpoint: Trace from each input metric to the NSM. If you can't draw a clear causal chain in 3 steps or fewer, the tree is too deep or has broken branches.

Step 3: Design Leading Indicators

For every L1 and L2 metric that has medium or longer lag time, identify 2-3 leading indicators.
Classify each leading indicator's temporal lag (Immediate / Short / Medium).
Set thresholds for alerts: "If leading indicator crosses X, investigate Y."
Design your activation metric using the Aha Moment Protocol (6 steps).
Produce the leading/lagging indicator table.

Quality checkpoint: If your leading indicators all have the same lag time as your lagging metrics, they're not leading — they're siblings. Leading indicators must be faster-moving.

Step 4: Pair Counter-Metrics

For every primary metric in the hierarchy, complete the Counter-Metric Pairing Template.
For each primary metric, identify the Goodhart variant most likely to apply (Regressional, Extremal, Causal, Adversarial).
Set guardrail thresholds: "Experiment fails if guardrail crosses X."
Document gaming detection patterns to monitor.

Quality checkpoint: If any primary metric lacks a counter-metric, that's a gaming vulnerability. If the counter-metric is just "user satisfaction" for everything, the pairing is lazy — be specific.

Step 5: Design Experiments

List top 2-3 hypotheses to test (from the decomposition tree — which input metrics do you believe will move L2s?).
For each hypothesis, run through the Experiment Type Decision Table to select the right method.
Check the "When NOT to Experiment" checklist.
Complete the A/B Test Design Protocol for each experiment.
Compute sample size and duration using sample_size_calculator.py.
Score each experiment against the Experiment Quality Rubric (target ≥ 5/6).

Quality checkpoint: If any experiment scores below 4/6 on the quality rubric, redesign before launching. If sample size requires more traffic than you have, document why the experiment is not feasible and propose alternatives.

Step 6: Set Up Cohort Tracking

Define primary cohort type (time-based is the default; add behavior-based or channel-based as secondary).
Set retention measurement windows (day 1, day 7, day 14, day 30, day 60, day 90).
Define degradation thresholds: ">2% absolute decline in day-30 retention across consecutive cohorts = urgent."
Determine whether to track logo retention, revenue retention, or both.
Produce the retention cohort table template.

Quality checkpoint: If you're only tracking time-based cohorts, you're missing the most valuable cuts. Add at least one behavior-based cohort (activated vs. not activated) to validate your activation metric.

Step 7: Establish Review Cadence

Set up the quarterly metric health review using the protocol from Framework 3.
Assign owners for each metric review.
Document the decision framework: Keep / Recalibrate / Replace.
Schedule the first review.

Quality checkpoint: If there's no calendar invite for the quarterly review, it won't happen. Metric systems decay without active maintenance.

Metric Intervention Protocol: When the review triggers an intervention, document it using the O→I→R→C→W cascade:

OBSERVATION [evidence tier] → IMPLICATION [mechanism] → RESPONSE [specific action + owner] → CONFIDENCE [H/M/L + key assumption] → WATCH INDICATOR [observable signal]

Example: OBSERVATION: Day-30 retention dropped 3pp in the Feb cohort vs. Jan cohort [T2: own data] → IMPLICATION: Onboarding change shipped Feb 3 likely reduced early activation (activation rate also dropped 4pp in same period) → RESPONSE: Roll back Feb 3 onboarding change and run A/B against original [PM: Lead, Data: Senior DE] → CONFIDENCE: M (key assumption: no other Feb changes affected activation; needs changelog audit) → WATCH INDICATOR: Day-7 activation rate recovers to Jan levels within 2 weeks of rollback.

Mandatory Output Sections

These sections must appear in every output from this skill, regardless of question type. They are quality assurance mechanisms, not optional extras.

Assumption Registry

A standalone table of every load-bearing assumption underpinning the metric framework:

Assumption	Framework Underpinned	Confidence	Evidence	What Would Invalidate

Minimum 3 assumptions. Common load-bearing assumptions for metric frameworks:

"The proposed NSM correlates with long-term retention" — must be validated causally; correlation alone is insufficient
"Industry benchmark retention rates apply to this segment" — may not hold if user profile or use case differs significantly
"The activation metric predicts retention" — correlation ≠ causation until a causal experiment is run (per Aha Moment Protocol Step 6)

Any L-confidence assumption must be [EVIDENCE-LIMITED]-flagged inline.

Why: Metric frameworks built on unexamined assumptions degrade invisibly. The Goodhart spiral (FM-6) typically begins when a load-bearing assumption erodes undetected.

Adversarial Self-Critique

A mandatory final step: "Identify ≥3 genuine weaknesses in this metric design."

Format per weakness:

What assumption is being made?
What evidence would disprove it?
What is the watch indicator?
Is there a scenario where this metric system is catastrophically wrong? (e.g., NSM can be gamed; cohort analysis misses a structural mix shift; activation metric correlation is spurious)

Critical: This is not the same as listing failure modes — failure modes are general patterns. Adversarial self-critique argues against your specific design decisions as forcefully as possible.

Why: Metric systems that haven't been stress-tested before deployment fail at the worst moment — when leadership has committed to the metrics and gaming begins.

Revision Triggers

When should this measurement framework be revisited? Specific, observable conditions:

NSM correlation with long-term retention drops below r = 0.5 in quarterly re-validation
Leading indicator no longer predicts lagging metric at its historical correlation level
A load-bearing assumption in the Assumption Registry is invalidated
Retention curve shape changes (smile curve → frown curve)
New acquisition channel added that materially shifts user mix
Counter-metric threshold crossed for 2+ consecutive review periods
A key competitor changes their metric strategy in a way that redefines relevant benchmarks

Why: Transforms a point-in-time framework into a living system. Without explicit triggers, metric frameworks ossify and diverge from reality.

Quality Gradients

Intern Tier

Single NSM identified but not scored against rubric — picked by intuition
Basic A/B test plan with a primary metric but no sample size calculation
No counter-metrics — every metric is treated as if gaming is impossible
No leading indicators — entire framework is lagging metrics monitored monthly
Retention tracked as a single blended number, not cohorted
No mention of Goodhart's Law, statistical power, or Simpson's paradox
Output: a KPI list, not a measurement framework

Consultant Tier

Full metric hierarchy (NSM + L1 + L2) with ownership and cadence
Leading/lagging pairs identified with temporal lag classification
Counter-metrics designed for top 3 primary metrics with thresholds
Experiment plan includes sample size calculation and guardrails
Basic time-based cohort analysis with retention curves
Statistical concepts applied correctly (significance, power, MDE all addressed)
Missing: Goodhart variant analysis, behavior-based cohorts, gaming detection patterns, quarterly review protocol

Elite Tier

Complete measurement framework: hierarchy + ownership map + temporal lag classification
Every metric paired with a counter-metric and reviewed for Goodhart vulnerability (all 4 variants assessed)
Experiment type selected via decision table — knows when MAB beats A/B, when switchback is required, when quasi-experimental is the only option
MAB algorithm selected with rationale when MAB is chosen; posterior probability reported, not a spurious p-value
HEART applied where experience quality matters — 2-3 dimensions selected with GSM, not all five forced
PMF signal assessed: Ellis test or retention curve shape interpreted; segmentation run to find where PMF is strongest
Statistical validity justified with plain-English reasoning: MDE is a business decision, sample size is computed, confidence intervals reported alongside p-values
Retention methodology includes time-based AND behavior-based cohorts, degradation detection, revenue vs. logo split
Gaming detection patterns documented for each key metric
Quarterly metric health review protocol established with owners and calendar
Produces artifacts a PM cannot produce unaided — metric hierarchies that resist gaming, experiments that validate causally, cohort analysis that detects PMF erosion before blended metrics show it

Failure Modes

FM-1: The Vanity Trap What it looks like: Metrics are impressive-sounding but don't drive decisions. "Total registered users" is 2M. So what? No one knows what to DO with that number. Why it happens: PM picks metrics that make the product look good in reviews, not metrics that surface problems or guide action. Detection: For each metric, ask: "If this moved 20% in either direction, what would we change?" If the answer is "nothing" or "I don't know," it's a vanity metric. Correction: Replace with the closest actionable metric. Registered users → weekly active users who complete a core action. Total downloads → day-7 retention rate.

FM-2: The Proxy Divergence What it looks like: The metric you're optimizing was once a good proxy for the outcome you cared about, but over time they've decoupled. You're winning the proxy war and losing the real one. Why it happens: All metrics are proxies. Proxies diverge as the product, market, or user behavior changes. Without periodic re-validation, divergence is invisible. Detection: Quarterly: correlate proxy metric with the true outcome. If r < 0.5, the proxy has diverged. Example: "signup rate" no longer predicts "active user at day 30" because acquisition channels shifted to low-intent traffic. Correction: Run the quarterly metric health review. Re-validate proxy-outcome correlation. If diverged: recalibrate the proxy definition or replace it entirely.

FM-3: Simpson's Paradox What it looks like: The aggregate result shows improvement, but when you segment by any meaningful dimension (platform, user type, geography), every segment got worse. Or vice versa — aggregate looks flat while one segment is quietly collapsing. Why it happens: Segment mix shifted. You added a large new segment with different baseline rates, which shifted the aggregate without changing underlying performance. Detection: Always segment experiment results by key dimensions before declaring a winner. If aggregate and segments disagree, investigate the mix shift. Correction: Report segment-level results alongside aggregate. Identify the mix shift. Decide based on within-segment effects, not aggregate.

FM-4: The Peeking Problem What it looks like: PM checks experiment results daily. On day 3, p=0.04. Ships the feature. But with a proper sample size, the test needed 3 more weeks. The "significant" result was noise amplified by early peeking. Why it happens: Impatience + misunderstanding of how significance works. With small samples, estimates are noisy. The first few days of an experiment are like the first few flips of a coin — wildly unreliable. Detection: Did the experiment run for its pre-committed duration with its pre-committed sample size? If stopped early without a sequential testing framework, results are suspect. Correction: Pre-commit to N and duration. Lock the dashboard for the first week if needed. If early stopping is genuinely needed, use sequential testing (which adjusts significance thresholds for continuous monitoring).

FM-5: Survivorship Bias What it looks like: "Our active users love the product! Engagement is through the roof!" Of course it is — you're only measuring the users who stayed. The 60% who churned would tell a different story. Why it happens: Churned users disappear from dashboards. Engagement metrics automatically exclude them. The healthier your remaining base looks, the more users you may have lost. Detection: Always include churned users in the denominator. Study exit points — where did users drop off? Run churned-user surveys or analyze their last sessions. Correction: Track "full-funnel" metrics that include all users from acquisition through churn. Build an exit-point analysis into your retention methodology. Don't celebrate engagement among survivors without knowing the survival rate.

FM-6: The Goodhart Spiral What it looks like: Team optimizes metric → metric gets gamed → team picks new metric → team optimizes new metric → new metric gets gamed → repeat. Each cycle erodes trust in measurement and burns team energy. Why it happens: Metrics are treated as permanent when they should be treated as instruments that need maintenance. No counter-metrics, no health reviews, no rotation plan. Detection: Count metric changes in the past year. If >3 metric changes without a documented health review process, you're in a Goodhart spiral. Correction: Design counter-metrics from day one. Establish quarterly health reviews. Accept that metrics have a lifespan — planned rotation is healthy, reactive replacement is a spiral.

FM-7: The Lagging Indicator Trap What it looks like: Every metric in the framework is lagging — revenue, churn, NPS, retention. The team reviews monthly. By the time they see a problem, it started 3 months ago. Why it happens: Lagging metrics are easier to define and harder to argue with. Leading indicators require predictive thinking, correlation analysis, and causal validation — harder work. Detection: Classify every metric by temporal lag. If nothing is in the Immediate or Short category, the framework has no early warning system. Correction: For every lagging metric, identify 2-3 leading indicators using the Leading Indicator Discovery Protocol. Set monitoring cadences that match lag categories.

FM-8: The Denominator Shift What it looks like: Conversion rate improved from 3% to 4%. Celebration. But visitors dropped from 100K to 50K — the low-intent visitors left, concentrating high-intent visitors. The numerator didn't change; the denominator shrank. Why it happens: Rates look cleaner than absolutes. PMs report rates without decomposing into numerator and denominator movements. Detection: For every rate metric, track the numerator and denominator separately. Use metric_decomposition.py to attribute rate changes. If denominator changed >10%, the rate change may be a mix effect, not a real improvement. Correction: Always report rates alongside absolutes. Decompose rate changes into numerator effect vs. denominator effect. Use the decomposition script for automated attribution.

FM-9: The Multiple Testing Trap What it looks like: Team runs an A/B test. Primary metric is flat. But they test 10 secondary metrics and find one that's significant at p=0.03. "Feature improves metric #7!" In reality, testing 10 metrics at α=0.05 gives a ~40% chance of at least one false positive. Why it happens: Desire to find a win. "We invested 4 weeks in this test — something must have worked." Detection: Count the number of metrics tested. If more than one metric is used for a ship decision without correction, the multiple testing trap is active. Correction: Pre-declare one primary metric. Apply Bonferroni correction to secondary metrics (α/number of tests). Treat secondary results as exploratory hypotheses for future experiments, not evidence for shipping.

FM-10: The Instrumentation Gap What it looks like: Beautiful metric framework on paper. But when it's time to measure, the data doesn't exist, isn't clean, arrives 3 days late, or has no historical baseline. Why it happens: Metric design happens in a planning meeting. Instrumentation happens in the codebase. No one bridges the gap until launch day. Detection: Before committing to any metric, answer 4 questions: (1) Does the event/data exist in our tracking? (2) Is it reliably logged (no gaps, no duplicates)? (3) Is it timely (available within the monitoring cadence)? (4) Do we have ≥30 days of historical baseline? Correction: Add "instrumentation feasibility" as a step in the metric design process. If any of the 4 questions is "no," either invest in instrumentation first or choose a measurable proxy.

FM-11: The Expert-Only Document What it looks like: Output is dense with statistical terminology (Bonferroni correction, Thompson Sampling, regressional Goodhart variant), notation (T1-T6, H/M/L, O→I→R→C→W), and framework names a non-data PM cannot parse. The PM lead who commissioned the framework cannot explain it to their VP. Why it happens: The skill optimizes for analytical rigor, not reader comprehension. The author knows all the notation; the reader doesn't. Detection: Show the output to the PM who will present it. If they can't explain any section without looking up a term, the document fails the reader test. Correction: The output template mandates a "How to Read This Document" section and a Notation Key. The Executive Summary uses zero jargon. Every framework reference gets a one-line contextual explanation the first time it appears.

FM-12: Designing Without Data Access What it looks like: A complete measurement framework with targets, thresholds, and retention benchmarks — but all targets are industry averages (T4) because the analyst had no access to internal data. The framework looks validated but is actually a hypothesis document. Why it happens: The skill produces the same output structure regardless of data access. A framework designed with internal data and one designed from SaaS benchmarks look identical. Detection: Check evidence tiers on targets. If >50% of targets are T4-T6, the framework is a starting template, not a validated system. Correction: Run the Context Fitness Check (Step 0). If internal data is unavailable, flag prominently and design the instrumentation to collect validation data as a first-phase deliverable.

What's Next

← This skill works best after: Spec Writing (metrics validate whether the specified feature achieves its goals) or Problem Framing (a well-framed problem defines what outcomes need measuring) → This skill's output feeds into: Discovery & Research (evidence gaps identified by the measurement framework drive research priorities), Competitive & Market Analysis (competitor metric comparison — how do our retention curves compare?) ⊕ Start here if: You have a product or feature and need to define success metrics, design experiments, or build a retention tracking system from scratch 💡 For a full product cycle: Problem Framing → Discovery & Research → Competitive Analysis → Spec Writing → [THIS SKILL] → ongoing measurement loop

Chain interface:

Receives: Product specification with defined user value proposition (from Spec Writing), or problem definition with target outcomes (from Problem Framing), or raw context (feature launch needing metrics, experiment design request, retention tracking setup)
Produces: Complete Measurement Framework — metric hierarchy with ownership, leading/lagging indicator pairs, counter-metric design, experiment plans with statistical validity, retention cohort methodology, quarterly review protocol
Handoff artifact: The Measurement Framework's "Experiment Plan" section produces results that feed back into Discovery & Research (what did we learn?) and Spec Writing (iterate the spec based on results). The "Metric Hierarchy" informs Competitive Analysis (benchmark our metrics against competitors).

Appendix: Quick-Reference Checklist

Use this to verify output completeness:

Format Rules compliance:

Step 0 routing table applied — only load-bearing frameworks for this question type selected
H/M/L confidence used on all metric design conclusions (no weasel words)
Per-cell evidence tier annotation applied in all comparison tables (T2/T4/T6 inline)
Staleness flags applied to any benchmark data or normative threshold older than 6 months
EVIDENCE-LIMITED flags applied to any recommendation based only on T6 (theoretical) reasoning
Metric intervention recommendations use O→I→R→C→W cascade (not bare assertions)
Contradictions between frameworks surfaced explicitly (e.g., unvalidated activation metric despite strong NSM alignment)

Mandatory output sections:

Assumption Registry completed — minimum 3 load-bearing assumptions with confidence, evidence, and invalidation conditions
L-confidence assumptions flagged [EVIDENCE-LIMITED] inline in Assumption Registry
Adversarial Self-Critique completed — ≥3 genuine weaknesses in this specific metric design identified with watch indicators
Revision Triggers defined — specific, observable conditions that would require framework update

Worked Example

Input

"We're launching a real-time co-editing feature for Vanta Analytics: multiple users can edit the same dashboard simultaneously. This is our biggest feature launch this quarter. Current state: 22K DAU, W4 retention 52%, avg session 8 min. We need a measurement framework before shipping. We have one dedicated data engineer for 2 weeks."

Output (abbreviated — demonstrates key formats; a full measurement framework covers all 7 steps)

NSM Candidate Evaluation:

We need a metric that reflects the value of co-editing — collaboration value, not just usage.

Candidate NSM	Value Reflection	Leading Nature	Influenceability	Simplicity	Non-Gameability	Score
Weekly collaborative sessions (≥2 users, same dashboard, <10 min gap)	✅	✅	✅	✅	✅	5/5
DAU	❌ DAU rises from solo use	✅	✅	✅	❌ Spam sessions	3/5
Dashboards created per week	❌ Can rise while collaboration fails	❌ Lagging	✅	✅	❌ Trivial dashboards	2/5

Selected NSM: Weekly collaborative sessions. "One sentence for new hire: the number of times two or more users actively edited the same dashboard in the same week."

Metric Hierarchy (top 2 levels):

Level	Metric	Owner	Cadence	Target (W8 post-launch)	Counter-Metric
NSM	Weekly collaborative sessions	CPO	Monthly	2,500	Collaboration quality score ≥4.0/5
L1	Co-edit activation rate (users who co-edit within 7 days of feature access)	PM, Growth	Weekly	18%	Support tickets from co-edit confusion
L1	W4 collaborative retention (users who co-edited in W1 still co-editing W4)	PM, Core	Weekly	55%	Session depth ≥3 collaborative actions
L1	Invite-to-collaborate rate (% of sessions with ≥1 invitation sent)	PM, Virality	Weekly	22%	Invite-spam reports
L2	Real-time presence awareness adoption	PM, Core	Daily	70% of co-edit sessions	Conflict resolution errors < 0.5%
L2	Onboarding-to-first-co-edit conversion	PM, Onboarding	Daily	35%	Time-to-first-co-edit ≤48 hours

Leading Indicator Design:

Lagging Metric	Leading Indicator	Lag	Correlation (est.)	Causal?	Alert Threshold
W4 collaborative retention	Co-edit session depth ≥3 actions in W1	3 weeks	r ≈ 0.70 (hypothesis — needs validation)	Not yet proven — must experiment	<40% of new co-editors = investigate
Monthly churn	Login frequency drop + no co-edit sessions	2 weeks	r ≈ 0.65 (from existing retention data)	Yes (validated pre-launch)	<50% of baseline login freq = alert

⚠️ Critical note: The co-edit depth → retention correlation is an estimate. Per Step 3 of the Aha Moment Protocol, we must run a causal experiment (prompt co-edit depth vs. don't prompt) before declaring this the activation metric. Treat it as a leading hypothesis, not a validated predictor, until that experiment runs.

Counter-Metric Pairing:

Primary Metric	Goodhart Variant	What Goes Wrong	Counter-Metric	Threshold
Co-edit activation rate	Causal	Force users into co-edit flows they don't want → activation up, satisfaction down	Day-7 CSAT from co-edit users	Must stay ≥ 3.8/5
Weekly collaborative sessions	Adversarial	Team opens the same dashboard "together" without real collaboration to hit metric	Actions-per-session in co-edit (≥3 distinct edits from ≥2 users)	If sessions rise but actions-per-session falls → gaming active
Invite-to-collaborate rate	Regressional	Invite feature is easy to trigger accidentally	Accepted invite rate	Must stay ≥ 45%

Experiment Plan — Co-Edit Onboarding Flow:

Field	Value
Hypothesis	If we show a "invite a teammate" prompt at first dashboard open, co-edit activation rate will increase by ≥5pp within 7 days
Primary metric	Co-edit activation rate (day 7)
Secondary metrics	Invite accepted rate, dashboard edits per session (exploratory, Bonferroni-corrected at α=0.01)
Guardrail metrics	Day-7 CSAT ≥ 3.8/5; unsubscribe rate ≤ 2x baseline
MDE	5pp (business decision: <5pp lift not worth maintaining the prompt flow)
α	0.05
Sample size	Run `python3 scripts/sample_size_calculator.py --baseline 0.13 --mde 0.05 --alpha 0.05 --power 0.80 --daily-traffic 1467`
Duration	14 days minimum (cover two full weekday/weekend cycles; B2B bias)
Randomization unit	User (not session — same user must get consistent experience)
Exclusions	Users who already co-edited before launch; mobile-only users (feature not yet on mobile)
Decision rule	Ship if primary ≥ 5pp AND both guardrails hold. Do NOT ship if CSAT guardrail fails even if primary wins.

Experiment Quality Score: 6/6 (pre-registered hypothesis, single primary metric, guardrails declared, duration committed, sample size to be computed, segmentation planned by plan tier and signup cohort).

Why This Works

The NSM was selected using the 5-criterion rubric (Framework 1), not by intuition — this prevents the most common failure mode (FM-1: Vanity Trap). The leading indicator carries an explicit causal uncertainty warning (Framework 2's Aha Moment Protocol Step 6), preventing teams from optimizing a correlation they haven't validated. The counter-metric pairing identifies the Goodhart variant for each metric (Framework 3), making the gaming risk specific and detectable. The experiment plan scores 6/6 on the quality rubric, satisfying the Elite Tier characteristic: "experiments that validate causally."

References

Framework Sources:

Sean Ellis & Morgan Brown, Hacking Growth (2017) — North Star Metric methodology
Amplitude, North Star Playbook (amplitude.com) — NSM framework and metric trees
Google, GSM (Goals-Signals-Metrics) — Kerry Rodden, HEART framework extension
Reforge, Growth Series — Metric decomposition trees, activation metric design
Manheim & Garrabrant, "Categorizing Variants of Goodhart's Law" (2018) — 4 Goodhart variants
Ron Kohavi, Diane Tang, Ya Xu, Trustworthy Online Controlled Experiments (2020) — A/B testing, sequential testing, guardrail metrics
Optimizely & Eppo documentation — Sequential testing, multiple testing correction
Casey Winters, Retention curves and cohort analysis — Reforge retention methodology

Computation Scripts:

scripts/sample_size_calculator.py — Sample size and duration estimation
scripts/significance_test.py — p-value, confidence interval, effect size, achieved power
scripts/retention_cohort.py — Cohort table, retention curves, degradation flags
scripts/metric_decomposition.py — Rate change attribution (numerator vs. denominator)

Related Skills in PM Skills Arsenal:

Competitive & Market Analysis — parallel (competitor metric benchmarking)
Problem Framing — upstream (defines what outcomes to measure)
Spec Writing — upstream (defines what to build; this skill defines how to measure it)
Discovery & Research — downstream (evidence gaps drive research) and upstream (research findings inform metric selection)

Created: 2026-02-18 | PM Skills Arsenal v1.0 | BLD-003 Quality tier: Elite (≥650 lines, 6 encoded frameworks, quality gradients, 12 failure modes, reader navigation, context gate) License: MIT