Measurement incrementality

Measurement Incrementality

Determine whether marketing activity caused business impact, rather than merely receiving attribution credit. Separate delivery, engagement, attribution, correlation, and true incrementality across all channels.

How This Skill Orchestrates

1. Script execution (scripts/baseline_gap_calculator.py): Estimates baseline (what would have happened without marketing) using 4 methods: simple average, trend-adjusted, seasonal (YoY), and control-based. Calculates incremental gap, lift %, iROAS, iCPA. Identifies confounders
2. Script execution (scripts/test_control_validator.py): Power analysis for experiment design — validates sample size, calculates minimum detectable effect, estimates required duration, and revenue-at-risk in the control group
3. Script execution (scripts/incrementality_claim_checker.py): Classifies claims across the 5-level evidence hierarchy (delivery → engagement → attribution → correlation → incrementality). Flags overclaiming. Provides safe executive language per level
4. Reference lookup (references/experiment_design_guide.md): 8 measurement methods with selection criteria, experiment templates (conversion lift, holdout, geo-lift, MMM), confounder library, benchmark search protocol, incremental lift benchmarks by channel, the "attribution tax" table
5. Reference lookup (references/executive_claims_policy.md): Claim language templates by evidence level, CFO test, report language templates (strong/moderate/weak), what to NEVER claim
6. Web search (Claude — MANDATORY): Search for "[PLATFORM] conversion lift [INDUSTRY]", "marketing incrementality benchmarks [YEAR]", "Kantar [CATEGORY] [COUNTRY]" before interpreting results
7. Upstream context (Claude): Pull media allocation from media-routing-planner, campaign data from weekly-control-tower, margin data from margin-simulation
8. Visual output (Visualizer): Evidence hierarchy dashboard + baseline vs observed chart + experiment design card

Models used (all from Measurement layer):

Model	Purpose	When
Baseline estimation (trend/seasonal projection)	Counterfactual: what would have happened without marketing	Always — the foundation of incrementality
Power analysis (z-test for proportions)	Validate experiment design before running it	Before any holdout/geo-lift/conversion lift test
Claim classification (rule-based hierarchy)	Governance: prevent overclaiming	When interpreting any marketing results
PyMC-Marketing MMM	Full Bayesian portfolio measurement	When 2+ years of data available → use mmm-modeling skill

No deep learning. Incrementality measurement is causal inference — statistics, not pattern recognition.

Quick Reference

Resource	Purpose	Usage
scripts/baseline_gap_calculator.py	Baseline estimation (4 methods), incremental gap, lift %, iROAS, iCPA, confounder identification, confidence assessment	python baseline_gap_calculator.py --input config.json --output baseline.json
scripts/test_control_validator.py	Power analysis for experiment design — sample size, MDE, duration, revenue-at-risk	python test_control_validator.py --input design.json --output power.json
scripts/incrementality_claim_checker.py	Classifies claims across 5-level evidence hierarchy, flags overclaiming, provides safe executive language	python incrementality_claim_checker.py --input claims.json --output checked.json
references/experiment_design_guide.md	8 measurement methods, experiment templates, confounder library, channel lift benchmarks, attribution tax table	Read for method selection and test design
references/executive_claims_policy.md	Claim language by evidence level, CFO test, report templates, what to never claim	Read before writing any performance report

Process

Step 1 — Define the Business Outcome

What are we trying to prove? "Did media drive incremental sales?" is different from "Did we deliver the campaign?"

Step 2 — Classify Available Evidence

Run the claim checker against current performance data. What level are we at — delivery, attribution, correlation, or incrementality?

Step 3 — Identify Confounders

List everything else that changed during the measurement window: seasonality, price, promotions, distribution, competitor activity, organic trend. Each confounder weakens the causal claim.

Step 4 — Calculate Baseline

Run the baseline gap calculator. Use the best method available: control-based > seasonal > trend > simple. The baseline IS the counterfactual.

Step 5 — Estimate Incrementality

Incremental = Observed - Baseline. Calculate lift %, iROAS (incremental revenue / spend), iCPA (spend / incremental conversions).

Step 6 — Design the Next Experiment

If current evidence is below incrementality level, design a test to upgrade. Run the power validator to ensure the test is large enough to detect the expected lift.

Step 7 — Validate Claims Against Evidence

Before any claim goes to leadership, run it through the claim checker. Match claim strength to evidence strength. Apply the CFO test.

Step 8 — Translate to Financial Impact

Convert incrementality into dollars: incremental revenue, incremental profit (after COGS), incremental LTV. This is what the CFO cares about.

Step 9 — State Confidence and Limitations

Every measurement output must include confidence level (high/medium/low) and explicit limitations. Overclaiming destroys credibility faster than underperforming.

Step 10 — Recommend Next Measurement Action

What should be tested next? Which channel needs a holdout? When should MMM be updated? What data gaps need closing?

Output Format

Produce in TWO forms: inline visual dashboard (Visualizer) and structured markdown report.

Visual Dashboard (Primary)

Render as inline HTML widget:

• Evidence hierarchy bar — 5-level visual showing where current evidence sits (delivery → incrementality), with the current level highlighted
• Baseline vs Observed chart — two bars showing baseline and observed, with the incremental gap visually highlighted
• Metrics row — 4 cards: Incremental value, Lift %, iROAS, Confidence level
• Confounder flags — each confounder as a badge with adjusted/unadjusted status
• Claim assessment — each claim with evidence level badge (color-coded)
• Experiment recommendation — if below incrementality level, show recommended test design with power analysis
• sendPrompt() buttons: "Run MMM for full portfolio measurement" → mmm-modeling, "Present measurement findings to leadership" → executive-growth-memo

Markdown Report (Secondary)

## 📐 MEASUREMENT & INCREMENTALITY — [Campaign/Channel] — [Client]

### Measurement verdict
[Current evidence level: delivery / attribution / correlation / incrementality]
[Confidence: high / medium / low]

### Evidence classification
| Claim | Metric | Evidence level | What it proves | Overclaim risk |

### Baseline analysis
| Metric | Observed | Baseline | Incremental | Lift % | Method | Confidence |

### iROAS / iCPA
| Metric | Value | vs. Platform-reported | Gap |
| iROAS | [X]x | Platform ROAS: [Y]x | [Z]x overclaimed |
| iCPA | $[X] | Platform CPA: $[Y] | $[Z] undercounted |

### Confounders
| Confounder | Present? | Direction | Adjusted? | Impact estimate |

### Experiment design (if needed)
| Parameter | Value |
| Test type | [holdout / geo-lift / conversion lift] |
| Test group | [X] ([Y]%) |
| Control group | [X] ([Y]%) |
| Power | [X]% |
| MDE | [X]% |
| Duration | [X] days |
| Revenue at risk | $[X] |

### Executive claims policy
| Level | Safe claim language |
[Language templates matching current evidence level]

### Recommendations
1. [Next measurement action]
2. [Data gap to close]
3. [Test to design]

Examples

Example 1 — "Platform says 4x ROAS": User: "Meta reports 4.2x ROAS for our Q3 campaign. Is this real?" → Claim checker: attribution level only. Attribution tax: Meta typically 2-3× over-reports. Estimated iROAS: 1.5-2.5×. Recommendation: run Meta Conversion Lift test to validate. Safe claim: "Platforms attribute $X in conversions; incremental validation pending."

Example 2 — Post-campaign measurement: User: "Sales increased 18% during the campaign vs. prior period. Can we claim the campaign drove this?" → Baseline calculator: pre-period average projected forward = 8% organic growth. Incremental = 10% (18% - 8%). But: promotion was active (confounder). Adjusted estimate: 5-10% incremental. Confidence: low-medium.

Example 3 — Experiment design: User: "We want to prove TikTok drives incremental sales. How do we set up a test?" → Power validator: need 50K in test, 5K in control (10% holdout). Baseline CVR 2.5%, expected lift 15%. Power at 82% → sufficient. Duration: 28 days. Revenue at risk: $12K in holdout group. Recommendation: proceed with Meta Conversion Lift as backup.

Skill Chaining

Direction	Skill	Connection
Upstream	media-routing-planner	Channel allocation → defines what to measure
Upstream	weekly-control-tower	Performance data → evidence for claim checking
Upstream	campaign-launch-qa	Tracking setup validated before measurement
Downstream	mmm-modeling	Full Bayesian MMM when 2+ years data available
Downstream	executive-growth-memo	Measurement findings → C-level narrative
Downstream	qbr-generator	Incrementality results → quarterly narrative
Downstream	media-routing-planner	Incrementality data → re-optimize allocation