Product analytics experiments
Skill charlieviettq/awesome-agent-skill/.claude/skills/product-analytics-experiments
Curated skill pack for LLM agents in engineer and science workflow (Cursor & Claude ready).
npx -y skills add charlieviettq/awesome-agent-skill --skill product-analytics-experimentsAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 22 stars22 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Product analytics and experimentation—event design, funnel metrics, tracking plans, and A/B test setup with statistical and operational gates. Use when defining metrics, launch experiments, or reviewing growth/DS product work.
SKILL.md
2.0 KB, as published. Nobody here has run it
Product analytics and experiments
Event and tracking design
- One event = one user action with stable name and versioned schema.
- Required context:
user_idor anonymous id,timestamp,session_id, product surface. - Document in a tracking plan: event, properties, trigger, owner, PII classification.
Core metrics
| Type | Examples |
|---|---|
| Acquisition | Signups, activation rate |
| Engagement | DAU/WAU, feature adoption |
| Conversion | Funnel step rates |
| Retention | D1/D7/D30 cohort retention |
| Quality | Error rate, task success |
Define denominator explicitly (eligible users, not all traffic).
Funnel analysis
- Define steps and entry criteria.
- Check identity stitching and time window.
- Segment by platform, cohort, campaign (avoid Simpson's paradox surprises).
A/B test gates (before launch)
- Hypothesis and primary metric (one primary)
- Guardrail metrics (latency, errors, revenue risk)
- Randomization unit correct (user vs session)
- Sample size / MDE estimated; duration planned
- No peeking-driven early stop without sequential plan
- Feature flags and exposure logging verified in staging
Analysis
- Intent-to-treat as default; document exclusions.
- Report: point estimate, CI, practical significance (not only p-value).
- Slice only with pre-registration or clear exploration label.
Anti-patterns
- Changing primary metric after results visible.
- Multiple simultaneous experiments on same surface without interaction analysis.
- Tracking PII without legal/product review.
Output
Experiment brief: hypothesis, design, metrics, ship/kill criteria, results template.