Changelog detective

Changelog Detective

Find what actually changed after a deploy — not what the release notes say, but what the session data shows. Catch undocumented changes, side effects, and surprises.

When to Use

• "What actually changed after the deploy?"
• "Something feels different since Tuesday — what changed?"
• "Check for undocumented changes in the last release"
• "Did the deploy introduce any new user behaviors?"
• "What new errors appeared this week that weren't here last week?"

Mental Model

Release notes tell you what was intended to change. Session data tells you what actually changed. These often differ. A "minor CSS fix" might accidentally break form validation. A "performance improvement" might introduce a new race condition. This skill hunts for the gaps between intent and reality.

Workflow

Step 1: Define the comparison window

Two time periods:

• Before: 24-72 hours before the deploy/change date
• After: 24-72 hours after (but not including the deploy window itself — things are chaotic during rollout)

Ask: "I'll compare 48 hours before and after the deploy (excluding the deploy hour). That OK?"

Step 2: Before/After diff — Errors

What errors existed before vs after?

fullstory:build_metric(query="errors by type", output_type="top_n")
fullstory:compute_metric(metric_id, time_range=before)
fullstory:compute_metric(metric_id, time_range=after)

Look for:

• New errors: Error types that appear in After but not Before → these are deploy regressions
• Resolved errors: Error types in Before but not After → these are fixed (celebrate these)
• Changed errors: Same error type, different frequency → possible impact change

Step 3: Before/After diff — Frustrations

fullstory:get_opportunities(time_range=before)
fullstory:get_opportunities(time_range=after)
→ compare the lists

Look for:

• New frustration signals: Rage clicks or dead clicks on elements that weren't problematic before
• Resolved frustrations: Frustrations that disappeared (fix worked!)
• Changed patterns: Same frustration, different page or device distribution

Step 4: Before/After diff — User behavior

Did user flows change?

fullstory:build_metric(query="page views by page", output_type="top_n")
→ compare before/after: any pages suddenly getting more/less traffic?

fullstory:build_metric(query="conversion rate", output_type="single_number")
→ compare before/after: did conversion change?

Look for:

• Traffic shifts: Pages with significantly different traffic share
• Flow changes: New common page sequences (users navigating differently)
• Time changes: Sessions getting longer or shorter
• Conversion changes: Key funnels performing differently

Step 5: Before/After diff — Device/Browser

Did the change affect specific platforms?

fullstory:build_metric(query="sessions by device type", output_type="top_n")
→ compare before/after: did the device mix change?

fullstory:build_metric(query="errors on Safari", output_type="single_number")
→ compare before/after: any browser-specific regressions?

Step 6: Assemble the changelog

## Actual Changelog — v2.3.1 Deploy (Aug 4 15:00 UTC)

### What the release notes said:
- "Fixed promo code validation"
- "Updated checkout button styling"  
- "Performance improvements"

### What actually changed (from session data):

✅ **Promo code fixed**: Rage clicks on "Apply Promo" dropped from 412/week to 24/week (94% reduction)
✅ **Button styling**: No new dead clicks on checkout elements — visual change was clean
🟡 **Performance**: Page load time unchanged (was 2.1s, now 2.1s). "Improvements" may not be user-visible.

### Undocumented changes:

🔴 **New error introduced**: `TypeError: Cannot read property 'validate' of undefined` on /checkout — 34 users in 48 hours. Did not exist before deploy. Stack trace points to the new promo validation code.
🔴 **Mobile regression**: Checkout form fields shifted 20px down on iOS Safari. Not visible on desktop. Affects ~15% of mobile checkout users.
🟡 **New user behavior**: 8% of users now click "Apply Promo" before entering a code (dead click). The new button styling may make it look like it does something before the field is filled.

### Net assessment:
The deploy fixed the #1 frustration (promo code rage clicks) but introduced a new error and a mobile layout regression. Worth fixing both, but the fix was net positive — 412 fewer frustrated users per week vs 34 users seeing a new error.

Changelog Diff Checklist

Dimension	Before	After	Change
Top 5 errors	[list]	[list]	New? Resolved? Changed frequency?
Top frustrations	[list]	[list]	New signals? Resolved? Shifted?
Conversion rate	X%	Y%	Δ = Y-X
Bounce rate	X%	Y%	Δ
Session duration	X min	Y min	Δ
Device mix	D/M/T%	D/M/T%	Shift?
Browser errors	Per browser	Per browser	New browser-specific issues?
Page traffic share	Top pages	Top pages	Any pages gaining/losing >20%?

Guidelines

• The deploy window (first hour after deploy) is noisy — caches warming, users getting new bundles, partial rollouts. Exclude it from before/after comparison. Start "after" at +1 hour.
• New errors are the strongest signal. An error that didn't exist before the deploy is almost certainly deploy-related. Prioritize these.
• Behavioral changes can be positive. If users are suddenly using a feature more, that might be intentional (better UX) rather than a bug. Investigate before flagging.
• Some changes are coincidental, not causal. A traffic spike might be a marketing campaign, not the deploy. Cross-reference with campaign-tracker before attributing to the deploy.
• If you find undocumented changes that look like bugs, flag them immediately. The team might not know. "Heads up: the deploy introduced a TypeError on /checkout that wasn't in the release notes. 34 users affected in 48 hours."
• For major deploys (50+ files changed), run this skill. For minor deploys (hotfix, config change), deploy-radar is usually sufficient.