Market basket analysis

Market Basket Analysis

Three capabilities in one skill: association rule mining → QAP store/shelf layout optimization → shopping journey segmentation. Always outputs dashboard first.

---

Plain Language: What This Does

ASSOCIATION RULES:   "Customers who buy pasta + wine also buy garlic (84% of the time)"
                     → Cross-sell widget on product page, bundle promotion, email trigger

STORE LAYOUT (QAP):  "Bakery and Drinks have lift=1.3 → place them adjacent"
                     → Physical store planogram, digital shelf ordering, category management

JOURNEY SEGMENTS:    "30% of baskets follow: bread→milk→eggs (quick trip)"
                     → Different CRM/loyalty messaging per journey type

Core insight (Katsov §6.9.1): Lift > 1 between two categories = they appear together more than chance predicts. QAP maximizes the sum of (lift × proximity) across all category pairs.

---

Core Equations (Katsov §5.11.2 + §6.9.1)

Association Rule Metrics (eq. 5.155–5.158)

# Support — how often does this pattern appear?  (eq. 5.155–5.156)
support(X) = |{t ∈ T : X ⊆ t}| / |T|
support(X→Y) = support(X ∪ Y)

# Confidence — given X, how likely is Y?  (eq. 5.157)
confidence(X→Y) = support(X ∪ Y) / support(X) = P(Y|X)

# Lift — is this better than chance?  (eq. 6.118)
lift(X→Y) = support(X ∪ Y) / (support(X) × support(Y))
           = confidence(X→Y) / support(Y)
# lift > 1 → positive affinity   lift < 1 → negative   lift = 1 → independent

# Expected revenue from rule (eq. 5.158):
revenue(X→Y) = support(X→Y) × Σ price(i) for i in Y

QAP Store Layout (eq. 6.119–6.121)

# Lift matrix (eq. 6.119): L[i,j] = λ(category_i, category_j)
# Distance matrix (eq. 6.120): D[i,j] = distance between location i and j
#   (binary: 1 if adjacent, 0 otherwise — or Euclidean)

# Objective — maximize co-purchase value × proximity (eq. 6.121):
max_π  Σ_i Σ_j  λ[i,j] × D[π(i), π(j)]

# π(x) = y means: category x assigned to location y
# NP-hard → use brute force (n≤8), simulated annealing, or OR-Tools

Katsov Example 6.8 — 6 Categories × 2×3 Grid

Lift matrix L (6.122):  highest affinities → Frozen↔Drinks (1.5), Bakery↔Dairy (1.3)
Distance matrix D (6.123): binary adjacency in 2×3 grid
Optimal layout: place Frozen next to Drinks, Bakery near Dairy
Full brute force: 6! = 720 permutations evaluated

---

Algorithm Selection

Task	Method	Script
Generate rules from transactions	Association rules (Katsov §5.11.2)	association_rules.py
Fast mining, large catalogs	FP-Growth	fp_growth.py
Classic textbook approach	Apriori	apriori_optimized.py
Optimize store/shelf category order	QAP (Katsov §6.9.1)	store_layout_qap.py
Segment basket journey types	Hierarchical clustering	journey_segmentation.py

When to use Apriori vs FP-Growth:

• Apriori: ≤ 50K transactions, interpretable, slower
• FP-Growth: > 50K transactions, memory-efficient, 10–100× faster
• Both produce identical rules — FP-Growth preferred at scale

---

Workflow

Step 1 — Mine Association Rules

python scripts/fp_growth.py \
    --transactions data/transactions.csv \
    --min-support 0.01 \
    --min-confidence 0.2 \
    --min-lift 1.5 \
    --output results/rules.json

Step 2 — Build Lift Matrix (for QAP)

python scripts/association_rules.py \
    --transactions data/transactions.csv \
    --mode lift-matrix \
    --categories data/category_map.csv \
    --output results/lift_matrix.json

Step 3 — QAP Layout Optimization

python scripts/store_layout_qap.py \
    --lift-matrix results/lift_matrix.json \
    --floor-plan "2x3" \
    --method annealing \
    --output results/optimal_layout.json

Step 4 — Journey Segmentation (optional)

python scripts/journey_segmentation.py \
    --transactions data/transactions.csv \
    --category-map data/category_map.csv \
    --n-clusters 5 \
    --output results/journey_segments.json

Step 5 — Export Dashboard

python scripts/export_mba_dashboard_json.py \
    --rules results/rules.json \
    --layout results/optimal_layout.json \
    --output dashboard_data.json

Output sequence:

1. [bash_tool] Mine rules + build lift matrix + run QAP
2. [web_search] Cross-sell lift benchmarks + basket size uplift by industry
3. [bash_tool] export_mba_dashboard_json.py → JSON
4. [show_widget] Dashboard: top rules + lift heatmap + layout grid + journey segments
5. [text] CMO/marketer insights + NBA (Next Best Actions)
6. [text] Caveats: causality vs correlation, seasonality, data volume requirements

---

Output Format — Visualization First

Dashboard panels (see references/mba_dashboard_template.html):

1. KPI bar — total rules found, top rule lift, avg basket size, % transactions with cross-sell opportunity
2. Top rules table — antecedent → consequent, support, confidence, lift, expected revenue
3. Lift heatmap — category × category matrix with color intensity = lift value
4. QAP optimal layout — grid visualization with category labels + adjacency arrows
5. Lift distribution — histogram of all rule lifts (identify high-value rules vs noise)
6. Journey segments — pie/bar of basket type distribution

---

Marketer Insights Layer (MANDATORY)

Search before benchmarking

web_search: "cross-sell association rules lift retail benchmark [year]"
web_search: "market basket analysis basket size uplift [industry] [year]"
web_search: "store layout optimization sales lift grocery [year]"

Translate Metrics to Business Language

Technical metric	Business meaning
support = 0.05	"1 in 20 baskets contains this product combination"
confidence = 0.72	"72% of customers who buy A also buy B — strong signal"
lift = 2.5	"A and B appear together 2.5× more often than random chance"
lift = 0.8	"Negative affinity — don't co-locate, don't bundle"
revenue(rule) = $4.50	"Each triggered cross-sell recommendation earns $4.50 on average"
n_rules_lift>2	"These are your high-confidence bundling/promotion opportunities"

NBA — Next Best Actions for Marketers/CMOs

Always produce 5–6 specific actions:

• Cross-sell widget: "Deploy top-5 rules (lift>2) as 'Frequently bought together' widget on PDP — target: +8–15% basket size"
• QAP adjacency: "Implement QAP layout in next planogram review — place top-lift category pairs adjacent; Frozen↔Drinks (1.5) is Katsov's canonical example"
• Email trigger: "When customer buys antecedent product, trigger cross-sell email within 24h featuring consequent product at discount — use uplift model to filter savable customers only"
• Bundle pricing: "High-confidence rules (conf>0.7, lift>2) → candidates for bundle pricing via two_part_tariff or price_segmentation skill"
• Lift threshold for promotion: "Only promote rules with lift>1.5 and support>0.02 — lower lift = noise, lower support = too rare to be worth the campaign cost"
• Seasonality retraining: "Retrain rules quarterly — FMCG seasonality shifts lift >20% between Q1/Q3. A beer+sunscreen rule in summer disappears in winter"

Connect to Business Objectives

MBA output	Objective	Activation
Top rules (lift>2)	Maximization — basket size	Cross-sell placement, bundle promotions
Negative lift pairs	Retention — reduce friction	Separate in-store to avoid confusion
QAP layout	Maximization — category revenue	Planogram reset, digital shelf reset
Journey segments	Acquisition + Retention	Personalized CRM per segment

---

Key Caveats

• Correlation ≠ causation: Beer+diapers is famous but may reflect demographics, not causality. Don't over-interpret low-lift rules.
• Support threshold matters: Too low → thousands of noisy rules. Too high → misses rare but valuable cross-sells. Default min_support=0.01 is a starting point.
• QAP is NP-hard: Brute force works for n≤8. Use simulated annealing (default) or OR-Tools for n>8.
• Asymmetry of rules: pasta→wine ≠ wine→pasta. Always check both directions.
• Digital vs physical: QAP optimizes physical adjacency. For digital shelves (eCommerce), replace distance matrix with scroll/click distance or recommendation slot position.
• Seasonality: Retrain at minimum quarterly — holiday seasons shift association patterns dramatically.

---

Integration with Agency Growth OS

Skill	Handoff
recommender-systems	Top association rules → non-personalized recommendation baseline (§5.11)
price-demand-optimization	High-lift pairs → bundling candidates (§6.5.4)
response-uplift-modeling	Cross-sell rules → uplift model: which customers to show the rec
crm-journey-architect	Journey segments → CRM sequence per basket type
creative-supply-planner	Top rules → define cross-sell asset requirements

---

Reference Files

• references/katsov_mba_excerpts.md — Eq. 5.155–5.158 + 6.117–6.123 + Example 6.8
• references/algorithm_notes.md — Apriori vs FP-Growth comparison + QAP heuristics
• references/industry_benchmarks_mba.md — Cross-sell lift benchmarks (SEARCH-FIRST)
• references/mba_dashboard_template.html — HTML dashboard; inject SKILL_DATA_JSON