Ai prioritize

Why this exists, and what I'd change

Why it exists. RICE doesn't work for AI features. Reach, impact, confidence, and effort assume predictable outputs. AI features live on a spectrum of "how well does it work" and need a different lens. This skill replaces RICE with four AI-native scores: model confidence, cost per inference, eval-readiness, and fallback quality.

Design tradeoffs.

• Four scores, not one composite. I don't roll them into a single number. Cost: harder to sort a backlog. PMs who want a ranked list have to apply their own weighting.
• Forces a ship/MVP/prototype/kill decision instead of a score. Every feature ends with a verb, not a number. Cost: less satisfying for execs who like a "RICE score of 47" answer.
• Eval-readiness is a hard gate. If you can't measure model quality on production data, the framework blocks shipping. Cost: real, slow features stuck in MVP because golden examples don't exist yet.

What I'd change. Add a calibration loop. After three AI features ship, compare predicted confidence against actual quality, and adjust the framework's thresholds based on the team's track record.

---

AI Feature Prioritization: Confidence-Gated Framework

When to use: When prioritizing AI-powered features, deciding what to ship vs prototype, or building an AI product roadmap. Traditional frameworks (RICE, WSJF, MoSCoW, ICE) break down for AI because they assume predictable impact and binary ship/no-ship decisions. AI features live on a spectrum of "how well does it work."

Framework source: Ashutosh Singh's AI Feature Prioritization framework (Confidence-Gated Prioritization)

Trigger phrases: "prioritize AI features", "should we ship this AI feature", "AI roadmap priority", "RICE for AI", "how confident is the model", "cost per inference", "eval-readiness", "AI prioritization matrix", "confidence-gated", "how do I prioritize this AI product"

---

Why Old Frameworks Fail for AI

Framework	Assumption	Why It Breaks for AI
RICE	You can estimate impact before launch	AI features have confidence intervals, not fixed impact
WSJF	Features are binary: shipped/not shipped	AI features work on a spectrum of "how well does it work"
MoSCoW	Scope is clear and fixed	AI scope depends on model capability. Changes weekly.
ICE	Confidence is about team certainty	AI confidence is about model accuracy. Different concept.

The core problem: A recommendation engine might work 92% of the time. Or 67%. You won't know until it's live. Old frameworks treat features as binary: shipped or not shipped. AI features are on a spectrum of "how well does it work."

---

The Four Lenses

1. Confidence-Gated Prioritization

Before you estimate impact, estimate model confidence.

The question: How confident are you that the model can do this task at production quality?

Confidence Level	Definition	Action
High (90%+)	Model reliably handles this task with minimal errors	Prioritize like a normal feature. Ship it. Standard sprint planning.
Medium (70-89%)	Model works most of the time but has notable failure modes	Build an MVP, test with real users, measure. MVP to 5% of users. Measure for 2 weeks. Decide.
Low (below 70%)	Model struggles or is inconsistent	Prototype only. Do not put on roadmap.

How to estimate confidence:

• Run the feature against 50-100 representative inputs
• Have domain experts grade the outputs
• Calculate the percentage that meets "production quality" bar
• If you can't run this test, your confidence is Low by default

Decision rule: Never roadmap a Low confidence feature. Prototype it, learn, revisit when model capability improves.

---

2. Cost-per-Inference Ranking

Every AI feature has a running cost. This is fundamentally different from traditional software where marginal cost is near zero.

The question: What does each API call cost, and does the business case support it?

How to calculate:

1. Estimate calls per user per month
2. Multiply by cost per call (API pricing, compute, etc.)
3. Get cost per user per month for this feature
4. Compare to the revenue or value the feature generates

Decision rules:

Cost/Inference	Required Business Case
< $0.01/user/mo	Ship freely. Cost is negligible.
$0.01 - $0.10/user/mo	Normal ROI justification. Show 3-5x return.
$0.10 - $0.50/user/mo	Strong ROI required. Show 10x return. VP approval needed.
> $0.50/user/mo	Requires VP approval + business case showing 10x ROI

Add "cost per inference" as a column in your prioritization spreadsheet. Most PMs skip this. Their CFO won't.

Pro tip: Costs drop over time as models get cheaper. A feature that's too expensive today might be viable in 6 months. Track it, don't kill it.

---

3. Eval-Readiness Scoring

If you can't measure whether an AI feature is working, you can't improve it. And you can't justify keeping it.

The question: Can you measure if this feature works?

Eval-readiness checklist:

Eval Component	Question	Status
Golden dataset	Do you have labeled test cases to score against?	Yes / No / Partial
Automated eval	Can you run evals without human review?	Yes / No / Partial
Success metric	Is the metric clearly defined (precision, recall, user satisfaction)?	Yes / No
Feedback loop	Can users flag bad outputs so you improve?	Yes / No
Baseline	Do you know performance BEFORE the AI feature?	Yes / No

Scoring:

• 5/5 (Eval-ready): Ship with confidence. You can measure improvement.
• 3-4/5 (Mostly ready): Ship, but invest in filling gaps first.
• 1-2/5 (Not ready): No eval = no improvement. Build evals first, then decide.
• 0/5: Do not ship. You are flying blind.

The hard truth: If you can't eval it, you can't improve it. Features without evals are features you'll regret launching.

---

4. Fallback Quality Assessment

What happens when the model fails? This is the question that separates mature AI products from reckless ones.

The question: Is the fallback graceful (show nothing) or catastrophic (show wrong info)?

Fallback Type	Definition	Confidence Threshold
Graceful	Feature hides or shows generic content when model fails. User doesn't notice.	Standard threshold (70%+ confidence to ship)
Informational	Feature shows a result that might be slightly wrong. User can verify.	Higher threshold (80%+ to ship). Add disclaimers.
Catastrophic	Wrong output causes real harm: bad financial advice, incorrect medical info, wrong transaction.	Raise confidence threshold to 95% before shipping. Add human-in-the-loop review.

Decision rule: Features with catastrophic fallbacks need higher confidence thresholds before shipping. Period.

Fallback strategies:

• Graceful: Hide the feature, show a generic alternative, say "I don't have enough info"
• Informational: Show the AI output with a confidence indicator, let user verify
• Catastrophic: Require human review before showing output, or don't launch until 95%+ confidence

---

The AI Prioritization Matrix

Combine confidence and impact into a 2x2:

	High Impact	Low Impact
High Confidence (90%+)	MUST SHIP - Prioritize like a normal feature	PRIORITIZE - Ship when capacity allows
Medium Confidence (70-89%)	MVP TEST - Build MVP, test with real users, measure	BACKLOG - Low priority. Revisit quarterly.
Low Confidence (below 70%)	PROTOTYPE ONLY - Don't roadmap. Explore and learn.	KILL - Not worth the investment right now.

---

New Columns for Your Prioritization Sheet

Add these four columns to your existing prioritization spreadsheet:

Feature	Confidence	Cost/Inference	Eval Ready	Fallback	Impact	Decision
AI search	92%	$0.03	Ready	Graceful	High	Ship
AI summaries	78%	$0.03	Ready	Informational	High	MVP test
AI agent	55%	$0.30	Not ready	Catastrophic	High	Prototype only
AI pricing	45%	$0.08	Partial	Catastrophic	Medium	Kill

---

How to Run an AI Prioritization Session

Step 1: List All AI Features

Gather every AI feature on your roadmap or backlog. Include proposed, in-progress, and shipped features.

Step 2: Score Each Feature on Four Lenses

For each feature, answer:

1. Confidence: What % of the time does the model get this right at production quality? (Run eval if possible, estimate if not)
2. Cost/Inference: What does it cost per user per month? (API costs + compute + any human review)
3. Eval-Readiness: Score 0-5 on the eval checklist
4. Fallback Type: Graceful, Informational, or Catastrophic?

Step 3: Apply Decision Rules

If...	Then...
Confidence > 90% AND eval ready	Ship it. Standard sprint planning.
Confidence 70-89%	MVP to 5% of users. Measure for 2 weeks. Decide.
Confidence < 70%	Prototype only. Do not put on roadmap.
No eval dataset exists	Stop. Build evals first. Then decide.
Cost/inference > $0.10	Requires VP approval + business case showing 10x ROI
Catastrophic fallback	Raise confidence threshold to 95% before shipping

Step 4: Build the Prioritized Roadmap

1. Immediate ship: High confidence + eval ready + acceptable cost
2. MVP test: Medium confidence + eval ready (or close)
3. Eval investment: Features worth pursuing but eval not ready (build evals first)
4. Prototype backlog: Low confidence features to explore when capacity allows
5. Kill list: Low confidence + low impact + high cost. Remove from roadmap.

---

Applying This in High-Stakes Domains

When prioritizing AI features in regulated or high-stakes domains (finance, healthcare, legal, education), these lenses are especially critical:

Fallback risk is elevated. High-stakes advice means wrong outputs are catastrophic, not just annoying. Apply the 95% confidence threshold and define every fallback before ship.

Cost per inference matters. Frequent interaction means even $0.05/call adds up fast across a user base. Model cost per feature per user per month.

Eval-readiness is non-negotiable. Domain accuracy requires golden datasets. Before shipping any AI skill, build test cases with known-correct outputs.

Confidence varies by skill type:

• Structured data + clear rules — likely high confidence
• Subjective or market-dependent advice — likely medium confidence
• Open-ended, emotional, or behavioral output — likely low-medium confidence
• Rule-based with edge cases — needs careful evaluation

---

Quick Start

When the PM says "prioritize this AI feature" or "should we ship this AI feature," run through this sequence:

1. Identify all AI features to prioritize (or focus on the one in question)
2. Score each on 4 lenses (Confidence, Cost, Eval-Readiness, Fallback)
3. Apply the prioritization matrix (Must Ship / MVP Test / Prototype Only / Kill)
4. Apply decision rules for edge cases
5. Output a prioritized table with clear ship/test/prototype/kill recommendations
6. Flag any features with catastrophic fallbacks that need elevated thresholds

---

Output Format

When delivering AI prioritization output, include:

AI Feature Prioritization Summary

Date: [today] Features assessed: [count]

Prioritization Table

#	Feature	Confidence	Cost/User/Mo	Eval Score	Fallback	Impact	Decision
1	[name]	[%]	[$]	[0-5]	[type]	[H/M/L]	[Ship/MVP/Proto/Kill]

Ship Now (High Confidence + Eval Ready)

• [features and why]

MVP Test (Medium Confidence)

• [features, test plan, success criteria]

Build Evals First

• [features that need eval investment before deciding]

Prototype Only (Low Confidence)

• [features to explore, not roadmap]

Kill (Not Worth Pursuing)

• [features and why]

Cost Alerts

• [any features where inference cost needs VP approval]

Catastrophic Fallback Warnings

• [any features with dangerous failure modes]

---

Related Skills

• /prioritize - LNO Framework for PM task prioritization (different: tasks vs features)
• /impact-sizing - Quantify feature value with driver trees (use alongside this for the "Impact" column)
• /experiment-decision - When to A/B test vs ship (complements the MVP Test decision)
• /feature-metrics - Define success metrics (feeds into Eval-Readiness scoring)
• /mece - Structure AI feature categories exhaustively

---

Context Routing Strategy

When the PM uses /ai-prioritize, I automatically:

1. Check Existing AI Features

Source: context-library/prds/, outputs/prds/, outputs/roadmaps/roadmap-data.json

• Look for AI-powered features already on the roadmap
• Pull confidence estimates from any prior analysis

2. Reference Cost Data

Source: context-library/metrics/, connected analytics MCPs

• Pull actual API costs if available
• Estimate based on model pricing if not

3. Check Eval Status

Source: context-library/research/, test datasets if they exist

• Identify which features have golden datasets
• Flag features without any eval capability

4. Apply Company-Specific Context

Source: context-library/business-info-template.md, context-library/research/

• Apply any elevated fallback risk (e.g., financial, medical, legal domains)
• Consider any regulatory or compliance requirements
• Reference your AI feature library for context

5. Cross-Reference with LNO

Source: /prioritize output if available

• If the PM has already classified tasks with LNO, layer AI prioritization on top
• A feature can be "Leverage" in LNO terms but still "Prototype Only" on confidence

---

Output Quality Self-Check

Before delivering the AI prioritization output, verify:

• Every AI feature scored on all 4 lenses: No feature left without Confidence, Cost, Eval, and Fallback scores
• Decision rules applied consistently: Matrix placement matches the scores
• Catastrophic fallbacks flagged: Any feature where failure is dangerous is explicitly called out
• Cost per user per month calculated: Not just cost per call, but monthly user-level cost
• Eval gaps identified: Features missing eval capability are flagged with "build evals first" recommendation
• Actionable next steps: Each feature has a clear action (ship, MVP test, build eval, prototype, kill)
• No traditional framework assumptions: Output does not assume binary ship/no-ship or fixed impact estimates

---

Framework credit: Ashutosh Singh (Product Growth). AI Feature Prioritization: Confidence-Gated Framework.