Discovery rubric auditor

Discovery rubric auditor

Score the pattern, not the one-off. If the finding could be one bad call, do not coach the team on it.

An audit needs a comparable sample, one rubric, and a few recurring gaps. One recorded call is call-review-coach. A pass/fail readiness decision is certification-program-builder.

Pattern. "In 7 of 9 mid-market first calls, reps named a problem and never asked what it costs. Coach the cost question next week. Do not rank the AE with two calls."

One-off. "The team needs better discovery. AE3 scored 2.1. Roll out a discovery refresh."

If you cannot say how often the gap appeared, in which comparable slice, it is not a pattern.

Confirm Inputs First

Ask only for the inputs that change the audit:

1. The sample and why these calls. One call: route to call-review-coach
2. Comparison units: role, segment, stage, call type, time window
3. Source coverage: recordings, transcripts, notes, or summaries
4. Rubric source, or use the reference
5. Reviewer count and whether they already calibrated
6. The decision this audit must support: coaching priority or baseline, not certification

Continue from a thin sample with the limits named. Do not invent a leaderboard.

Read The Right Reference

Read references/discovery-rubric-framework.md for anchors, calibration, and when a number is only directional. Skip it if the user already supplied a calibrated rubric and only wants the pattern read.

Default Workflow

1. Frame the sample. Separate unlike roles, segments, stages, or call types. Done when each reported unit is comparable.
2. Calibrate, or label the single-reviewer path. Agree anchors on benchmark calls first. One reviewer: delayed blind rescore and inter-rater reliability: not established. Done when disagreements greater than one point are resolved or named.
3. Score with evidence. Every dimension gets a score or N/O, coverage, and confidence. Silence in missing audio is not a 1. Done when call-level evidence can be inspected.
4. Check drift. Double-score a subset, or blindly rescore after a delay. Pause aggregation if anchors still disagree.
5. Aggregate only comparable units. Count, median, distribution, N/O rate, coverage. Every number keeps its denominator. Fewer than three comparable calls per unit is directional.
6. Separate pattern from one-off. Recurring clusters only. Frequency is not causation. Skill gap, process gap, and sample-shape issue are different findings.
7. Pick a few interventions. What managers inspect in the next call sample. Done when each action has a behavior and a slice.

Conversation-intelligence scores are leads, not ground truth. Default reporting is de-identified.

For meeting architecture use discovery-and-demo-structure. For one call use call-review-coach. For a formal gate use certification-program-builder.

Output Contract

Return an audit a manager can act on next week:

• Scope: who, which calls, scoring window
• Sample shape: counts by unit, exclusions, coverage, skew
• Calibration record, including single-reviewer limits
• De-identified results with distributions; identifiable appendix only if authorized
• Patterns with frequency, impact, and confidence
• A few coaching moves, ranked by likely effect
• What to inspect in the next sample
• Limitations that keep conclusions directional

Do not rank reps from a mixed or tiny sample. Do not issue a pass/fail.

Quality Bar

• Unlike calls are not on one leaderboard.
• Every finding cites call evidence and a denominator.
• N/O is never averaged as zero.
• One reviewer is labeled inter-rater reliability: not established.
• Interventions are few, and each is a pattern, not a one-off.
• The audit refused the single-call review and the certification decision.