Ai benchmark distribution

AI Benchmark Distribution

Role / Purpose

Eval Strategist. For AI products competing on capability — foundation models, AI coding tools, AI infrastructure — benchmarks are not just technical artifacts. They are the primary distribution mechanism in the developer and research community. This skill is about designing that moment intentionally.

Note: the capability claims precision framework, unsubstantiated comparison rules, and benchmark methodology requirements are in pmm/DOMAIN.md (AI product content section). This skill covers the distribution pipeline — what to publish, where, how, and when.

---

Contract

This skill guarantees:

• No benchmark is published without a documented methodology (this is a hard gate, not a preference)
• The "when NOT to publish" check runs before the distribution plan is built
• Platform-specific format is defined for each distribution surface
• The 48-hour seed window is designed before public posting
• Failure cases appear in the same post as the headline result

---

Before starting

Confirm (ask or infer):

• Benchmark type — head-to-head comparison / task-specific deep dive / eval methodology contribution / community leaderboard entry?
• Result novelty — does this show something the audience doesn't already know?
• ICP relevance — is this the capability that matters to the target engineering audience?
• Methodology status — is the methodology fully documented and reproducible?
• Response availability — can the team respond to technical questions within 72 hours of posting?

---

Inputs

Required before proceeding:

• The benchmark result or eval finding
• The task, evaluation setup, and metric (must be fully described)
• The comparison baseline (or confirmation that none is claimed)
• The target distribution surfaces
• Response availability in the 48–72 hours after posting

---

Step 1 — Publish decision gate

Run this before any distribution planning. All three must be true to proceed:

PUBLISH DECISION GATE

Criterion 1 — Methodology is credible:
  [ ] Test set is fully described and could be reproduced by a third party
  [ ] Evaluation criteria and scoring are documented
  [ ] Comparison baseline is named explicitly (model name + version + date)
  IF any box unchecked → DO NOT PUBLISH. A benchmark without methodology is
  a marketing claim. It invites replication attempts that undermine trust.

Criterion 2 — The result is novel:
  [ ] Shows something the audience does not yet know about the model or category
  [ ] Does not merely confirm what everyone already believes
  IF criterion fails → consider whether the distribution value justifies the effort;
  a benchmark that confirms the obvious gets no distribution.

Criterion 3 — ICP-relevant:
  [ ] The capability being measured matters to the target engineering community
  [ ] Example: a coding benchmark for a developer tool; a legal reasoning benchmark is not
  IF criterion fails → wrong benchmark for this audience; reconsider the claim.

ALSO CHECK — when NOT to publish (see Step 2)

---

Step 2 — When NOT to publish

Situation	Why to wait
Methodology is not reproducible yet	Will be challenged and found lacking; damages credibility more than silence
Result is strong only on benchmarks the model was trained on	Community will discover contamination; damage is larger than the benefit
Team cannot respond to technical questions for 72 hours	HN/Twitter discussions peak within 48 hours; unanswered technical challenges read as evasion
Benchmark is standard and result is middling	Neutral results get no distribution and signal that you are lagging

---

Step 3 — Benchmark type and distribution surface

Type	What it measures	Primary distribution surface	Best for
Head-to-head comparison	Your model vs a named alternative on a shared task	Twitter/X, HN, AI newsletters	Attention in a competitive launch
Task-specific deep dive	Your model on a narrow, high-value capability	Technical blog, HN, Discord	Developer credibility in a specific use case
Eval methodology contribution	A new way to measure something the community cares about	arXiv, GitHub (open eval), HN	Thought leadership; positions as research contributor, not just vendor
Community leaderboard entry	Submission to MMLU, HumanEval, HELM, LMSYS Chatbot Arena	Leaderboard drives discovery automatically	Ongoing presence on trusted third-party surfaces

Leaderboard entries first: leaderboard entries have the highest credibility-per-unit-of-effort because the methodology is not yours. Enter every relevant leaderboard before publishing proprietary benchmarks.

---

Step 4 — Benchmark-to-distribution pipeline

BENCHMARK DISTRIBUTION PIPELINE

1. INTERNAL EVAL RESULT
   ↓
2. METHODOLOGY DOCUMENTATION
   Write this first — before distribution planning.
   What was tested, how, using which evaluation criteria, compared to what baseline.
   If this cannot be written clearly, do not proceed.
   ↓
3. FRAMING DECISION
   Which story does this result support?
   Name the one finding that matters most to the ICP.
   This becomes the lead, not the last paragraph.
   ↓
4. ASSET PREPARATION
   [ ] Technical blog post (methodology → finding → failure cases → raw data table)
   [ ] Reproducibility package (eval code on GitHub if possible; open eval = 10× credible)
   [ ] Data table (in a format readers can export or reproduce)
   ↓
5. SEEDING (48 hours before public posting)
   Identify 5–10 researchers or developers who will find this genuinely interesting.
   Share early — not for endorsement, but for engagement.
   Why: the first upvotes and shares come from people who saw it early and had time to think.
   A cold post with zero initial engagement dies in the algorithm.
   ↓
6. PRIMARY POST
   Technical blog → HN (Show HN or Tell HN depending on type)
   See Step 5 for platform format.
   Timing: Tuesday–Thursday, late morning Pacific.
   Avoid major conference weeks (NeurIPS, ICML) unless attending.
   ↓
7. SECONDARY DISTRIBUTION
   Twitter thread summarizing key finding
   AI newsletter pitch (3–5 sentence summary + link)
   Discord/Slack communities relevant to the capability
   ↓
8. RESPONSE
   Engage with technical criticism within 4 hours.
   Do not deflect from methodology challenges — engage specifically.
   A team that engages with criticism earns more credibility than one that goes silent.

---

Step 5 — Writing rules and platform formats

Writing rules (apply to all benchmark posts)

Do:

• State the exact task, the exact evaluation setup, and the exact metric in the first paragraph
• Name the comparison baseline explicitly: "compared to [model/version] on [date] using [method]"
• State what the model does NOT do well in the same post — this is the single most credibility-building move
• Show raw numbers in a table that readers can export or reproduce
• Link to the eval code if possible

Do not:

• Lead with the conclusion before the methodology ("X% better than [alternative]" as the headline)
• Use relative improvement without absolute numbers ("50% improvement" means nothing without the baseline)
• Suppress failure cases from the data
• Publish benchmarks where the model was known to be trained on the test set (contamination is tracked by the community)

HN format

HN POST FORMAT

Title: Show HN: [What you built / measured] — [single most interesting finding, factual]
Example: "Show HN: Open eval for function-calling accuracy across 8 models
          — surprising spread on multi-step tasks"

Rules:
  - Factual title: no adjectives, no marketing claims
  - The finding is in the title — not "we built an eval" but "here is what we found"

First comment (required — write before posting):
  Cover:
  - Methodology: what you tested, how, with what evaluation criteria
  - What you did NOT test (important for credibility)
  - Known limitations of the evaluation
  - Link to raw data and/or eval code
  This is where the credibility lives. The post earns the click; the first comment earns the trust.

Twitter/X format

TWITTER/X THREAD FORMAT

Tweet 1: The finding, stated precisely, with the key number
  Example: "[Task]: [Model A] scored [X%] vs [Model B] at [Y%] on [specific evaluation].
            Full methodology and raw data in thread."

Tweets 2–4: One finding per tweet, with data visualization or table screenshot
  Limit: one number per tweet (not a list)

Last tweet: Link to full post + link to reproducibility package on GitHub
  "Full methodology and raw data: [link] | Eval code: [link]"

Reply rule: respond to every substantive technical challenge within 4 hours.
  "We will look into this" is not a response. Engage with the specific concern.

---

Output format

## Benchmark Distribution Plan

**Benchmark type:** [Head-to-head / Task-specific / Eval methodology / Leaderboard]
**Finding:** [One sentence: task, metric, result]
**Comparison baseline:** [Model/version/date]
**Methodology status:** [Fully documented / In progress — DO NOT PUBLISH until complete]

### Publish decision gate
Criterion 1 (Methodology): [Pass / Fail — with note]
Criterion 2 (Novelty): [Pass / Fail — with note]
Criterion 3 (ICP relevance): [Pass / Fail — with note]
Verdict: [Proceed / Do not publish — reason]

### Distribution plan
Leaderboard submission: [Which leaderboards / None applicable]
Primary post: [Technical blog + HN / HN only / arXiv]
Secondary: [Twitter thread / Discord / Newsletter]
Timing: [Day and time]
Seed list: [N people identified / Not yet identified]

### Asset checklist
[ ] Technical blog post draft
[ ] Raw data table (exportable)
[ ] Eval code / reproducibility package
[ ] HN first comment (written before posting)
[ ] Twitter thread (if secondary)

### Failure cases in post
[ ] Confirmed: failure cases appear in same post as headline result

---

Brain reads / writes

If a companion brain repo is connected:

Before starting:

• Read playbooks/messaging.md — benchmark framing must align with approved positioning; the benchmark should be evidence for the narrative, not a separate story
• Read knowledge/icp-map.md — ICP determines which benchmark type has the most credibility (ML researchers respond to arXiv; developers respond to GitHub + HN)

Brain not connected: proceed normally.

---

Anti-patterns

Anti-pattern	Why it fails	Fix
Publishing before methodology is reproducible	Community replication attempts expose gaps; credibility damage exceeds the initial boost	Methodology documentation is the first step, not the last
Leading with the conclusion before the methodology	AI developer community reverse-engineers the methodology; leading with a claim signals it won't hold up	State methodology in paragraph 1; lead finding in paragraph 2
Suppressing failure cases	Community will run your eval and find them; finding them before you published them destroys credibility	State what the model does not do well in the same post
Publishing on a Friday or during a conference week	HN/Twitter engagement peaks on weekday mornings; conference weeks are attention voids	Tuesday–Thursday, late morning Pacific; avoid major ML conference weeks
Seeding zero people before public posting	Initial velocity determines algorithmic distribution; a cold post dies	5–10 people with genuine interest, 24–48 hours before public
Not responding to technical challenges	Silence reads as evasion; the community assumes the challenge is valid	Plan response coverage for 72 hours post-publication; engage specifically

---

Validation criteria

• Publish decision gate passed (all three criteria)
• When-not-to-publish check completed
• Methodology documented before distribution planning
• Failure cases included in post, not omitted
• HN first comment written before posting
• Seed list identified before public posting
• Response coverage planned for 72 hours post-publication

---

References & Sources

Tier 2 (operator source — adapted, not authoritative):

• ai-benchmark-distribution (growth-skills v1.0): publish decision gate, benchmark type taxonomy, distribution pipeline, HN/Twitter format rules, when-not-to-publish criteria