Reasoning self consistency

Self-Consistency — Sample and Vote

What this actually does (and doesn't)

This does not make the underlying model smarter. It spends more compute to buy reliability: instead of trusting one reasoning chain, it runs several independent ones and keeps whichever final answer shows up most. It works because on tasks with a single correct answer, wrong reasoning tends to scatter across different wrong answers while correct reasoning converges on the same right one — so the majority is more often right than any single attempt. It does nothing for open-ended tasks (there is no "majority essay") and it does not fix a model that is systematically wrong the same way every time (correlated errors defeat the vote).

When to use it

Use when ALL of these hold:

• The task has a discrete, checkable correct answer (a number, a label, a short exact string, yes/no, a ranked option, a code/calculation output).
• Different phrasings of the same answer can be normalized into one comparable form (e.g. "8", "eight", "8.0" all mean the same thing).
• The stakes or error cost justify N× the compute/latency of a single pass.

Do NOT use for: open-ended writing, subjective judgment calls, tasks needing a live tool/fact lookup where the model would just repeat the same wrong lookup N times, or anything whose failure mode is "confidently wrong the same way every time" rather than "wrong in different, random ways."

Inputs this skill needs

1. The task/question to solve.
2. N — number of independent samples. Default 5. Use 3 for cheap/low-stakes tasks, 7–9 for high-stakes tasks or when votes are close. Always pick an odd N to avoid ties.
3. The answer-normalization rule — how to decide two attempts "agree" (exact match, case-insensitive match, numeric equality/tolerance, same multiple-choice letter, etc.). If none is given, state the rule you are using before starting.
4. Optional: a tie-break rule (default: no consensus -> report all leading answers, do not silently pick one).

Steps

1. Confirm it qualifies. State the discrete answer format and the normalization rule you will use to compare attempts. If the task is actually open-ended, stop and say this technique does not apply.
2. Pick N (default 5, odd).
3. Generate N independent attempts. For each attempt:
   • Solve the task from scratch as if the other attempts did not exist.
   • Vary the approach where possible (different method, different order of operations, different starting assumption) — this is what creates real independence. Producing N copies of the identical reasoning chain is not sampling, it is one answer typed out N times, and it will not improve reliability.
   • Do not look back at a previous attempt's scratch work while producing the next one.
4. Extract the final answer from each attempt into the normalized form from step 1.
5. Tally votes. Count how many attempts landed on each normalized answer.
6. Pick the result:
   • Majority/plurality answer wins.
   • If two or more answers tie, apply the tie-break rule (default: report the tie, do not guess).
7. Report honestly using the output format below. Always show the tally — never present the majority answer as if it were unanimous when it was not.

Output format

Task: <one-line restatement>
N attempts: <N>
Normalization rule: <how answers were compared>

Attempts:
  1. <final answer>  — <one-line method/approach used>
  2. <final answer>  — <one-line method/approach used>
  ...

Vote tally: <answer A>: x/N, <answer B>: y/N, ...
Result: <winning answer>
Confidence: unanimous (N/N) | strong majority (>=2/3) | weak majority (>1/2) | no consensus (tie)

If "no consensus," say so explicitly and either present the leading candidates or escalate (larger N, a stronger model, human review) instead of forcing a pick.

Worked example

Task: "Apples are sold in packs of 3 for $2 per pack. A customer needs 11 apples and can only buy whole packs. How much do they pay?"

Normalization rule: compare as a dollar amount rounded to 2 decimals.

Attempts (N=5, varied approaches):

1. $8.00 — packs needed = ceil(11/3) = 4 -> 4 x $2
2. $8.00 — listed multiples of 3 (3, 6, 9, 12) -> 12 is the first >= 11 -> 4 packs x $2
3. $7.33 — treated it as (11/3) x $2, did not round up to whole packs
4. $8.00 — built a table of packs 1..5 and picked the first that covers 11 apples -> 4 packs
5. $8.00 — divided 11 by 3 = 3.67, rounded up to 4 packs x $2

Vote tally: $8.00: 4/5, $7.33: 1/5 Result: $8.00 Confidence: strong majority (4/5)

The single wrong attempt (#3) failed by skipping the "whole packs only" constraint — a reasoning slip, not an arithmetic one. The vote absorbed it because the other four independent approaches converged on the same right answer.

Limitations to state out loud when reporting results

• Self-consistency raises the odds the majority is right; it does not guarantee it. A model that is wrong for a shared, systematic reason (misreads the question the same way every time) can produce a confident, unanimous, wrong vote.
• Cost scales linearly with N — N samples is roughly N times the tokens/latency of one pass. Do not use N=9 where N=3 would do.
• This is one building block, not a full reasoning system. It pairs well with a verification/self-critique pass on the winning answer when the stakes justify it — but that is a separate technique, not part of this one.