Research integrity audit

Research Integrity Audit

Audit without modifying the input. Treat block as a release stop, warn as a required review, and pass only as “no rule violation detected,” never as proof that a historical claim is true.

Run

From this Skill's own folder, so the command works both inside the repository and in a copy of the folder downloaded on its own:

python3 scripts/audit_case.py \
  path/to/research-case.json \
  --output path/to/integrity-audit.json \
  --pretty

Existing report paths are not overwritten. Exit codes are 0=pass, 1=warn, 2=block, and 3=input/output error.

Interpret

Read status, then substance, then checks and findings. Resolve every block before packaging. Review every warn, recording why it was resolved, accepted, or left unresolved. Preserve the report as an audit event and rerun after changing claims or evidence.

Vacuous versus substantive

status: pass alone never means a case earned anything. Read it together with substance:

field	meaning
claims_examined	how many claims the gate looked at, in total
substance.publishable_claims_examined	how many carried publication_status: publishable
substance.substantive_claims_examined	how many of those had at least one existing, non-HYP, non-generated evidence record
substance.level	substantive when that last count is above zero, otherwise vacuous
publishable	true only when status is pass and substance.level is substantive

A case with zero publishable claims is vacuous and is reported as block (VACUOUS_AUDIT_NO_PUBLISHABLE_CLAIMS). This gate exists to certify claims; certifying nothing is not a pass. An empty case and a validated case must never be readable as the same result.

Checks

Every rule is reported individually in checks, with its own finding_ids.

rule	name	severity
RIA-001	Publishable claims have evidence	block
RIA-002	HYP is isolated from publishable claims	block
RIA-003	Conflicting evidence is surfaced	warn
RIA-004	Qualified case reaches pass	reporting marker only
RIA-005	Claim substance is declared and non-vacuous	block
RIA-006	Completion requires substantive claims	block
RIA-007	Evidence references resolve	block
RIA-008	Evidence records are not orphaned	warn
RIA-009	Machine output is adjudicated before it supports a claim	block
RIA-010	AI actors do not satisfy human gates	block
RIA-011	Accuracy figures require a trusted transcription	block
RIA-012	Case hash binding is intact	block
RIA-013	Unresolved blockers are still in force	warn, or block during a completion attempt
RIA-014	Sticky blockers are not silently superseded	block
RIA-015	Evidence layers are not confused	block
RIA-016	High-impact uncertainties stay visible	warn
RIA-017	Rejected evidence never supports a publishable claim	block
RIA-018	Adjudication records name an accountable human reviewer	block

Definitions the checks rely on:

• Substantive claim. A publishable claim with at least one linked evidence record that exists in the case, is not HYP, is not registered under a HYP material, is not produced by a generated material, and is not rejected.
• Trusted support. Only verification_status: human-checked. This is an allowlist, never a denylist: support whose status is missing, non-string, misspelled, outside the schema enum, unverified, machine-checked, or conflicted counts as unadjudicated. Unknown provenance is untrusted, so the gate fails closed.
• Completion attempt. stage is packaging or complete, or extensions.stop_condition_met is true, or a decision with action: complete exists.
• Adjudication. For RIA-009, a claim resting only on untrusted support needs a decision carrying an adjudication object that names an accountable human — a non-empty reviewer_role that is not an automated actor, plus a non-empty rationale and reviewed_at — and whose affected_ids, evidence_ids, or evidence_assessments reach the claim or its supporting evidence. An empty or half-written adjudication object clears nothing and is itself reported as RIA-018.
• Automated actor. A reviewer role or evidence method naming AI, an agent, an LLM, or generated or automated production. An AI actor never satisfies a human gate, and an AI adjudication never clears RIA-009.
• Trusted transcription. For RIA-011, an evidence record that is human-checked, sits on the OBS or SRC layer, was not produced by an automated actor, and either reads as a transcription or ground truth in its own text or is designated by extensions.trusted_transcription.
• Reported metric figure. For RIA-011, an accuracy, CER, WER, precision, recall, or F1 word within forty characters of a digit, or a metric-named key with a numeric value. An honest negative sentence such as “these differences do not establish OCR accuracy” carries no digits and is not treated as a figure.
• Sticky blocker. A decision with action human_review, stop_unresolved, or rollback. It stays in force until a later continue, reroute, or complete decision names it in resolves_decision_id. Resuming without that reference is RIA-014, not a resolution: the blocker stays open and is still reported by RIA-013. Blockers accumulate — every unresolved blocker is tracked, so a second blocking decision never buries the first, and a resumption that names one blocker lifts only that one. RIA-013 reports each still-open blocker separately.
• Intact hash binding. extensions.latest_decision_state exists exactly when decisions exist, identifies the last decision, reproduces its digest, carries a SHA-256 input case hash, and never claims more records than the case holds. Marker counts that lag the case are ordinary staleness from a non-decision append and are accepted.
• Layer confusion. Evidence presented as OBS or SRC whose material is registered as DER, INF, or HYP, or whose material role is generated, or whose derived material sits on a different layer.

Report contract

The report envelope is laddered through report_schema_version, currently 1.1. Every field emitted by version 1.0 is still emitted with the same meaning; 1.1 only adds report_schema_version, claims_examined, substance, publishable, and extra summary counters.

skill_version deliberately stays 1.0.0. skills/rubbing-research-orchestrator/scripts/orchestrate.py pins that exact string, together with skill_id, case_id, input_case_sha256, input_unchanged, and matching status/gate_status. Ladder the report contract; do not repurpose those fields.

Limits

The audit does not fetch sources, authenticate artifacts, give legal clearance, detect records omitted from the input, or replace expert review. RIA-004 is a legacy reporting marker with no independent finding path; it is retained for report compatibility. RIA-018 checks that an adjudication names a reviewer, not that the named reviewer exists or is qualified; identity cannot be verified from the JSON alone. A pass means no implemented rule was violated by the supplied case, nothing more.