Science council

Science Council

A deliberation harness where several independent LLMs act as panelists on a scientific question, and R is the chair: it fans out the calls, enforces the protocol, runs the arithmetic, and owns the evidence store.

The premise: a single model's answer is an opinion. An answer that survived independent proposal, mandatory two-sided argument from every panelist, recomputation in R, and citation resolution against Crossref/Europe PMC is something you can build on. Only the latter is written to the database.

Ground rules the protocol enforces

1. Round 1 is blind. Panelists commit before seeing each other — no anchoring.
2. Right and duty to argue both sides. Every panelist owes a pro AND a contra on every claim, including claims it intends to vote against (or for). A panelist that supplies only one side has its vote on that claim discarded (R/council.R, discharged flag).
3. Facts outrank votes. An R check that fails refutes a claim no matter how the panel voted. A DOI that does not resolve marks its claim contested.
4. Dissent is recorded, not smoothed away. Surviving minority positions stay in the report and the database.
5. Only validated claims enter the store — with their full provenance.

Quick start

cd ~/.claude/skills/science-council

./bin/council doctor                          # who is reachable right now
./bin/council science                         # Claude Science envs + connectors
./bin/council run --question "..." --rounds 2 # convene
./bin/council search --query "..." --status validated
./bin/council report --qid q_xxx --out report.md   # full deliberation record (md)
./bin/council stats-report --qid q_xxx        # R statistics: charts + HTML
./bin/council stats                           # store + panelist calibration

Always run doctor first if anything looks off — it live-probes every provider, the R sandbox and the citation APIs, and it is the fastest way to see which panelist is down.

How to drive this as Claude

Pick the panel deliberately. --panel takes seats, not just providers: gemini,ollama/llama3.1:8b,ollama/qwen2.5-coder:7b seats three distinct minds, two of them free and local. A seat is provider or provider/model-override. Two reachable seats is the minimum; three or more makes the agreement statistics meaningful. Include at least one frontier seat (claude/openai/gemini) — without one, no R checks run at all, by design: a small local model asked to author a check writes code that refutes true claims, and a failing check overrides the whole panel. An all-local panel still debates and still gets its citations checked.

Sessions are slow by design (minutes: N panelists × claims × rounds, plus local model latency). Launch with run_in_background: true, then read the log. Never wrap it in a short timeout. Do not wait on it with until ! pgrep -f "council run" — that pattern matches the waiting shell's own command line and hangs forever. Wait on the task notification, or pgrep -f Rscript.

Report honestly. Lead with what got validated and what got refuted or contested — a session where the panel disagreed is a result, not a failure. Cite the consensus score and κ. If a citation failed to resolve, say so plainly: that is a model having fabricated evidence, and it is the single most valuable thing the run found.

Do not launder the output. Never present a contested or unresolved claim as settled, and never fill a gap with your own knowledge without labelling it as yours rather than the council's.

Claude's own seat (bridge mode)

doctor reports a Claude reachability mode:

• api — ANTHROPIC_API_KEY is set. Claude runs autonomously inside R. Best.
• cli — the claude CLI is logged in (claude /login in a terminal). Also autonomous.
• bridge — neither. R writes each Claude prompt to runs/bridge/req_*.json and blocks; you, the driving session, answer it.

Bridge mode is the only way Claude gets a seat without a key — and it is where Claude earns its place, because you can ground your answers with WebSearch, the deep-research skill and any connected Life Sciences connectors, which no other panelist can do. To serve the bridge:

1. Launch the run in the background with SCICOUNCIL_BRIDGE_ACTIVE=1 and claude in --panel.
2. Poll runs/bridge/ for req_*.json. Read the file: it has prompt, system, want_json.
3. Answer it as a panelist under the protocol above — research the claims properly, argue both sides, never invent a DOI.
4. Write runs/bridge/resp_<id>.json containing {"text": "<your JSON answer>"}, where <id> is the request's id. R picks it up within a second.

Requests time out after SCICOUNCIL_BRIDGE_TIMEOUT (default 1200s), so serve them promptly. ./bin/council bridge --status lists pending requests; --clear empties the queue.

What comes back

council run prints each claim with its status, consensus score and vote split, then the session id. report expands that into the full record: every panelist's pro and contra, every vote and rationale, the R check code and its result, and each citation with its resolution status.

• validated — panel consensus ≥ threshold, no surviving refutation, any R check passed
• refuted — consensus against, or the R check failed (arithmetic wins)
• contested — real disagreement, or the supporting citations do not resolve
• unresolved — too few valid votes, or the panel is collectively uncertain

Every claim carries a status_reason saying which rule fired — read it. A contested verdict sitting next to a unanimous vote is not a bug: it means the panel agreed but the evidence it offered does not exist.

consensus_score ∈ [-1, +1] is confidence-weighted and calibration-weighted: a panelist whose past votes matched the settled outcome carries more weight (calibration table, weight ∈ [0.25, 2.0]). Fleiss' κ and Krippendorff's α report how much the panel actually agreed — α is the honest one when a panelist dropped out mid-run.

Two things worth knowing before you trust a session

Citations are checked twice. A DOI/PMID must resolve (Crossref, Europe PMC) and be on-topic for its claim — models cite real papers from the wrong field, and a real DOI about protein folding is not evidence about blood pressure. Both gates must pass before a citation counts. In the first live session, 7 of 22 offered DOIs did not exist at all. Expect this; it is the point.

R checks only bite on decidable claims. Mathematical and probabilistic claims (and questions about what a statistical method can show) are genuinely settled by computation. Empirical claims about the world usually are not — unless real data is at hand (see below), a model asked to check one will cheerfully simulate data with the effect baked in and report passed = TRUE. That is circular. A passed = TRUE on an empirical claim is a red flag: read r_checks.code.

Claude Science ↔ R

If Claude for Life Sciences is installed locally (~/.claude-science/), checks automatically run in its r-stats-methodologist conda environment instead of system R — detected by check_rscript(), overridable with SCICOUNCIL_RSCRIPT.

This is not cosmetic. That environment ships metafor, meta and metadat, and metadat carries real published meta-analytic datasets. So a check can fit an actual random-effects model on actual trial data rather than simulating its own answer — the difference between evidence and circularity. The check prompt tells the author which packages exist and instructs it to use a real dataset when one genuinely matches, and to return NA rather than substitute simulated data when none does.

./bin/council science prints the whole picture: install path, daemon state, every specialist env with the interpreters it actually has and the roles bound to it, and the authorized connectors.

Two specialist roles are wired beyond the R stats env:

• crosscheck → chirality-data (scipy, statsmodels, patsy). science_crosscheck_ancova() re-fits an ANCOVA in statsmodels and returns the group-effect p-value. Because the checks and the statistics layer both run in R, a mis-specified model would reproduce itself rather than show up; a second implementation catches that. Verified: statsmodels reproduces this project's p-values to four decimals (0.5137, 0.2008).
• biolit → case-report-assistant (biopython/Entrez, python-docx, reportlab) and figures → figs (matplotlib) are bound and available, though the citation layer currently uses Crossref/Europe PMC directly and charts come from ggplot2.

Five of the eleven envs on a stock install are half-provisioned — the interpreter file exists but is not executable. science_env_bin() tests the execute bit rather than mere existence, so those are reported as empty instead of being routed to and failing at the point of use.

Claude's own seat is the other half of the link. Through bridge mode it brings web search and the authorized Life Sciences connectors into the debate, which no API-only panelist can do — and the bridge request now tells the serving session which connectors are live (science_connector_briefing() in R/science.R), so the Claude seat is instructed to ground claims in the user's own Drive/Microsoft 365 documents and in BioRender figures rather than reasoning from memory. The request JSON carries a connectors array listing them.

R as a statistics maker

Validation is one job; generating statistics is the other. R/stats.R + R/stats_runner.R turn a session into real analysis, executed in the r-stats-methodologist env (ggplot2 4.x, metafor, meta, metadat):

./bin/council stats-report --qid q_xxx                    # charts + HTML report
./bin/council stats-report --qid q_xxx --meta spec.json   # + real meta-analysis
./bin/council run --question "..." --stats                # generate it inline

Output lands in runs/<qid>/: four ggplot2 PNGs (consensus per claim, vote composition, panelist calibration, citation grounding) and a self-contained HTML report — PNGs base64-embedded, no external files, opens anywhere — with descriptive statistics, agreement coefficients and KPI tiles.

Pass --meta a JSON spec to fit a genuine random-effects model and draw a forest plot. Either a metadat dataset with a column mapping:

{"dataset":"dat.bcg","measure":"RR","ai":"tpos","bi":"tneg",
 "ci":"cpos","di":"cneg","slab":"author"}

or raw effect sizes: {"yi":[...],"vi":[...],"slab":[...]}. The report then carries pooled estimate, CI, p, I², τ² and the forest plot. Verified: dat.bcg (13 BCG trials) → pooled log RR −0.715 [−1.067, −0.362].

Because the stats env has no duckdb, system R exports the session via export_session_json() and the specialist env reads that JSON — the two R worlds stay cleanly separated. council_meta() in R/stats.R is callable directly for any dataset, independent of a council session.

The store

DuckDB at db/council.duckdb, FTS index on claim text (rebuilt after each run). questions → claims → (arguments, verdicts, r_checks, evidence), plus calibration and claim_links. Nothing is deleted; a claim's history is reconstructible. Query it directly for anything the CLI does not cover:

con <- DBI::dbConnect(duckdb::duckdb(), "db/council.duckdb", read_only = TRUE)
DBI::dbGetQuery(con, "SELECT claim, consensus_score FROM claims WHERE status='validated'")

Full schema and protocol rationale: references/protocol.md.

Configuration

.env next to this file, chmod 600, git-ignored. SCICOUNCIL_PANEL, per-provider keys and models, SCICOUNCIL_MAX_ROUNDS, SCICOUNCIL_CONSENSUS_THRESHOLD, SCICOUNCIL_ALLOW_RCODE. Adding a provider is a key plus a line in the panel list; xAI and DeepSeek are wired and idle, waiting for keys.

Claude Science plumbing: CLAUDE_SCIENCE_HOME (default ~/.claude-science) and SCICOUNCIL_RSCRIPT to pin a specific Rscript, overriding specialist-env routing for both checks and statistics.

Safety

• R checks execute model-written code. A separate process, a timeout, and a static deny-list (no system, no network, no file writes) make this a guardrail against accidents — not a security sandbox. Never point a session at untrusted input. Set SCICOUNCIL_ALLOW_RCODE=false to disable entirely; claims then simply go unchecked rather than failing.
• Cost. Hosted panelists bill per session. Local Ollama seats are free — prefer them for exploratory runs.
• Keep .env out of any repository. If a key ever lands in a chat or a log, rotate it.