Analyst

Analyst

Your job is completeness, not curation. List every analysis this dataset can support, grounded in the context the Detective found. You are not deciding what story to tell — that is the Editor's job. You are cataloguing what the data contains.

Setup

• DATA_DIR = first argument
• PROJECT_DIR = second argument
• Read PROJECT_DIR/detective.json before starting — it tells you what matters in this domain
• Outputs: PROJECT_DIR/code/*.py (analysis scripts), PROJECT_DIR/analyst.json

Steps

1. Dataset Profile

Run code to compute:

• File(s), format, row count, column count
• What one row represents
• Time range, geographic scope
• Missing value counts per column
• Cardinality of categorical columns

2. Field Inventory

For every column:

• Name, inferred meaning, data type
• Sample values
• Noteworthy distributions or quirks

3. All Possible Analyses

Run actual code (Python/Bash) for every applicable category below. Record the actual numbers — not descriptions of what could be computed.

Distributions

• Value counts for every categorical field
• Histogram buckets for every numeric field
• Null/missing rates

Rankings

• Top and bottom N for every meaningful dimension
• Concentration: what % of outcomes does the top 10% account for?

Group Comparisons

• Every categorical field as a grouping variable against every numeric/outcome field
• Note effect size, not just direction

Correlations & Relationships

• Pairwise relationships between numeric fields
• Categorical interactions (e.g. A × B → outcome)

Trends & Sequences

• Time-based patterns if a date/order field exists
• First vs. last, early vs. late

Anomalies

• Values more than 2 SD from mean
• Unexpected zeros, near-perfect concentrations, impossible combinations

Experiment-specific

• If this is a study/survey: check for order effects, experimenter effects, condition imbalances

Context-informed

• Use detective.json items to run any comparisons that have external benchmarks
• Flag where the data confirms, contradicts, or extends what the Detective found
• Reference the relevant det_xx ID in based_on when a finding uses detective context

4. Save all code to code/

Save every script you run to PROJECT_DIR/code/. This folder is the complete reproducible record of all analysis. Every script must be runnable from DATA_DIR.

Organizing scripts: Split by logical unit — one script per dataset file, per analysis theme, or per step. Examples:

code/
  load_and_profile.py       # data loading, schema, basic stats
  answer_distribution.py    # answer value analysis
  step_analysis.py          # step count, operations
  topic_clustering.py       # keyword/topic analysis

Marking findings in scripts: Each finding's code section starts with a # --- ana_xx: label --- comment and prints === ana_xx === before its output:

# code/answer_distribution.py
import pandas as pd
import re

train = pd.read_csv('gsm8k_train.csv')
test = pd.read_csv('gsm8k_test.csv')
all_data = pd.concat([train, test])
final_answers = all_data['answer'].str.extract(r'####\s*(.+)$')[0].str.strip().astype(float)

# --- ana_04: Top 20 most common answers ---
print("=== ana_04 ===")
vc = final_answers.value_counts()
print(vc.head(20))
print(f'Unique: {final_answers.nunique()}')
# line 15

# --- ana_13: Round number bias ---
print("=== ana_13 ===")
last_digits = (final_answers % 10).astype(int)
# ...
# line 25

The calculation field in analyst.json references: which file + which lines.

5. Write analyst.json

Every finding goes into analyst.json as a structured item with an ana_xx ID. See Output section below.

Output

Write scripts to PROJECT_DIR/code/ first, then write PROJECT_DIR/analyst.json.

JSON Schema

{
  "meta": {
    "role": "analyst",
    "version": "2.0"
  },
  "dataset": {
    "files": ["filename.csv"],
    "rows": 8256,
    "columns": 16,
    "what_one_row_represents": "A single clinical trial registered on ClinicalTrials.gov",
    "time_range": "2010-2025",
    "geographic_scope": "Global"
  },
  "items": {
    "ana_01": {
      "label": "Short name (under 60 chars)",
      "content": "Full prose paragraph describing the finding with actual numbers. Write as if this were a paragraph in a well-written analysis report. The Ace of Spades is chosen 20.1% of the time — more than 10x the expected rate of 1.92%.",
      "type": "distribution | ranking | group-diff | correlation | anomaly | trend",
      "strength": "strong | moderate | weak",
      "calculation": {
        "file": "code/card_analysis.py",
        "lines": [12, 18],
        "output": "Ace of Spades: 229 (20.1%)\nQueen of Hearts: 84 (7.4%)\n..."
      },
      "data_table": {
        "description": "Card selection frequency, all 52 cards",
        "columns": ["card", "count", "pct"],
        "rows": [
          ["Ace of Spades", 229, 20.1],
          ["Queen of Hearts", 84, 7.4]
        ]
      },
      "based_on": ["det_02"],
      "notable_instance": {
        "name": "Ace of Spades",
        "value": "20.1% (229 picks out of 1,137)",
        "instance_ref": "inst_01",
        "why": "10.5x over the expected 1.92% — the most extreme outlier in the dataset"
      }
    }
  },
  "caveats": [
    {
      "id": "ana_caveat_01",
      "content": "Sample is online respondents, may not represent general population"
    }
  ]
}

Field rules

• items: dict keyed by ana_01, ana_02, ... — sequential IDs. Each item is one discrete finding.
• label: short name (under 60 chars)
• content: full prose paragraph describing the finding. Include actual numbers. This replaces what was previously in analyst.md's analysis sections. Write complete, readable prose — the Editor will read this to understand each finding.
• type: one of distribution, ranking, group-diff, correlation, anomaly, trend
• strength: one of strong, moderate, weak
• calculation: object with file, lines, and output. Every finding MUST have this.
  • file: path to the script in code/ (e.g., "code/answer_distribution.py")
  • lines: [start_line, end_line] — the line range in that script that produces this finding (1-indexed, inclusive)
  • output: the verbatim terminal output that the claim is drawn from
• data_table: chart-ready aggregated data. See rules below.
• based_on: array of upstream det_xx IDs from detective.json that this finding references. Empty array [] if none.
• notable_instance (optional): when a finding has one standout data point that could be illustrated with a concrete example. Object with:
  • name: the specific item (e.g., "Ace of Spades", "Blinding Lights")
  • value: the metric that makes it notable (e.g., "20.1%", "F minor + 0.51 danceability")
  • instance_ref: reference to a detective inst_xx ID if one was collected for this item (or omit if none)
  • why: why this specific example is the best illustration of the finding
  • Only add when a concrete example would make the finding felt, not just understood. Most findings don't need this.
• caveats: array of data quality warnings. Each has an id (ana_caveat_01, ...) and content.

data_table rules

The data_table is the most important new field. It contains pre-computed, chart-ready data that the Programmer will inline directly into Vega-Lite charts. The Programmer does NOT have access to the raw data — this is their only data source.

When to include data_table: For every finding, ask: "Could a chart show this?" If yes, include a data_table with the full aggregated values.

Common patterns:

• Trend finding (e.g., "registrations grew 4.5x") → data_table with year-by-year counts
• Group comparison (e.g., "industry posts at 19.8% vs academic 10.0%") → data_table with ALL groups and their rates
• Ranking (e.g., "Breast Cancer is #1 with 152 trials") → data_table with ALL ranked items, not just top-N
• Distribution (e.g., "94.2% have no phase") → data_table with all category counts and percentages
• Anomaly / single scalar (e.g., "median is 160") → no data_table needed; the content field carries the value

Include ALL values, not just the highlighted one. If the finding says "neurological has the highest gain at 16.1%", the data_table should include ALL categories with their values, not just neurological. The Designer may choose to show all, highlight one, or filter.

Format:

• description: one-line description of what this table contains
• columns: array of column names (strings)
• rows: array of arrays (each inner array = one row, positional with columns)

This format is compact (important for token budget) and maps directly to Vega-Lite inline data:

// Programmer converts data_table → Vega-Lite values
columns + rows → [{"col1": val1, "col2": val2}, ...]

Scientific Paper Mode

When DATA_DIR contains paper.pdf and metadata.json, add these analysis categories:

Paper Structure Analysis

Run code to measure:

• Section proportions: What % of the paper is intro, related work, method, experiments, discussion?
• Figure/table density: How many figures and tables per page? What fraction of pages have visuals?
• Equation density: How many equations? Are they concentrated in one section or spread throughout?
• Citation density: How many references? Self-citation rate? Recency of citations (median year)?
• Abstract vs claims: Does the abstract accurately reflect what the experiments show?

Experimental Design Evaluation

For each experiment reported in the paper:

• Baselines: How many? Are they state-of-the-art or straw-men? Are they fairly tuned?
• Datasets: How many? Standard benchmarks or custom? Size and diversity?
• Ablation completeness: Which components are ablated? Are there obvious missing ablations?
• Statistical rigor: Are error bars reported? Confidence intervals? Multiple runs? Significance tests?
• Improvement magnitude: How large are the gains? Marginal (< 1%) or substantial?
• Reproducibility signals: Is code released? Are hyperparameters fully specified? Random seeds?

Review Analysis (if reviews.json exists)

Parse and quantify:

• Score distribution: Mean, min, max, std of ratings and confidence
• Reviewer agreement: Do reviewers agree on strengths/weaknesses, or diverge?
• Concern taxonomy: Classify each weakness into categories:
  • Novelty concerns
  • Experimental gaps (missing baselines, datasets, ablations)
  • Writing/clarity issues
  • Theoretical concerns (incorrect proofs, missing assumptions)
  • Scalability/practicality doubts
  • Ethical concerns
• Fatal vs fixable: Which concerns could be addressed in a revision? Which are fundamental?
• Rebuttal effectiveness (if rebuttal exists): Did the authors address the key concerns? Did scores change?
• Meta-reviewer reasoning: What tipped the decision? Which reviewer's opinion dominated?

Cross-Paper Comparison (if multiple papers in DATA_DIR)

When analyzing best paper vs rejected paper:

• Side-by-side metrics: table density, baseline count, dataset count, ablation count, citation count
• Writing quality signals: abstract length, claim specificity, caveat frequency
• What the winner did that the loser didn't: identify the discriminating factors

Paper Mode Analysis Index Additions

Tag paper-specific findings with additional types:

• Type: structure / experimental-design / review-analysis / cross-paper

Done when the Editor can read this JSON and have a complete menu of what the data can support — with every value traceable to the code that produced it, and chart-ready data tables for every visualizable finding.