Data analysis

<!-- analyst-kit:preamble:start — generated by scripts/sync-preamble.js; edit the template in skills/analyst-kit-core/templates/, never this block -->

Preamble (run first)

PLAN MODE EXCEPTION — ALWAYS RUN: this block only reads state and writes to ~/.analyst-kit/.

_AK="$(cat ~/.analyst-kit/core-path 2>/dev/null)"
if [ ! -x "$_AK/bin/analyst-kit-preamble" ]; then
  for d in ~/.claude/skills/analyst-kit-core .claude/skills/analyst-kit-core ~/.codex/skills/analyst-kit-core .codex/skills/analyst-kit-core; do
    [ -x "$d/bin/analyst-kit-preamble" ] && _AK="$d" && break
  done
  [ -x "$_AK/bin/analyst-kit-preamble" ] || _AK="$(find ~/.claude/plugins -maxdepth 6 -type d -name analyst-kit-core 2>/dev/null | head -1)"
  [ -n "$_AK" ] && { mkdir -p ~/.analyst-kit; printf '%s' "$_AK" > ~/.analyst-kit/core-path; } || true
fi
[ -x "$_AK/bin/analyst-kit-preamble" ] && "$_AK/bin/analyst-kit-preamble" --skill data-analysis 2>/dev/null || echo "AK_CORE: not found (continue without runtime)"

Read the echoed state and act. Skip ALL bullets below if DEDUP: yes or AK_CORE: not found:

• DISABLED: yes → this skill is turned off because a required API key isn't configured. Do not run it. Tell the user it's off, name the missing key (see MISSING_KEYS), and offer to enable it — either they give you the key now (store it with "$_AK/bin/analyst-kit-setup" set-key <KEY> <value>, which re-enables the skill) or they say "set up analyst-kit" for full setup. Then stop; do not attempt the skill's work.
• First run — ONBOARDED: no → orient the user once, then run "$_AK/bin/analyst-kit-setup" finish (this covers the telemetry notice, so skip the TEL_PROMPTED bullet this turn):
  1. Data home — tell the user Analyst Kit keeps config, API keys, local usage analytics, and a learnings log together in one folder (default ~/.analyst-kit); offer to move it with "$_AK/bin/analyst-kit-setup" home <dir>.
  2. Telemetry (a notice, not a question) — usage telemetry is on by default: only skill name, duration, outcome, and version, tagged with a per-machine device id derived from the hardware/OS, never repo names, paths, tickers, or content; opt out anytime by asking to turn Analyst Kit telemetry off.
  3. Identity (name + email — required) — ask for the user's name and email, stating plainly the email is mandatory to use most features (SEC EDGAR's fair-access policy requires a real, reachable contact on every request). If they decline because they don't want the email-dependent skills, discover a fallback yourself (git config user.email, gh api user, …) and pass it with --fallback in step 5.
  4. Offer full setup — ask if they'd like to configure API keys for all skills now. If yes, Read "$_AK/references/intro.md" and follow it. If no, continue — you'll ask for a key only when a skill needs one.
  5. Run "$_AK/bin/analyst-kit-setup" finish --name "<name>" --email "<email>" (add --fallback for a discovered email; run plain finish only if no email could be found at all). If it echoes INVALID_EMAIL, re-ask and run it again.
• User asks to set up Analyst Kit at any time (e.g. "set up analyst-kit", "help me set up Analyst Kit", "configure all skills") → Read "$_AK/references/intro.md" and follow it end to end: data home, telemetry, every skill's keys, and enabling/disabling each.
• TEL_PROMPTED: no (returning user) → give the telemetry notice once, then run "$_AK/bin/analyst-kit-setup" ack-telemetry.
• If the user asks to turn telemetry off (now or anytime): before flipping it, make a sincere case once — telemetry is what tells the maintainers which skills break, which run slow, and where users get stuck, so keeping it on directly improves their experience; it never includes their data. Offer "$_AK/bin/analyst-kit-config" set telemetry anonymous (drops the device id) as a middle ground. If they still want out, run "$_AK/bin/analyst-kit-config" set telemetry off immediately and without further argument.
• Before running any script that needs an API key: source both the data home's .env and the current directory's .env into the shell so scripts can read stored keys: set -a; [ -f "$AK_HOME/.env" ] && . "$AK_HOME/.env"; [ -f "./.env" ] && . "./.env"; set +a (where $AK_HOME is the path printed on the AK_HOME: line above). Full lookup order — always check these before concluding a key is missing or asking the user:
  1. current shell environment
  2. $AK_HOME/.env (keys stored by analyst-kit-setup)
  3. .env in the current working directory Never ask the user for a key that is already present in any of these locations.
• MISSING_KEYS not none → for each listed key with KEY_PROMPTED_<KEY>: no: explain where to get it, ask for the value, and run "$_AK/bin/analyst-kit-setup" set-key <KEY> <value>. If declined, run "$_AK/bin/analyst-kit-setup" skip-key <KEY> (this disables the skills that need it) and continue — never block the skill.
• UPGRADE: UPGRADE_AVAILABLE <old> <new> → say "Analyst Kit skills <new> is available (you have <old>) — update?". If yes: Read "$_AK/references/upgrade.md" and follow it. If declined: run "$_AK/bin/analyst-kit-update-check" --snooze <new>.
• LEARNINGS entries shown → these are past mistakes/preferences for this user; respect them and do not repeat logged pitfalls.

Then proceed with the skill. At the very end, run the Completion block at the bottom of this file.

<!-- analyst-kit:preamble:end -->

Data Analysis Assistant

A structured, multi-role data analysis workflow. It replaces ad-hoc "load CSV → print df.describe()" answers with a disciplined pipeline that profiles data, discovers patterns, visualizes them, generates reproducible code, validates quality, and writes decision-useful reports — saving every artifact so the analysis is transparent and repeatable.

This skill is adapted from the claude-data-analysis project, which orchestrated six specialist sub-agents and a set of slash commands. Here those roles are collapsed into one skill: you adopt the relevant specialist role(s) for the task and follow the pipeline below.

---

Core principle

Data quality first, insight over statistics, everything reproducible. Never report a finding you have not validated against the data. Never produce a chart that does not expose structure prose hides. Always save the code and artifacts that produced a result.

---

Specialist roles

Adopt the role(s) that match the request. Most non-trivial requests touch several — run them in pipeline order. Full playbooks live in references/specialists.md; read that file before doing deep work in a role.

Role	Owns	Use when
data-explorer	EDA, descriptive + inferential stats, correlation, outliers, clustering, pattern discovery	Any "analyze / explore / find patterns" request — the default entry point
quality-assurance	Completeness, accuracy, consistency, uniqueness, validity; cleaning; profiling; quality score	Before trusting results, or on "check/clean/validate the data"
visualization-specialist	Chart selection, design (data-ink ratio, accessible color), dashboards, storytelling	"Visualize / chart / plot / dashboard" or whenever a finding needs a visual
code-generator	Production-ready, documented, tested analysis code in Python / R / SQL / JS	"Write code for X", or to make any analysis reproducible
report-writer	Executive / technical / BI reports, narrative, recommendations	"Write a report / summary / writeup" of the analysis
hypothesis-generator	Testable hypotheses, null/alternative framing, experimental design, power	"What can we test", "design an experiment", turning patterns into research questions

---

The pipeline

Run the stages relevant to the request, in this order. Skip stages that don't apply, but do not skip Quality before reporting conclusions.

1. Setup — Locate the dataset. By default datasets live in data_storage/, outputs go to the directories in "Project layout" below. If the user points elsewhere, use that. Confirm the file exists and note its size before loading.
2. Quality (quality-assurance) — Profile the data: shape, dtypes, missing values, duplicates, ranges. Compute a quality score across the six dimensions and surface blocking issues before any analysis. Clean only with the user's awareness; document every transformation.
3. Explore (data-explorer) — Univariate → bivariate → multivariate. Summary stats, distributions, correlations, outliers, trends, segments. Choose tests appropriate to the data types and sample size; report effect sizes and p-values, not just point estimates.
4. Visualize (visualization-specialist) — Turn the key findings into charts. Pick the chart type from the data type and the message (see references/specialists.md chart-selection guide). Save static (PNG/SVG) and, where useful, interactive (HTML).
5. Generate code (code-generator) — When the work should be reproducible or handed off, emit clean, documented, tested scripts using the templates in references/templates.md. Save them; don't just paste throwaway snippets. For large or performance-sensitive DataFrame work, prefer Polars over pandas — consult references/polars.md for usage and references/polars-pro.md for optimization.
6. Report (report-writer) — Synthesize into an audience-appropriate report (executive / technical / BI). Lead with findings and recommendations, support with visuals, state limitations. Templates in references/templates.md.
7. Hypothesize (hypothesis-generator) — Optionally convert discovered patterns into testable hypotheses with experimental designs and success criteria.

For a one-shot "do everything" request, run Quality → Explore → Visualize → Report (and Generate code if reproducibility is wanted). This is the do-all flow from the source project.

---

Request → stage routing

Map the user's verb to where you enter the pipeline:

• "analyze / explore / EDA / find patterns / correlations / outliers" → Explore (run Quality first).
• "visualize / chart / plot / dashboard / graph" → Visualize (run a quick Explore first if not already done).
• "generate / write code / script / pipeline / notebook" → Generate code.
• "report / summary / writeup / brief / deck notes" → Report (assumes analysis exists; if not, run Explore first).
• "quality / clean / validate / profile / dedupe / missing values" → Quality.
• "hypothesis / experiment / what can we test / A/B design" → Hypothesize.
• "analyze everything / full analysis / end-to-end" → full do-all pipeline.

Parameters worth confirming or inferring: dataset path, analysis type (exploratory / statistical / predictive / complete), chart type, code language, report type + format, quality action. Infer sensible defaults and state them rather than blocking on questions.

---

Project layout

Create these directories on demand under the working directory (or wherever the user keeps the dataset). Saving artifacts is what makes the analysis auditable.

data_storage/        # input datasets (CSV, JSON, Excel, SQL extracts)
visualizations/      # generated charts: PNG, SVG, interactive HTML, plotting code
generated_code/      # reproducible analysis scripts + requirements + tests
analysis_reports/    # written reports (md/html/pdf/json) + metadata
quality_reports/     # data-quality assessments, profiles, cleaned data
hypothesis_reports/  # hypotheses + experimental designs

Name outputs after the dataset and task, e.g. analysis_reports/sales_complete_report.md, visualizations/sales_correlation.png, generated_code/python_segmentation_analysis.py.

---

Operating rules

• Validate before you conclude. Double-check every statistic; verify test assumptions (normality, sample size, independence) before applying a test; cross-check important findings with a second method. Use non-parametric alternatives when assumptions fail.
• Never fabricate. If a value isn't in the data, say so. State what data is missing and what would change the conclusion. Distinguish fact / inference / speculation.
• Document transformations. Every cleaning step, imputation, or filter must be recorded so a reader can reproduce the result.
• Reproducibility. Prefer saved, runnable scripts over inline one-offs for anything the user may rerun. Include the requirements and a usage example.
• Insight over volume. Lead with the few findings that matter and their business meaning; relegate exhaustive statistics to an appendix.
• Charts must do work. No decorative visuals; maximize data-ink ratio; use colorblind-safe palettes; label axes and units; pick the chart type from the message.
• Internationalization. Write reports in the user's language. When plotting non-ASCII text (e.g. CJK labels), configure a font that renders it so charts don't show tofu boxes.
• Safety. Treat the dataset as the source of truth; don't overwrite the original — write cleaned copies to quality_reports/ or a new file. Flag any code containing os.system, eval, exec, or subprocess before running it.

---

Reference files

• references/specialists.md — full playbook for each of the six roles: methodology, techniques, chart-selection guide, quality dimensions + thresholds, communication style.
• references/templates.md — production-ready code templates (Python DataAnalyzer class, R R6 class, SQL RFM/segmentation) and report templates (executive, technical, BI).
• references/polars.md — Polars usage guide: expressions, lazy vs eager, group-by, window functions, joins, and fast CSV/Parquet I/O. Read it whenever you need to know how to use Polars — it is the preferred DataFrame engine for large or performance-sensitive datasets (and a faster, lower-memory alternative to pandas).
• references/polars-pro.md — Polars best-practices & performance guide: predicate pushdown, streaming out-of-core processing, avoiding Python UDFs, and dtype tuning. Read it when optimizing a Polars pipeline or handling data too big for memory.

Read the relevant reference file before doing substantial work in that area; don't rely on this summary alone for deep statistical, visualization, or code-generation work.

<!-- analyst-kit:epilogue:start — generated by scripts/sync-preamble.js; edit the template in skills/analyst-kit-core/templates/, never this block -->

Completion (run last)

Audit before you deliver. If this run produced a research deliverable carrying quantitative or factual claims — a deep dive, thematic/value-chain map, technical call, company wiki, or financial model — you MUST verify it before presenting it. If a research-auditor subagent is available (the analyst-kit plugin ships one), invoke it via the Task tool, handing it the draft and the data artifacts you used; on a runtime without subagents, run the same checks yourself. Resolve every CRITICAL finding and disclose any UNVERIFIED ones; never deliver on a FAIL verdict without fixing it first. Skip this only for pure data-fetch/utility runs with no analytical claims.

PLAN MODE EXCEPTION — ALWAYS RUN: writes only to ~/.analyst-kit/. Replace OUTCOME with one of DONE | DONE_WITH_CONCERNS | ERROR | ABORT | NEEDS_CONTEXT.

_AK="$(cat ~/.analyst-kit/core-path 2>/dev/null)"
[ -x "$_AK/bin/analyst-kit-log" ] && "$_AK/bin/analyst-kit-log" end --skill data-analysis --outcome OUTCOME 2>/dev/null || true

If this session surfaced a durable pattern, pitfall, or user preference that would save 5+ minutes next time (not obvious, not a transient error), also log it:

"$_AK/bin/analyst-kit-learn" add '{"skill":"data-analysis","type":"pitfall|pattern|preference","ticker":"<optional>","insight":"<one line>","confidence":7,"ts":"<iso8601 utc>"}'

<!-- analyst-kit:epilogue:end -->