Omics dataset retrieval

Omics Dataset Retrieval

Build a deduplicated, relevance-audited catalog of public omics datasets for a disease, phenotype, gene, or biological process. Query every reachable public repository, classify each hit by omics type, score its relevance, and emit a CSV catalog + Markdown summary (+ optional landscape figure).

Boundary: catalog metadata only. This skill does not download raw data files and does not perform downstream analysis.

Worked examples (Alzheimer's, BCL11A/erythroid, PDAC) are in references/examples.md.

Inputs

Parameter	Type	Description
disease_or_topic	string	Disease, phenotype, gene, or process (e.g. "sickle cell disease", "BCL11A")
synonyms	list[str]	Alternative names, abbreviations, gene symbols (e.g. ["SCD","SCA","HbSS"])
omics_types	list[str] or "all"	Restrict to omics types, or "all" (default)
organism	string	"all" (default) — human, mouse, and all others. Restrict to "human"/"mouse" only if explicitly requested
year_min	int	Earliest publication year (default: no limit)
output_dir	path	Where to write outputs (default: /mnt/results/)

Outputs

File	Description
<disease>_omics_datasets_MASTER.csv	Full catalog with all metadata + relevance labels
<disease>_omics_datasets_VALIDATED.csv	Filtered to CORE/DIRECT + ADJACENT only
<disease>_omics_summary.md	Counts by omics type / repository, top datasets, limitations
<disease>_omics_landscape.png	Optional overview: donut + repository bar + timeline

Workflow

Step 1 — Ask clarification questions (MANDATORY before any search)

Not optional. Before querying, use AskUserQuestion to collect the items below in a single call. Skip only questions already answered in the user's initial message; always ask the rest. Answers materially change which repositories run, result counts, and relevance tuning.

• Q1 Disease/topic — name; synonyms/abbreviations (more = better recall); key genes/pathways (drive extra targeted GEO queries).
• Q2 Omics types — all (default) / transcriptomics only / epigenomics only / proteomics+metabolomics only / custom.
• Q3 Organism — all (default) / human only / specific species.
• Q4 Year range — no restriction (default) / recent only / custom.
• Q5 Controlled-access repos (dbGaP, EGA, JGA) — include+flag (default) / open-access only.
• Q6 Goal — browse & select (default) / landscape overview (figure) / download & analyze (add links) / all.
• Q7 Tissue/cell-type focus (optional, big recall boost) — e.g. "whole blood", "brain cortex", "CD34+ HSCs". Added as targeted terms.

Proceed to Step 2 only once answers collected.

Step 2 — Build search term matrix

Combine: primary name + synonyms × omics-type terms × tissue/cell terms × clinical-context terms (treatment, crisis, biomarker, pediatric, longitudinal) × key gene/pathway terms.

GEO syntax tips: "<disease>"[Title] for precision; [DataSet Type] for omics filtering; retmax=100 per query; run 20–40 queries; dedupe by GEO UID.

Step 3–12 — Query repositories

Work through all tiers in priority order — rare diseases often only have data in Tier 2/3. Full API base URLs, per-repository query code, and quirks are in the reference files:

• references/repositories.md — Tier 1/2/3 coverage map + known API limitations + workarounds.
• references/query-code.md — runnable query snippets per repository (GEO, SRA, ArrayExpress/BioStudies, PRIDE, jPOST, OmicsDI, Metabolomics Workbench, CELLxGENE, GDC/TCGA, ENCODE, Expression Atlas, Tier-3 web search).

Tier 1 (always query): GEO, SRA, ArrayExpress/BioStudies, PRIDE/ProteomeXchange, OmicsDI, CZ CELLxGENE, GDC/TCGA. Tier 2 (when relevant): ENCODE, Expression Atlas, HCA, Metabolomics Workbench, MetaboLights, MassIVE/GNPS, jPOST, iProX, cBioPortal, EpiRR/IHEC, HPA, ENA. Tier 3 (WebSearch tool): Zenodo, Figshare, Dryad, OSF, Harvard Dataverse, Synapse, ICGC, CPTAC, AWS Open Data, dbGaP/EGA/JGA (controlled), UK Biobank, FinnGen.

GEO is the primary source — run 20–40 targeted queries. Use OmicsDI as the single entry point for metabolomics (aggregates MetaboLights, MassIVE, GNPS, Metabolomics Workbench). Only query GDC/TCGA/CPTAC/cBioPortal for cancer topics; only ENCODE when the topic has epigenomic/regulatory components.

Step 13 — Classify omics type

Apply rule-based classify_omics() to each record. The GEO gdstype field is unreliable — always use Title + Summary as the primary signal (check single-cell/spatial before bulk RNA-seq; ArrayExpress labels arrays as "transcription profiling by array"). Function in references/classification.md.

Step 14 — Relevance audit (4 tiers)

Score each record with audit_relevance() using disease/weak/exclude keyword lists tuned per topic:

Tier	Meaning	Action
CORE/DIRECT	Real patient samples or validated disease model	Primary catalog
ADJACENT	Mechanistically relevant, no patient data	Include, flagged
WEAK	Epidemiologically linked / tangential	Include, flagged; user decides
REMOVE	False positive, no disease connection	Exclude — always manually review before deleting

Function + example keyword lists in references/classification.md.

Step 15 — Assemble master catalog

Concat all sources (list df_geo before df_biostudies so richer native GEO records win dedup). Deduplicate across repositories by normalizing accessions: E-GEOD-NNN → GSENNN, so GEO/ArrayExpress mirrors collapse while genuine ArrayExpress studies (E-MTAB-, E-MEXP-, E-TABM-) survive. Do NOT pre-filter E-GEOD- accessions at query time — that silently drops studies GEO missed. Backfill authoritative organism for CORE/ADJACENT ArrayExpress rows from /studies/{accession}. Add Access column (Controlled for dbGaP/EGA/JGA/UK Biobank/FinnGen). Sort by relevance tier then year desc. Assembly code in references/classification.md.

Step 16 — Markdown summary report

Write <disease>_omics_summary.md: overview (totals, repos searched, date); counts by repository; counts by omics type (CORE/DIRECT only); relevance breakdown; top datasets (largest N, most recent); controlled-access list with access instructions; coverage gaps and limitations (which APIs failed / need keys / are data-sparse for this topic).

Step 17 — Overview figure (if requested)

3-panel <disease>_omics_landscape.png — Panel A donut (CORE/DIRECT by omics type), Panel B horizontal bar (all by repository), Panel C stacked-bar timeline (datasets/year by omics type). After saving, verify render with Read(file_path=..., mode="media_output_check"). Plot boilerplate in references/query-code.md.

Key scientific caveats

• GEO gdstype is unreliable — reclassify from Title + Summary. ChIP/ATAC/ Ribo-seq are often mislabeled; some RNA-seq mislabeled "Hi-C".
• Cross-repository duplicates are common — same study in GEO + ArrayExpress + SRA + OmicsDI. Normalize E-GEOD- → GSE before dedup (Step 15).
• Relevance audit is keyword-based — produces false positives and negatives. Always manually review REMOVE candidates before excluding.
• Controlled-access (dbGaP/EGA/JGA) — catalog and label Controlled; download needs institutional data-access agreements.
• Metabolomics is data-sparse for rare diseases — check Metabolomics Workbench
  • HMDB disease pages manually if OmicsDI returns nothing.
• CELLxGENE uses MONDO ontology — look up the correct MONDO ID before filtering cell metadata.
• NCBI rate limits — 3 req/sec without key, 10 with. time.sleep(0.34) between calls; set api_key param to raise the limit.