Data summary statistics

data-summary-statistics

Summary

Compute and report frequency distributions and summary counts across categorical bins to characterize the composition and coverage of a large structure-organism dataset. This skill validates data integrity by comparing observed bin membership against reference gold-standard counts.

When to use

Apply this skill when you have a curated relational dataset (structure-organism pairs) and need to quantify how structures distribute across a categorical variable (e.g., organism prevalence). Use it as a validation checkpoint to detect data loss, miscuration, or processing errors by comparing bin counts against published benchmarks.

When NOT to use

• Input is a pre-computed summary table or aggregated report (you would be double-summarizing).
• The analysis goal is to identify which specific structures or organisms are anomalous rather than to validate global bin distributions.
• Organism prevalence counts have not been computed or validated upstream; use organism-count binning as a prerequisite first.

Inputs

• Flat-file table of structure-organism pairs (TSV or TSV.GZ format)
• Column defining unique structure identifier (SMILES or InChI)
• Column defining organism association for each pair
• Reference gold-standard bin counts (from published dataset version)

Outputs

• Summary table with four rows (one per frequency bin) and columns for bin label, count, and discrepancy flag
• Text or markdown report documenting bin membership and any divergence from reference counts
• Validation flag (pass/fail) indicating whether observed counts match reference within tolerance

How to apply

Load the flat-file table of structure-organism pairs (e.g., from LOTUS as .tsv.gz). Group all unique 2D structures by their organism count (the number of distinct organisms in which each structure appears). Define four frequency bins: singleton structures (1 organism), low-diversity structures (2–10 organisms), medium-diversity structures (11–100 organisms), and high-diversity structures (>100 organisms). Count the number of unique structures falling into each bin. Generate a summary report tabulating bin membership counts and compute any numeric discrepancies between observed counts and the gold-standard reference counts reported in the literature or prior dataset releases. Use R or Python (pandas, groupby, value_counts) to aggregate and validate.

Related tools

• R (Execute grouping, binning, and count aggregation; generate summary reports) — https://www.r-project.org/
• Python 3 (Load TSV/GZ files, perform pandas groupby and value_counts operations to bin and count structures) — https://www.python.org/
• lotus-processor (Source repository containing validated LOTUS data tables and example curation/validation workflows) — https://github.com/lotusnprod/lotus-processor

Examples

python3 -c "import pandas as pd; df = pd.read_csv('lotus_pairs.tsv.gz', sep='\t', compression='gzip'); bins = df.groupby(df.groupby('structure_id').size().rename('org_count')).size(); print(f'Singleton: {bins[1]}, Low (2-10): {bins[2:11].sum()}, Medium (11-100): {bins[11:101].sum()}, High (>100): {bins[101:].sum()}')"

Evaluation signals

• Observed bin counts for all four frequency categories (1, 2–10, 11–100, >100 organisms) are non-zero and match published LOTUS gold-standard counts to within ±1–2% (accounting for rounding and incremental updates).
• Total count of unique 2D structures across all bins matches the expected curated structure count (e.g., 153956 unique 2D structures for LOTUS).
• No structure appears in more than one bin; bin membership is mutually exclusive and exhaustive.
• Summary report is human-readable, clearly labeled by bin, and includes numeric discrepancy values flagged when observed ≠ reference.
• All input rows are accounted for in the output; no structures are dropped during grouping or binning.

Limitations

• Binning thresholds (1, 2–10, 11–100, >100) are fixed and may not suit datasets with very different organism prevalence distributions.
• The skill depends on accurate upstream organism counting; errors in organism deduplication upstream will propagate into bin counts.
• Comparison to gold-standard reference counts assumes the reference is correct and applicable to the current dataset version; dataset growth or curation changes will render older benchmarks obsolete.
• The skill does not identify which structures or organisms are anomalous—only whether the global distribution is as expected.

Evidence

• [other] Group all unique 2D structures by their associated organism count. Bin structures into four frequency bins: singleton structures (1 organism), low-diversity structures (2–10 organisms), medium-diversity structures (11–100 organisms), and high-diversity structures (>100 organisms).: "Group all unique 2D structures by their associated organism count. Bin structures into four frequency bins: singleton structures (1 organism), low-diversity structures (2–10 organisms),"
• [other] Count the number of structures in each bin and compare against the reported gold-standard counts.: "Count the number of structures in each bin and compare against the reported gold-standard counts."
• [other] LOTUS is a comprehensive collection of documented structure-organism pairs designed to enable computational understanding of organisms and their chemistry.: "LOTUS is a comprehensive collection of documented structure-organism pairs designed to enable computational understanding of organisms and their chemistry."
• [other] 231330 | 153956 (3D|2D) unique curated structures: "231330 | 153956 (3D|2D) unique curated structures"
• [other] Load the LOTUS 2D structure-organism pairs table from the published flat file.: "Load the LOTUS 2D structure-organism pairs table from the published flat file."