Benchmark table generation and reporting

benchmark-table-generation-and-reporting

Summary

Systematic execution and compilation of computational performance metrics (wall-clock time) across multiple datasets, sample sizes, and analysis configurations to generate a reproducible benchmark table. This skill ensures consistent measurement of pipeline performance across real and synthetic datasets and validates reported execution times against published benchmarks.

When to use

When you need to quantify and document the computational cost of a multi-stage analysis pipeline (such as MetaboDirect) across varying input sizes (e.g., 40 vs. 120 samples) and optional feature sets (main pipeline only vs. with KEGG annotation vs. full analysis with transformation networks). Use this skill to populate a summary table with elapsed times for publication or to validate performance claims in an existing paper.

When NOT to use

• When you are profiling memory usage or CPU utilization rather than total execution time—use a dedicated profiler (e.g., cProfile, htop) instead.
• When your input datasets are not yet assigned molecular formulas or are not in Formularity .csv format—preprocess them first using appropriate formula-assignment software.
• When you need to compare performance across fundamentally different tools (e.g., MetaboDirect vs. MetaboAnalyst vs. UltraMassExplorer)—this skill is designed for benchmarking a single pipeline across conditions, not for cross-tool comparison.

Inputs

• Mock FT-ICR MS datasets (40 and 120 samples, generated by random subsampling)
• Real FT-ICR MS datasets in Formularity .csv format (assigned molecular formulas, peak intensities, m/z values)
• MetaboDirect pipeline (v0.3.4 or later)
• System environment specifications (OS, Python version, R version, Cytoscape version)

Outputs

• Benchmark summary table (TSV or CSV format) with columns: dataset name, sample count, analysis configuration, pipeline scope, elapsed time (minutes)
• Wall-clock time measurements for each pipeline execution
• Consistency validation report comparing recorded times to published benchmarks

How to apply

Run the target pipeline (e.g., MetaboDirect v0.3.4) on each test dataset using a single standardized command, recording elapsed wall-clock time for each invocation. Test both mock datasets (generated by random subsampling of real data) and real FT-ICR MS datasets in standardized input format (.csv with assigned molecular formulas, peak intensities, m/z values). For each dataset, execute the main pipeline (six steps: data pre-processing, diagnostics, exploration, chemodiversity, statistics, transformation networks) and optional enhanced configurations (KEGG-annotated, full analysis). Compile all recorded times into a structured table with columns for dataset name, sample count/configuration, pipeline scope, and elapsed time in minutes. Verify consistency between recorded times and published benchmark values—deviations may indicate environment differences (OS, hardware, dependencies) or version changes.

Related tools

• MetaboDirect (Target pipeline to be benchmarked; executes six-step workflow (preprocessing, diagnostics, exploration, chemodiversity, statistics, transformation networks) on FT-ICR MS data) — https://github.com/Coayala/MetaboDirect
• Python (Orchestration and timing measurement; used to record wall-clock time and compile results)
• KEGGREST (Optional R package invoked by MetaboDirect to query KEGG database for pathway/module annotation; adds computational overhead)
• Formularity (Input data format standard: .csv files with assigned molecular formulas, peak intensities, and m/z values)

Examples

metabodirect --input bacterium_phage_36samples.csv --kegg --transformation-networks > benchmark_run.log 2>&1 & time wait $!

Evaluation signals

• All test datasets execute without errors and produce valid output (Van Krevelen diagrams, chemodiversity metrics, transformation networks as applicable).
• Recorded wall-clock times are internally consistent across repeated runs on the same dataset (within ±5% variance, accounting for system load).
• Recorded times align with published benchmark values in the paper (bacterium-phage 36 samples: ~36 sec main, ~10 min with KEGG, ~21 min full; S. fallax 4 samples: 30 sec main, 32 min full).
• Time ratios are sensible: KEGG annotation and transformation network steps add clear, measurable overhead; larger sample counts (120 vs. 40) show proportional increases.
• Benchmark table is complete and includes all required columns (dataset, sample count, configuration, scope, time) with no missing entries.

Limitations

• Wall-clock time measurements are environment-dependent: OS (Windows, Linux, macOS), hardware (CPU, RAM, disk I/O), Python/R/Cytoscape versions, and concurrent system load all affect reported times. Benchmarks are most comparable when run on similar hardware and OS.
• MetaboDirect's transformation network analysis step is computationally expensive and scales nonlinearly with peak count; full analysis on large, complex datasets (e.g., 1793 average formulas per sample) may exceed acceptable runtime (S. fallax dataset: 32 minutes for 4 samples).
• KEGG database queries via KEGGREST add variable overhead depending on network latency and database size; reported times may differ if KEGG is unavailable or if queries are performed offline vs. online.
• The skill measures total pipeline execution time but does not isolate the cost of individual workflow steps (data pre-processing, diagnostics, exploration, etc.); fine-grained profiling requires instrumentation within MetaboDirect code.

Evidence

• [other] Run the main MetaboDirect pipeline (six steps: data pre-processing, data diagnostics, data exploration, chemodiversity analysis, statistical analysis, and transformation network analysis) on each of the 40-sample and 120-sample mock datasets using a single command, recording elapsed wall-clock time.: "Run the main MetaboDirect pipeline (six steps: data pre-processing, data diagnostics, data exploration, chemodiversity analysis, statistical analysis, and transformation network analysis) on each of"
• [other] Compile all recorded compute times into a summary table with columns for dataset name, sample count / configuration, pipeline step scope, and elapsed time in minutes, and verify that reported times are consistent with the paper's benchmark table.: "Compile all recorded compute times into a summary table with columns for dataset name, sample count / configuration, pipeline step scope, and elapsed time in minutes, and verify that reported times"
• [other] MetaboDirect processed 40 samples in <1 minute and 120 samples in 2 minutes for main pipeline steps; bacterium-phage dataset (36 samples, 495 avg peaks) completed in ~36 seconds (main), 10 minutes (with KEGG), and 21 minutes (full analysis): "MetaboDirect processed 40 samples in <1 minute and 120 samples in 2 minutes for main pipeline steps; bacterium-phage dataset (36 samples, 495 avg peaks) completed in ~36 seconds (main), 10 minutes"
• [other] Obtain or generate mock datasets with 40 and 120 samples by random subsampling from unpublished data, and obtain real FT-ICR MS datasets from the bacterium-phage system (Pseudoalateromonas with phages HP1 and HS2) and S. fallax leachate experiment, each in Formularity .csv format (assigned molecular formulas, peak intensities, m/z values).: "Obtain or generate mock datasets with 40 and 120 samples by random subsampling from unpublished data, and obtain real FT-ICR MS datasets from the bacterium-phage system (Pseudoalateromonas with"
• [other] Install MetaboDirect (v0.3.4) via Python Package Index with dependencies NumPy, pandas, seaborn, py4cytoscape, and matplotlib, ensuring compatibility on the target OS (Windows, Linux, or MacOS).: "Install MetaboDirect (v0.3.4) via Python Package Index with dependencies NumPy, pandas, seaborn, py4cytoscape, and matplotlib, ensuring compatibility on the target OS (Windows, Linux, or MacOS)."
• [readme] MetaboDirect can be installed directly from PyPi using: pip install metabodirect: "MetaboDirect can be installed directly from PyPi using: pip install metabodirect"