Tabular data field comparison

Tabular Data Field Comparison

Summary

Compare structured tabular outputs (CSV, Parquet) exported from independent implementations of the same file reader to detect field-level discrepancies in names, types, row counts, and numerical values. This skill validates implementation interoperability and identifies divergences in data interpretation across language/library variants.

When to use

Apply this skill when you have multiple independent implementations of the same data format reader (e.g., Rust, Python, R versions) and need to verify they produce identical or equivalent output. Use it to validate that a newly written reader implementation (in one language) produces the same spectrum table structure and numerical content as an existing trusted implementation.

When NOT to use

• Input is a single implementation or vendor output; comparison requires at least two independent readers.
• Field definitions or data types are intentionally different across implementations by design (e.g., different precision levels for intentional lossy compression).
• Input file format is already validated or certified by a reference standard; this skill is for early-stage interoperability testing, not final certification.

Inputs

• mzPeak file (ZIP archive containing Parquet files)
• Three or more independent reader implementations (Rust library, Python pyarrow implementation, R arrow implementation)

Outputs

• CSV export of spectrum metadata table from each implementation
• CSV export of spectrum signal data table from each implementation
• Consistency report documenting field-level agreement and discrepancies
• Validation status summary (pass/fail per implementation pair)

How to apply

Load the same input file (e.g., an mzPeak archive) sequentially using each independent implementation—Rust, Python/pyarrow, and R/arrow—and export the resulting spectrum metadata or signal tables to CSV format for easy textual comparison. Compare the three CSV outputs systematically: first check schema agreement (field names, declared data types, row counts match), then validate numerical equality field-by-field, paying attention to floating-point precision and integer representation. Generate a structured consistency report documenting which fields agree, which diverge, and whether divergences are due to implementation bugs, format interpretation differences, or acceptable numerical tolerances. Use field-level agreement as the primary validation criterion, since row count and schema matching alone are insufficient to confirm correctness.

Related tools

• Rust mzPeak library (Primary read/write implementation used to load test file and export reference spectrum table) — https://github.com/HUPO-PSI/mzPeak
• pyarrow (Python Parquet reader for loading mzPeak table files and exporting to CSV) — https://arrow.apache.org/docs/python/index.html
• R arrow (R Parquet reader for loading mzPeak table files and exporting to CSV) — https://arrow.apache.org/docs/r/
• OpenMS (Reference mass spectrometry toolkit; mzPeak name held in trust by OpenMS Inc.)

Evaluation signals

• Field names match exactly across all three CSV exports (including order and case sensitivity).
• Data types inferred from CSV content (integer, float, string, null) agree across implementations for each column.
• Row counts are identical across all three CSV outputs.
• Numerical values in floating-point columns agree within an acceptable precision tolerance (e.g., within machine epsilon or a domain-specific threshold like 1e-6 relative error).
• No spurious nulls or NaN values appear in one implementation but not others (unless intentional per format design, such as null marking for zero-run-stripped regions).

Limitations

• The mzPeak format is work-in-progress with no stability guarantee at this point; field definitions or semantics may change.
• Python and R implementations support reading only; if testing write capability, only Rust can produce reference outputs.
• Floating-point precision differences (e.g., f32 vs. f64 representation) may introduce small numerical divergences that are valid but require tolerance tuning.
• The skill does not validate semantic correctness (e.g., whether a m/z value makes physical sense); it only checks consistency across implementations.

Evidence

• [other] Do the Rust, Python/pyarrow, and R/arrow implementations of mzPeak file readers produce field-level agreement when loading the same input file?: "Do the Rust, Python/pyarrow, and R/arrow implementations of mzPeak file readers produce field-level agreement when loading the same input file?"
• [other] Compare the three CSV outputs field-by-field to identify any discrepancies in field names, data types, row counts, and numerical values.: "Compare the three CSV outputs field-by-field to identify any discrepancies in field names, data types, row counts, and numerical values."
• [other] Three independent mzPeak reader implementations exist: a Rust library, a Python implementation using pyarrow, and an R implementation using arrow, all capable of reading mzPeak files.: "Three independent mzPeak reader implementations exist: a Rust library, a Python implementation using pyarrow, and an R implementation using arrow, all capable of reading mzPeak files."
• [readme] The primary work shown here is written in Rust at the repository root, including a library for reading and writing mzPeak files, as well as command line tools for converting existing formats into mzPeak.: "The primary work shown here is written in Rust at the repository root, including a library for reading and writing mzPeak files, as well as command line tools for converting existing formats into"
• [readme] There is a separate Python implementation in python/ which is a complete re-implementation for reading mzPeak files using pyarrow, and the PyData stack.: "There is a separate Python implementation in python/ which is a complete re-implementation for reading mzPeak files using [pyarrow]"