Interoperability testing

interoperability-testing

Summary

Validate that multiple independent implementations of a file format specification produce and consume compliant output by cross-checking mandatory fields, structural elements, and data integrity across language-specific parsers. This skill ensures that a format remains truly interoperable rather than implementation-specific.

When to use

When a new file format specification exists as a living document (e.g., in HTML or Markdown), multiple prototype implementations have been developed in different programming languages (Rust, Python, R, etc.), and you need to verify that all implementations respect the same schema, field requirements, and structural constraints before the format enters production use or is submitted for standardization approval.

When NOT to use

• The specification is still in concept/pre-draft phase and mandatory fields have not yet been finalized; defer testing until the specification document is published and stable.
• Only one implementation exists; interoperability testing requires at least two independent, language-distinct implementations to verify cross-compatibility.
• The file format is proprietary or vendor-locked and multiple open-source implementations are not available; compliance testing is vendor-dependent and cannot validate true interoperability.

Inputs

• Formal specification document (Markdown or HTML) defining mandatory fields, structural elements, data types, and controlled vocabulary constraints
• mzPeak files or equivalent format files generated by the primary implementation (Rust read/write)
• Test data sets or reference files generated by or compatible with secondary implementations (Python read-only, R read-only, .NET, JavaScript)
• JSON Schema files governing file structure (if published separately in schema/ directories)

Outputs

• Compliance report matrix documenting which mandatory fields are present in each implementation's output
• List of absent or non-conformant fields, organized by implementation language
• Overall specification conformance status (pass/fail per implementation, per file type)
• Annotated diff or structured log of field-by-field validation results

How to apply

First, retrieve the authoritative specification document (e.g., from https://hupo-psi.github.io/mzPeak-specification/) and extract the exhaustive list of mandatory fields and required structural elements organized by section (e.g., file-level metadata, spectra metadata, data layouts). Next, obtain reference files generated by the primary (read/write) implementation and test files created or parsed by secondary (read-only) implementations in each language. Parse each file using its native library to inspect internal structure and field contents. Cross-check every mandatory field and structural constraint from the specification against each parsed file's output, recording presence, absence, type correctness, and value ranges. Generate a compliance matrix documenting which implementation(s) satisfy each specification requirement and flag any deviations. Pay special attention to optional vs. mandatory distinction, since some implementations (e.g., Python and R) may only support reading and thus cannot generate files for full round-trip testing; in these cases, validate read-path compliance only.

Related tools

• Rust mzPeak library (Primary read/write implementation used to generate reference mzPeak files for compliance validation) — https://github.com/HUPO-PSI/mzPeak
• Python pyarrow implementation (Read-only secondary implementation for validating mzPeak file parsing and field extraction in Python ecosystem) — https://github.com/HUPO-PSI/mzPeak
• R arrow implementation (Read-only secondary implementation for validating mzPeak file parsing and metadata extraction in R ecosystem) — https://github.com/HUPO-PSI/mzPeak
• mzPeak-specification repository (Authoritative source of the living specification document and JSON Schemas against which implementations are validated) — https://github.com/HUPO-PSI/mzPeak-specification
• .NET C# implementation (Secondary implementation for validating file reading and structural compliance in .NET ecosystem) — https://github.com/HUPO-PSI/mzPeak.NET
• JavaScript/TypeScript implementation (Secondary implementation for validating browser-based file reading and interoperability) — https://github.com/HUPO-PSI/mzpeakts

Evaluation signals

• All mandatory fields listed in the specification document are present in output files from the primary Rust implementation with correct data types and non-null values.
• Secondary implementations (Python, R, .NET, JavaScript) successfully parse Rust-generated files without schema errors or missing required group/struct elements.
• Round-trip validation (if write support exists): reading a file written by one implementation using a different implementation produces identical field values and structural checksums.
• Null and optional field handling is consistent across implementations—expected nulls appear in all parsers, unexpected nulls trigger validation errors.
• Packed parallel table structures (spectra_metadata.parquet, chromatograms_metadata.parquet) correctly expose all parallel branches (spectrum descriptions, scans, precursors, selected ions) in all implementations' parsed output.

Limitations

• Python, R, and some secondary implementations only support read functionality, preventing full round-trip (write-then-read) compliance testing; validation must be restricted to read-path conformance.
• The mzPeak format is still in work-in-progress status with no stability guarantee, so the specification is a living document and may change, invalidating previous compliance claims.
• Null marking and zero-run stripping optimizations, while part of the specification, introduce lossy approximation in profile data reconstruction; compliance testing must account for acceptable reconstruction error (e.g., m/z spacing model accuracy) rather than exact bit-for-bit equality.
• JSON Schema files governing file-level metadata (instrument descriptions, software, transformation pipeline) may be maintained separately and may lag behind prose specification changes; validate both prose and schema sources.
• Some controlled vocabulary terms (e.g., for spectrum and chromatogram descriptions) reference external HUPO-PSI ontologies; validation of controlled term correctness requires lookup against live ontology servers, which may be unavailable or versioned.

Evidence

• [readme] Specification document location and purpose: "The draft mzPeak specification document is publicly available at https://hupo-psi.github.io/mzPeak-specification/ against which Rust implementation output can be validated for compliance."
• [other] Mandatory fields and structural elements are defined in the specification: "extract the list of mandatory fields and required structural elements. 2. Obtain a reference mzPeak file generated by the Rust implementation from the HUPO-PSI/mzPeak repository"
• [readme] Multiple language implementations exist with varying read/write support: "The primary work shown here is written in Rust at the repository root, including a library for reading and writing mzPeak files, as well as command line tools for converting existing formats into"
• [readme] Read-only implementations cannot generate files for full round-trip testing: "The Python codebase does not support writing at this time although this is subject to change in the future"
• [other] Cross-checking methodology against parsed file contents: "Parse the mzPeak file using the Rust library (reading functionality) to examine its internal structure and field contents. 4. Cross-check all mandatory fields and structural elements from the"
• [readme] Packed parallel table structures as core specification elements: "spectra_metadata.parquet and chromatograms_metadata.parquet store multiple schemas in parallel. In these Parquet files, the root schema is made up of several branched "group" or "struct""
• [readme] JSON Schema files as secondary specification artifacts: "JSON Schemas that govern the file formats live in schema/, and figures in [docs/assets/img/]"