Metadata normalization and reconciliation

metadata-normalization-and-reconciliation

Summary

Harmonize and reconcile feature metadata (identifiers, retention time, m/z, and other fields) from multiple mass spectrometry feature lists generated by different acquisition methods or processing software, then perform cross-list comparison to identify overlapping and unique features. This skill is essential when integrating heterogeneous MS datasets that must be unified before comparative analysis.

When to use

You have multiple CSV feature lists from different acquisition methods (e.g., LC-MS vs LC-IMS-MS) or processing software, each using different naming conventions, retention time scales, or m/z precision; you need to identify which features are the same across lists and which are unique to each method before performing downstream analysis or meta-analysis.

When NOT to use

• Input data are already from a single acquisition method or processing pipeline (no cross-method reconciliation needed)
• Feature lists lack common measurable fields such as m/z or retention time (comparison cannot be performed reliably)
• The goal is quality control of a single feature list rather than integration of multiple lists

Inputs

• CSV files containing feature lists from different acquisition methods or processing software
• Each CSV must include columns for feature identifiers, m/z values, retention time, and other metadata fields

Outputs

• Harmonized feature table (CSV or tabular format) with unified feature IDs and normalized metadata
• Comparison report documenting feature overlap statistics, source method tags, and presence/absence across input lists

How to apply

Load all CSV feature lists using pandas, then apply harmonization by matching on common fields (m/z, retention time, feature identifiers) and resolving naming conflicts through field mapping and tolerance-based matching (e.g., m/z and RT windows). Cross-list comparison logic identifies overlapping features across all input lists by finding matches within specified tolerances, while tagging each feature with its source method. Generate a structured comparison report documenting feature overlap statistics, harmonized feature IDs, source method annotations, and presence/absence patterns. The rationale is that direct string matching or ID comparison fails when software or instruments use different scales or nomenclature; tolerancing on continuous fields (m/z, RT) enables robust reconciliation even when nomenclature differs.

Related tools

• Comparador (Primary tool for harmonizing and comparing feature lists from different acquisition methods or processing software; ingests CSV files and applies reconciliation logic) — https://github.com/pnnl/IonToolPack
• IonToolPack (Software suite housing Comparador; provides GUI interface for feature list comparison and supports multiple MS data formats and omics domains) — https://github.com/pnnl/IonToolPack

Evaluation signals

• Harmonized feature IDs are consistent and resolvable across all input lists; no duplicate or conflicting IDs in the output
• Feature overlap statistics are symmetric or near-symmetric (if list A matches feature X from list B, then list B's feature X is tagged as matching list A)
• All input features are accounted for in the output; presence/absence patterns match the input data
• Retention time and m/z values in the harmonized output fall within the specified tolerance windows of their matched counterparts
• Source method tags correctly identify the origin of each feature and reflect input list boundaries

Limitations

• Harmonization accuracy depends on the choice of tolerance windows for m/z and retention time matching; overly tight tolerances may miss true matches across methods; overly loose tolerances may create false matches
• Naming conflicts in feature identifiers may not be fully resolved if software systems use orthogonal naming schemes with no common measurable anchor
• The workflow assumes CSV format; other feature list formats (e.g., mzTab, proprietary binary formats) must be converted to CSV first
• Cross-method comparison is limited to common fields present in all input lists; method-specific metadata may be lost during harmonization

Evidence

• [other] Comparador ingests feature lists in CSV format from different acquisition methods or processing software, applies harmonization procedures, and performs comparative analysis on the results.: "Comparador ingests feature lists in CSV format from different acquisition methods or processing software, applies harmonization procedures, and performs comparative analysis on the results."
• [other] Harmonize feature identifiers, retention time, m/z, and other metadata across lists by matching on common fields and resolving naming conflicts.: "Harmonize feature identifiers, retention time, m/z, and other metadata across lists by matching on common fields and resolving naming conflicts."
• [other] Identify overlapping and unique features across all input lists using cross-list comparison logic.: "Identify overlapping and unique features across all input lists using cross-list comparison logic."
• [other] Generate a structured comparison report (CSV or tabular format) documenting feature overlap statistics, harmonized feature IDs, source method tags, and analysis results.: "Generate a structured comparison report (CSV or tabular format) documenting feature overlap statistics, harmonized feature IDs, source method tags, and analysis results."
• [readme] Tool to compare lists of features (CSV files) from different acquisition methods or processing software, by harmonizing and analyzing results.: "Tool to compare lists of features (CSV files) from different acquisition methods or processing software, by harmonizing and analyzing results."