Cloud compatible file output

cloud-compatible-file-output

Summary

Format and finalize metabolomics feature tables using the Formation module to generate cloud-compatible output files that meet standardized schema and quality-control criteria. This ensures downstream interoperability and reproducibility across the BMXP pipeline.

When to use

Apply this skill after feature clustering and drift correction (Gravity and Blueshift outputs) are complete and you need to produce a final, validated feature table ready for storage, archival, or downstream analysis. Trigger when you have a processed feature table that must conform to BMXP's shared schema (Feature Metadata, Injection Metadata, Sample Metadata, and Feature Abundances pivot tables) and pass QC checks before release.

When NOT to use

• Input is raw LCMS data (.raw or .mzml files) — use Chroma first to read and convert to feature format.
• Feature table has not yet undergone drift correction and redundancy clustering — use Blueshift and Gravity before Formation.
• You need to align multiple datasets from different LCMS methods — use Eclipse before clustering and formation.

Inputs

• Processed feature table from Blueshift or Gravity (pivot table of Feature x Injection abundances)
• Feature Metadata table (bmxp.FMDATA) with Compound_ID, RT, MZ, Intensity, Method, Annotation_ID, Adduct, Metabolite columns
• Injection Metadata table (bmxp.IMDATA) with injection_id, broad_id, program_id, injection_type, QCRole columns
• Sample Metadata table (bmxp.SMDATA) with broad_id and arbitrary metadata columns

Outputs

• Formatted feature table adhering to BMXP shared schema with standardized column headers and naming conventions
• QC report documenting pass/fail status for feature integrity, missing-value thresholds, and metadata completeness checks
• Cloud-compatible output file(s) suitable for archival, storage, and downstream analysis

How to apply

Load the processed feature table (output from Blueshift or Gravity) into Formation in the appropriate LCMS data format. Apply Formation-specific formatting rules to standardize feature annotations, metadata structure, and column naming conventions according to the BMXP shared schema (e.g., Compound_ID, RT, MZ, injection_id, broad_id). Execute quality-control checks on feature integrity, missing-value thresholds, and metadata completeness. Generate a formatted output file and a QC report documenting pass/fail status for each check. Validate that the output meets Formation acceptance criteria before finalizing and deploying to cloud storage.

Related tools

• bmxp (Python package providing the Formation module and shared schema for metabolomics pipeline standardization) — https://github.com/broadinstitute/bmxp
• Formation (Standalone BMXP module that performs formatting, standardization, and final QC on processed LCMS feature data) — https://github.com/broadinstitute/bmxp/blob/main/bmxp/formation/readme.md
• Blueshift (Upstream module providing drift-corrected feature table as input to Formation) — https://github.com/broadinstitute/bmxp/blob/main/bmxp/blueshift/readme.md
• Gravity (Upstream module providing clustered redundant features as input to Formation) — https://github.com/broadinstitute/bmxp/blob/main/bmxp/gravity/readme.md
• Python (Programming language used to implement Formation and all BMXP modules)

Evaluation signals

• All feature metadata columns (Compound_ID, RT, MZ, Intensity, Method, Annotation_ID, Adduct, Metabolite, Non_Quant) are present and populated according to BMXP.FMDATA schema.
• All injection metadata columns (injection_id, broad_id, program_id, injection_type, QCRole) are present and populated according to BMXP.IMDATA schema.
• Feature x Injection abundance pivot table uses Compound_ID as row index and injection_id as column index with no missing values above the specified threshold.
• QC report shows pass status for feature integrity checks, missing-value thresholds, and metadata completeness for all records.
• Output file conforms to cloud-compatible format (e.g., parquet, HDF5, or CSV) and is free of encoding errors, special characters in column headers, or schema violations.

Limitations

• Formation is a standalone module and requires upstream preprocessing by Blueshift and Gravity; it cannot recover from errors introduced in earlier pipeline stages.
• The shared schema and column naming conventions are global and must be configured consistently across all BMXP modules before running Formation; schema mismatches will cause validation failures.
• Formation does not perform feature annotation or metabolite identification — only formatting and validation of metadata supplied by upstream modules or external sources.

Evidence

• [other] Formation is a standalone module in the BMXP pipeline that performs formatting and final quality control on processed LCMS feature data.: "Formation is a standalone module in the BMXP pipeline that performs formatting and final quality control on processed LCMS feature data."
• [other] Apply Formation-specific formatting rules to standardize feature annotations, metadata structure, and column naming conventions.: "Apply Formation-specific formatting rules to standardize feature annotations, metadata structure, and column naming conventions."
• [other] Execute quality control checks on feature integrity, missing-value thresholds, and metadata completeness.: "Execute quality control checks on feature integrity, missing-value thresholds, and metadata completeness."
• [readme] Each tool is meant to be a standalone module that performs a step in our processing pipeline. They are written in Python and C, and designed to be perfomant and cloud-compatible.: "Each tool is meant to be a standalone module that performs a step in our processing pipeline. They are written in Python and C, and designed to be perfomant and cloud-compatible."
• [readme] All BMXP modules use a shared schema and file formats with our prefered columns headers.: "All BMXP modules use a shared schema and file formats with our prefered columns headers."
• [other] Load the processed feature table (output from Blueshift or Gravity) in the appropriate metabolomics data format.: "Load the processed feature table (output from Blueshift or Gravity) in the appropriate metabolomics data format."