Molecular dataset partitioning and preprocessing

molecular-dataset-partitioning-and-preprocessing

Summary

Prepare molecular structure and collision cross section data for training and evaluating graph neural network models by applying standardized preprocessing and partitioning into train/validation/test splits. This skill ensures data consistency and reproducibility across GNN generalizability studies.

When to use

You have raw molecular datasets (e.g., METLIN-CCS, CCSBase) with SMILES strings, 3D coordinates, adduct information, and ground-truth collision cross section labels, and you need to format them for GNN training and held-out test evaluation with consistent standardization.

When NOT to use

• Data is already in standardized parquet format with verified column alignment — proceed directly to model training.
• You are evaluating a pre-trained GNN on predictions without retraining — load pre-computed prediction outputs instead.
• Input lacks ground-truth CCS labels or adduct information — preprocessing cannot proceed without complete molecular metadata.

Inputs

• Raw molecular database (CSV/Excel format) — METLIN or CCSBase database
• SMILES strings for each molecule
• 3D coordinates (optional, generated from SMILES if not provided)
• Adduct type labels
• Ground-truth collision cross section values
• Parameter configuration file (JSON) specifying model hyperparameters

Outputs

• Formatted training dataset (parquet file, e.g., metlin_train_3d.parquet)
• Formatted test dataset (parquet file, e.g., ccsbase_3d.parquet)
• Train/validation/test data splits with consistent column schema
• Preprocessed molecular features ready for GNN input

How to apply

Load the raw molecular database (CSV/Excel) and reformat it according to the expected schema: SMILES column, optional 3D coordinates column, adduct column, and CCS label column (see notebooks/data_processing/2_data_splits.ipynb for format details). Apply any required preprocessing or standardization consistent with training (e.g., coordinate normalization, SMILES canonicalization, adduct encoding). Partition data into train/validation/test splits or load a pre-defined held-out test set. The repository provides wrapper functions and example commands in the Makefile that specify input/output parquet files, column names (smiles-column-name, coordinates-column-name, adduct-column-name, ccs-column-name), and preprocessing flags (coordinates-present) to standardize the workflow.

Related tools

• poetry (dependency/environment manager) (Install development dependencies and runtime packages for data processing scripts) — https://github.com/enveda/ccs-prediction
• Python scripts/train-test.py (Execute train/test workflow with specified input files, preprocessing parameters, and output paths) — https://github.com/enveda/ccs-prediction
• Jupyter notebooks (notebooks/data_processing/2_data_splits.ipynb) (Interactive data exploration, format validation, and split verification for METLIN and CCSBase datasets) — https://github.com/enveda/ccs-prediction
• Makefile commands (make train-metlin-test-metlin, etc.) (Standardized shell invocations for reproducible preprocessing with fixed parameter sets) — https://github.com/enveda/ccs-prediction
• Mol2CCS library functions (mol2ccs/train_and_predict.py) (Wrapper for training dataset preprocessing and GNN model input preparation) — https://github.com/enveda/ccs-prediction

Examples

poetry run python scripts/train-test.py --prefix "train-metlin-test-ccsbase" --train-input-file "ccs-prediction/metlin_train_3d.parquet" --test-input-file "ccs-prediction/ccsbase_3d.parquet" --smiles-column-name "smiles" --adduct-column-name "adduct" --ccs-column-name "ccs" --coordinates-column-name "coordinates" --coordinates-present

Evaluation signals

• Output parquet files contain all required columns (smiles, coordinates/adduct, ccs) with no missing values in critical fields
• Row counts and data distributions in train/test splits match documented split ratios and match those used in reproducibility notebooks
• SMILES strings parse correctly and canonicalize consistently (no formatting drift across partitions)
• CCS values fall within expected ranges for the source database (e.g., METLIN vs. CCSBase ranges) with no outliers exceeding documented extrema
• Column names and data types match those specified in parameter configuration (e.g., coordinates-column-name, ccs-column-name) and are consistent with train-test.py invocation arguments

Limitations

• Original METLIN-CCS and CCSBase datasets must be downloaded separately by the user under their respective licenses; the repository does not bundle them
• 3D coordinates are optionally generated from SMILES using an unspecified coordinate generation method if not pre-computed; results may vary by coordinate generation tool
• Preprocessing scripts assume specific input CSV/Excel structures; nonstandard database formats require manual schema mapping before partition scripts will execute correctly
• Data processing notebooks are exploratory and must be manually executed; no automated pipeline currently enforces version-controlled preprocessing across all raw datasets

Evidence

• [readme] Each user should download the raw database (as excel/csv) and read them in the two notebooks for each database located at https://github.com/enveda/ccs-prediction/tree/main/notebooks/data_processing. Each notebook reads the csv/excel and formats it according to the input of Mol2CCS.: "Each user should download the raw database (as excel/csv) and read them in the two notebooks for each database located at https://github.com/enveda/ccs-prediction/tree/main/notebooks/data_processing."
• [other] Load the CCS dataset and apply any required preprocessing or standardization consistent with training. Partition data into train/validation/test splits or load the held-out test set as specified in the repository.: "Load the CCS dataset and apply any required preprocessing or standardization consistent with training. Partition data into train/validation/test splits or load the held-out test set as specified in"
• [readme] See the commands in the Makefile to train the models. Run them as make train-metlin-test-metlin: "See the commands in the Makefile to train the models. Run them as make train-metlin-test-metlin"
• [readme] train-input-file is the training set (see notebooks/data_processing/2_data_splits.ipynb for details on the format)/ test-input-file test set (see notebooks/data_processing/2_data_splits.ipynb for details on the format): "train-input-file is the training set (see notebooks/data_processing/2_data_splits.ipynb for details on the format)/"
• [readme] smiles-column-name column name of the smiles adduct-column-name column name of the adduct ccs-column-name column name of the ccs coordinates-column-name column name of the 3d coordinates for each smiles: "smiles-column-name column name of the smiles adduct-column-name column name of the adduct ccs-column-name column name of the ccs"
• [readme] coordinates-present if the coordinates are present (if not given, the model will use the smiles to generate the 3d coordinates): "coordinates-present if the coordinates are present (if not given, the model will use the smiles to generate the 3d coordinates)"