Spectral token design

spectral-token-design

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->

Summary

Design and validate a unified token vocabulary that encodes both mass spectral features (m/z values, intensities) and molecular structures (SMILES or graph tokens) into a single discrete token set for end-to-end pretraining in a language model. This skill ensures bidirectional mapping and complete coverage of the chemical data domain.

When to use

When you need to prepare mass spectra and molecular structures for joint modeling in a BART or transformer-based sequence model, and you lack a unified representation scheme that allows both modalities to be encoded and decoded without collision or information loss. Specifically, when pretraining or fine-tuning a model on paired spectra–structure data where both input and output must be tokenized using the same vocabulary.

When NOT to use

• The input data is already tokenized using an existing, well-validated vocabulary that covers both spectra and structures without collision.
• You are working with only one modality (e.g., structures alone) and do not need joint end-to-end pretraining.
• The model architecture does not support discrete tokenization (e.g., it operates directly on continuous embeddings or graph neural networks).

Inputs

• Mass spectra (m/z values and intensity pairs)
• Molecular structures (SMILES strings or graph representations)
• Data range and distribution statistics (spectral peaks, structural complexity)

Outputs

• Unified token vocabulary (token ID assignments)
• Bidirectional mapping dictionary (raw data ↔ token ID)
• Vocabulary validation report (coverage metrics, collision checks, statistics)

How to apply

First, define separate token vocabularies for mass spectral peaks (m/z values and intensities) and molecular structures (SMILES strings or graph tokens), partitioning the value ranges based on typical data characteristics and chemical complexity observed in your dataset. Merge the two vocabularies into a unified token set, assigning unique token IDs to each spectral feature and structural element while preserving special tokens (BOS, EOS, PAD, UNK) at fixed positions. Create a bidirectional mapping dictionary between raw spectral and structural data and their token IDs to enable efficient encoding and decoding. Validate the vocabulary by encoding representative examples from both modalities, verifying that all data types tokenize correctly, no token ID collisions occur, and domain coverage is complete (e.g., all m/z ranges and SMILES substructures are representable). Generate a summary report with vocabulary statistics (total token count, spectral vs. structural token breakdown, special token assignments) and coverage metrics to confirm readiness for pretraining.

Related tools

• BART (Sequence-to-sequence language model framework that consumes the unified token vocabulary for pretraining and fine-tuning on spectra and molecules)
• MS-BART (Reference implementation demonstrating unified vocabulary design and validation for mass spectra and molecular structure modeling) — https://github.com/OpenDFM/MS-BART

Evaluation signals

• All representative mass spectra and molecular structures encode without error and decode back to the original or near-identical form (lossless or acceptable loss).
• No token ID collisions occur: each unique spectral feature or structural element maps to exactly one token ID.
• Coverage metrics confirm that the full ranges of m/z values, intensities, and SMILES substructures present in the dataset are representable within the vocabulary.
• Vocabulary statistics report shows zero unknown tokens (UNK assignments) when encoding a held-out validation set of spectra and structures.
• The bidirectional mapping dictionary has symmetric cardinality: every token ID in the vocabulary maps to a unique data element, and vice versa.

Limitations

• Vocabulary size grows linearly with the number of unique spectral features and structural motifs; very large datasets or high-resolution spectra may require aggressive quantization or hierarchical tokenization.
• The choice of token ID assignment strategy and special token positioning can affect downstream model learning efficiency; suboptimal vocabulary design may not be detected until pretraining.
• Coverage validation depends on the representativeness of the validation set; rare spectral features or unusual chemical structures not present in the validation set may cause OOV errors during deployment.

Evidence

• [other] MS-BART introduces a unified vocabulary as a core mechanism to enable end-to-end pretraining, fine-tuning, and alignment of mass spectra and molecular structures within a language model framework.: "MS-BART introduces a unified vocabulary as a core mechanism to enable end-to-end pretraining, fine-tuning, and alignment of mass spectra and molecular structures within a language model framework."
• [other] Define token vocabularies for mass spectral peaks (m/z values and intensities) and molecular structures (SMILES strings or graph tokens) based on typical data ranges and chemical complexity.: "Define token vocabularies for mass spectral peaks (m/z values and intensities) and molecular structures (SMILES strings or graph tokens) based on typical data ranges and chemical complexity."
• [other] Merge the two vocabularies into a unified token set, assigning unique token IDs to spectral features and structural elements while preserving special tokens (BOS, EOS, PAD, UNK).: "Merge the two vocabularies into a unified token set, assigning unique token IDs to spectral features and structural elements while preserving special tokens (BOS, EOS, PAD, UNK)."
• [other] Create a bidirectional mapping dictionary between raw spectral/structural data and token IDs for efficient encoding/decoding.: "Create a bidirectional mapping dictionary between raw spectral/structural data and token IDs for efficient encoding/decoding."
• [other] Validate the vocabulary by encoding representative mass spectra and molecular structures to verify all data types tokenize correctly, no token collisions occur, and coverage of the data domain is complete.: "Validate the vocabulary by encoding representative mass spectra and molecular structures to verify all data types tokenize correctly, no token collisions occur, and coverage of the data domain is"
• [intro] MS-BART is the first to leverage language model for mass spectra structure elucidation by introducing a unified vocabulary and enabling end-to-end pretraining, fine-tuning, and alignment.: "MS-BART is the first to leverage language model for mass spectra structure elucidation by introducing a unified vocabulary and enabling end-to-end pretraining, fine-tuning, and alignment."