Hyperdimensional vector encoding

hyperdimensional-vector-encoding

Summary

Encodes mass spectra into binary hypervectors of ultra-high dimensionality (>1000) using level-id encoding, enabling lightweight yet powerful data representation for downstream clustering. This brain-inspired approach projects m/z–intensity pairs into hyperdimensional space to improve clustering quality and speed.

When to use

Apply this skill when you have preprocessed mass spectra (mz/intensity pairs in MGF format) and need to convert them into a compact, fixed-dimensional representation suitable for fast similarity computation and clustering. The technique is especially valuable when clustering large-scale datasets (millions of spectra) where traditional distance metrics become computationally expensive.

When NOT to use

• Input spectra are already in feature vector form (e.g., fingerprints, already-embedded representations) — encoding is redundant.
• You require interpretability of individual dimensions or direct correlation to m/z bins — hyperdimensional encoding obscures peak-level attribution.
• GPU memory is unavailable and CPU-only encoding is mandatory — the method is optimized for GPU and slower on CPU alone.

Inputs

• MGF files containing raw mass spectra (mz/intensity peak pairs)
• Preprocessed spectra with filtering applied (peak count, intensity thresholds, m/z range)
• Configuration parameters: HD dimensionality, quantization level, batch size

Outputs

• Binary hypervectors (high-dimensional, typically 2048–8192 dimensions)
• Encoded spectra checkpoint file (optional parquet or binary format)
• Metadata linking each hypervector to original spectrum identifiers and precursor properties

How to apply

Load raw mass spectra from MGF input files containing mz/intensity peak pairs. Apply level-id encoding to project each spectrum into a binary hypervector of configurable dimensionality (default: 2048) with quantization level (default: 16). The encoding is GPU-accelerated and batched for efficiency (default batch size: 5000). Each spectrum is transformed into a high-dimensional binary representation that preserves spectral similarity relationships while enabling Hamming distance computation. Validate that output hypervectors are binary format and match the specified HD dimensionality before export.

Related tools

• HyperSpec (Python library providing GPU-accelerated level-id encoding and HD projection for mass spectra) — https://github.com/wh-xu/Hyper-Spec
• Python 3.8+ (Language runtime for encoding pipeline and CUDA integration)
• CUDA (GPU compute platform enabling parallel hypervector encoding and Hamming distance computation)

Examples

python src/main.py ~/dataset/ ./output.csv --cpu_core_preprocess=4 --hd_dim=2048 --hd_Q=16 --batch_size=5000 --use_gpu_cluster --cluster_alg dbscan

Evaluation signals

• Output hypervectors are binary (all values 0 or 1) and match the configured HD dimensionality exactly.
• Hamming distances between encoded spectra are computable in O(n/64) time per pair (bitwise operations), confirming binary format.
• Spectra with high cosine similarity in original m/z space produce low Hamming distance in HD space (inverse relationship validation).
• Encoding batch throughput meets expected GPU performance (e.g., >5000 spectra/second on GTX 3090) indicating correct GPU utilization.
• Clustering results on encoded vectors show comparable or improved silhouette score and purity compared to baseline clustering on raw spectra.

Limitations

• Encoding quality depends on careful tuning of quantization level (HD_Q) and dimensionality (HD_DIM); suboptimal settings reduce clustering coherence.
• Large-scale datasets (>25M spectra) require high-memory GPU (GTX 3090 recommended; GTX 1080Ti sufficient for smaller datasets per README).
• Brain-inspired level-id encoding is a lossy projection; original m/z values cannot be recovered from hypervectors.
• GPU availability is mandatory for production performance; CPU fallback is orders of magnitude slower.

Evidence

• [readme] HyperSpec first encodes the processed spectra into binary hypervector (HV) with ultra-high dimension (>1000) based on level-id encoding method.: "HyperSpec first encodes the processed spectra into binary hypervector (HV) with ultra-high dimension (>1000) based on level-id encoding method."
• [intro] HyperSpec adopts brain-inspired hyperdimensional computing to project spectra data into binary hyperdimensional space for better clustering quality and faster clustering speed.: "HyperSpec adopts the brain-inspired hyperdimensional (HD) computing to project the spectra data into binary hyperdimensional space to obtain better clustering quality and faster clustering speed."
• [other] Apply brain-inspired HD computing projection to encode each spectrum into a binary hyperdimensional vector representation; validate encoding dimensionality and binary format compliance.: "Apply brain-inspired hyperdimensional (HD) computing projection to encode each spectrum into a binary hyperdimensional vector representation. 3. Validate encoding dimensionality and binary format"
• [readme] The HD dimension defaults to 2048 and quantization level defaults to 16.: "--hd_dim The HD dimension. (default: 2048)"
• [intro] HyperSpec shortens runtime on 25 million spectra from hours to <15 minutes using HD encoding and GPU acceleration.: "HyperSpec shortens the runtime on the draft human proteome dataset with 25 million spectra from a few hours to <15 minutes."
• [readme] The encoding module is implemented and optimized for GPU for shorter runtime.: "The encoding module is implemented and optimized for GPU for shorter runtime."