Dataset storage interface architecture

dataset-storage-interface-architecture

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->

Summary

A conditional selection strategy for choosing among multiple storage backend implementations (SubMatrixFile, BigMappedMatrixFile, MappedSubMatrixFile, MappedMatrixFile) based on dataset metadata, memory constraints, and block layout configuration. This skill optimizes memory-mapped file access and cache performance for large NMR datasets.

When to use

Apply this skill when you have NMR dataset metadata (cache-file flag, total point count, block layout configuration) and need to select an appropriate storage backend that balances memory efficiency, access patterns, and platform-specific constraints. Use it specifically when dataset size exceeds available heap memory or when multi-block (tiled) layouts are present.

When NOT to use

• Dataset size is small enough to fit entirely in heap memory and single-block layout applies — use direct in-memory storage instead
• Storage backend is externally mandated by downstream API contract or immutable configuration

Inputs

• Dataset metadata object (containing cache-file flag, total point count, block layout configuration)
• System platform identifier (Windows vs. other)
• Memory availability constraints

Outputs

• Selected DatasetStorageInterface implementation class (SubMatrixFile, BigMappedMatrixFile, MappedSubMatrixFile, or MappedMatrixFile)
• Selection justification and performance rationale

How to apply

First, parse the dataset metadata to extract the cache-file flag, total number of points, and block layout configuration. Then apply conditional logic in sequence: (1) if cache file is enabled in a Windows processing context, select SubMatrixFile; (2) if total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile to enable memory-mapped access; (3) if layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access; (4) otherwise, fallback to MappedMatrixFile for single-block layouts. Document the selection criteria and rationale, especially the performance trade-offs between MappedSubMatrixFile and MappedMatrixFile for borderline single-block cases.

Related tools

• NMRFx (NMR data processing and visualization platform in which Dataset.createDataFile() storage-backend selection logic is embedded) — https://github.com/nanalysis/nmrfx

Evaluation signals

• Verify that selected storage class matches the decision tree: SubMatrixFile only when cache flag + Windows, BigMappedMatrixFile only when byte count > Integer.MAX_VALUE / 2, MappedSubMatrixFile only when multiple blocks are present, MappedMatrixFile as fallback
• Confirm that memory-mapped implementations (BigMappedMatrixFile, MappedSubMatrixFile, MappedMatrixFile) are chosen for datasets exceeding heap limits
• Check that single-block layouts never select MappedSubMatrixFile (confirm block count == 1 before fallback decision)
• Validate that platform-specific path (SubMatrixFile for Windows cache mode) is taken only when both conditions hold
• Review generated documentation comments address stated performance trade-offs, especially rationale for MappedSubMatrixFile vs. MappedMatrixFile boundary

Limitations

• Selection logic depends on accurate metadata parsing; malformed or incomplete cache-file flag, point count, or block layout will trigger fallback (MappedMatrixFile) regardless of actual dataset size
• Integer.MAX_VALUE / 2 threshold is architecture-dependent; 32-bit systems may require different thresholds than 64-bit platforms
• Windows-specific SubMatrixFile path assumes cache file is pre-allocated and accessible; failure to stage cache file will degrade performance
• No dynamic re-selection once DatasetStorageInterface is instantiated; suboptimal choice (e.g., MappedMatrixFile for a multi-block layout) persists for the dataset's lifetime

Evidence

• [other] Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration.: "Parse the dataset metadata to extract cache-file flag, total number of points, and block layout configuration."
• [other] Implement conditional logic to select SubMatrixFile if cache file is enabled (Windows processing context).: "Implement conditional logic to select SubMatrixFile if cache file is enabled (Windows processing context)."
• [other] If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets.: "If total points converted to bytes exceeds Integer.MAX_VALUE / 2, select BigMappedMatrixFile for memory-mapped access to large datasets."
• [other] If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access.: "If layout defines multiple blocks (sub-matrix case), select MappedSubMatrixFile for efficient tiled access."
• [other] Fallback to MappedMatrixFile for single-block layouts.: "Fallback to MappedMatrixFile for single-block layouts."