Sequential and random access dual mode support

sequential-and-random-access-dual-mode-support

Summary

Implement dual-mode file access (sequential iteration and random indexing) for compressed mass spectrometry data formats by creating a wrapper class that bridges pymzML's FileInterface with custom storage backends, enabling both blockwise iteration and key-based random access to indexed gzip archives.

When to use

When parsing mzML or other blockwise-structured scientific data files where analysis requires both sequential scanning (e.g., iterating all spectra) and random direct access by identifier (e.g., retrieving a specific chapter or spectrum by name/ID), and file compression is necessary to reduce storage from RAW format sizes while maintaining seek capability.

When NOT to use

• Input is already a compressed indexed gzip file (use GSGR reader directly rather than reconstructing)
• Analysis requires only sequential access with no random lookups (standard gzip or streaming decompressor is simpler and faster)
• File size is small enough that full decompression into memory is feasible (overhead of indexing not justified)

Inputs

• uncompressed blockwise-structured text file (e.g., mzML, chapters of Moby Dick)
• logical block identifiers (integer or string keys for each block)

Outputs

• indexed gzip file with byte-offset index mapping keys to block locations
• wrapped file handler object supporting both read() and __getitem__() interfaces

How to apply

Create a custom wrapper class that implements both read() (for sequential iteration returning one logical block per call) and __getitem__(key) (for random access returning a block by integer or string identifier). Parse the input file blockwise using the Generalized Seekable Gzip Writer (GSGW) pattern, writing each logical unit (e.g., spectrum, chapter) with add_data() and finalizing with write_index() to map keys to byte offsets. Integrate the wrapper into pymzML's FileInterface by adding an elif clause in _open() to detect the file type and instantiate your handler. Validate that random access via bracket notation returns correct blocks and sequential reads iterate without data loss or corruption.

Related tools

• pymzML (Provides FileInterface abstraction layer, GSGW/GSGR indexed gzip classes, and spectrum/mzML parsing; wrapper integrates into pymzML.FileInterface._open() to enable dual-mode access) — https://github.com/pymzml/pymzML
• ElementTree (XML parsing for mzML structure (blockwise parsing may leverage ElementTree to identify logical boundaries))
• Python (Core language for implementing wrapper class and integration)
• Black (Code formatter for maintaining style consistency in wrapper and integration code) — https://github.com/psf/black

Examples

from pymzml import FileInterface; handler = FileInterface._open('moby_dick.indexed.gzip'); chapter_5 = handler['5']; all_chapters = [block for block in iter(handler.read, None)]

Evaluation signals

• Random access via bracket notation (e.g., handler[key]) returns the exact block corresponding to that key without loading the entire file
• Sequential iteration using read() returns blocks in order, with each call advancing position, and stops cleanly at EOF
• Byte offsets in the index file are correct (verify by seeking to offset, decompressing, and confirming block identity)
• No data corruption or truncation detected when comparing decompressed block content to original blockwise input
• File size of indexed gzip archive is comparable to or smaller than original RAW format (per pymzML design goal)

Limitations

• Index must be finalized with write_index() before use; dynamic updates to keys after finalization require index reconstruction
• Blockwise parsing assumes clear logical boundaries (e.g., chapter markers, spectrum delimiters); ambiguous block boundaries may cause data loss
• Random access performance depends on index lookup speed; very large indices may incur overhead
• The wrapper must be tightly coupled to pymzML's FileInterface; compatibility issues may arise if pymzML's API changes

Evidence

• [other] GSGW accepts data parsed blockwise from a source file, with each logical block (e.g., chapter) written in a single operation using add_data() and indexed by either integer or string identifiers, then finalized with write_index() to create an indexed gzip file.: "GSGW accepts data parsed blockwise from a source file, with each logical block (e.g., chapter) written in a single operation using add_data() and indexed by either integer or string identifiers, then"
• [readme] pymzML is an extension to Python that offers a) easy access to mass spectrometry (MS) data that allows the rapid development of tools b) a very fast parser for mzML data, the standard mass spectrometry data format c) a set of functions to compare and/or handle spectra d) random access in compressed files e) interactive data visualization: "pymzML is an extension to Python that offers a) easy access to mass spectrometry (MS) data that allows the rapid development of tools b) a very fast parser for mzML data, the standard mass"
• [other] a new class needs to be written, which implements a read and a __getitem__ function: "a new class needs to be written, which implements a read and a __getitem__ function"
• [other] In order to allow pymzML to use this new file class, the filehandler needs to be able to detect when to use this class. The easiest way is, to add another elif statement which decides which handler: "In order to allow pymzML to use this new file class, the filehandler needs to be able to detect when to use this class. The easiest way is, to add another elif statement"
• [readme] indexed gzip which allows mzML file sizes to reach the levels of the original RAW format: "indexed gzip which allows mzML file sizes to reach the levels of the original RAW format"