Lexical analysis tokenization

lexical-analysis-tokenization

Summary

Tokenize domain-specific query strings (e.g., MassQL) into a stream of meaningful lexical units (keywords, operators, numeric literals, identifiers) to enable downstream parsing and AST construction. This is the first stage of a compiler/interpreter pipeline for mass-spectrometry-centric query languages.

When to use

You have a mass-spectrometry query string written in MassQL (or similar domain-specific SQL-inspired syntax) that must be converted into structured form for execution. The input is raw, unparsed text containing SQL keywords, MS-specific operators (e.g., MS1MZ, MS2PREC, TOLERANCEMZ, INTENSITYPERCENT), numeric m/z and intensity thresholds, and logical connectives (AND, OR). Tokenization is necessary before you can build an Abstract Syntax Tree (AST) or validate query semantics.

When NOT to use

• Input is already a pre-tokenized or parsed representation (e.g., JSON-serialized query object, AST). Skip directly to semantic validation or execution.
• You only need to execute a pre-compiled query; the query text has already been tokenized and validated by an earlier stage of the pipeline.
• Input is free-form natural language prose, not a formal query syntax. Domain-specific tokenization assumes formal grammar; unstructured text requires NLP preprocessing instead.

Inputs

• MassQL query string (raw text)
• MassQL grammar specification (keywords, operators, token patterns)

Outputs

• Token stream (sequence of (token_type, token_value, position) tuples)
• Lexical error report (if unrecognized input encountered)

How to apply

Implement a lexer that scans the input MassQL query string character-by-character, recognizing and classifying tokens according to the MassQL grammar. Group characters into meaningful units: (1) SQL and MS-specific keywords (QUERY, WHERE, MS1DATA, MS2DATA, scaninfo, PREC, MZ); (2) comparison and logical operators (=, AND, +, −); (3) numeric literals (m/z values, tolerance thresholds, intensity percentages); (4) special syntax (parentheses, colons for key-value pairs like TOLERANCEMZ=0.1). Use a whitespace-aware strategy to delimit tokens. Emit a stream of (token_type, token_value) pairs, preserving enough positional information to generate diagnostic error messages if parsing fails later. Validate that all tokens are recognized; if an unrecognized character sequence is encountered, halt with a clear error indicating the location and unexpected input.

Related tools

• MassQL (Domain-specific query language whose syntax defines the lexical grammar (keywords, operators, literals) to be tokenized) — https://github.com/mwang87/MassQueryLanguage

Examples

from massql import msql_engine; results_df = msql_engine.process_query('QUERY MS2DATA WHERE MS1MZ=572.828 TOLERANCEMZ=0.1 AND MS2PREC=572.828', 'sample.mzML')

Evaluation signals

• All valid MassQL tokens (SQL keywords like QUERY, WHERE; MS-specific operators like MS1MZ, MS2DATA, TOLERANCEMZ; numeric literals; parentheses) are correctly classified and emitted in order.
• Whitespace and comment regions are properly skipped; output token stream contains no spurious whitespace tokens.
• Compound tokens (e.g., 'MS1MZ=0.1' vs. separate 'MS1MZ' and '=' and '0.1' tokens) are split or grouped according to the grammar specification; test with multi-clause queries (AND chains with nested parentheses).
• Unrecognized input (typos, invalid operators, malformed numeric literals) triggers a lexical error with source location (line, column), enabling user diagnosis.
• Token stream is sufficient to reconstruct a recognizable approximation of the original query (round-trip test): serialize tokens back to text and compare against normalized input.

Limitations

• Lexical analysis does not validate semantic correctness (e.g., whether MS1MZ tolerance is physically reasonable or whether a referenced filter exists). Semantic checks are deferred to the parser and AST validator.
• Error recovery is minimal: the lexer typically stops at the first unrecognized token. Batch error collection (reporting all lexical issues in one pass) would require more sophisticated tokenization.
• The lexer assumes well-formed input encoding (UTF-8 or ASCII); binary or mixed-encoding query strings may cause silent misclassification.
• No support for comments or query metadata (e.g., /* comment */ syntax) unless explicitly defined in the grammar.

Evidence

• [other] Implement a lexer to tokenize the input MassQL query string, identifying SQL keywords, MS-specific operators, and numeric literals.: "Implement a lexer to tokenize the input MassQL query string, identifying SQL keywords, MS-specific operators, and numeric literals."
• [readme] MassQL is inspired by SQL, but it attempts to bake in assumptions of mass spectrometry to make querying much more natural for mass spectrometry users.: "MassQL is inspired by SQL, but it attempts to bake in assumptions of mass spectrometry to make querying much more natural for mass spectrometry users."
• [other] Define the MassQL grammar as an extended SQL syntax incorporating mass-spectrometry-specific constructs (e.g., spectrum filters, m/z ranges, intensity thresholds, fragmentation patterns): "Define the MassQL grammar as an extended SQL syntax incorporating mass-spectrometry-specific constructs (e.g., spectrum filters, m/z ranges, intensity thresholds, fragmentation patterns)"
• [other] Serialize the AST to JSON or canonical text format and report parsing success/failure with diagnostic error messages.: "Serialize the AST to JSON or canonical text format and report parsing success/failure with diagnostic error messages."