Mapping agreement cross tool validation

mapping-agreement-cross-tool-validation

Summary

Quantify per-read and per-transcript agreement between two transcript-quantification tools (e.g., salmon versions or implementations) to assess consistency and expose algorithmic differences. This skill validates that a reimplementation or alternative tool produces equivalent quantification outputs within acceptable tolerance.

When to use

When you have built a new tool implementation or major version and need to verify it produces equivalent results to a reference implementation on the same input data and index. Specifically: when two versions claim to use the same algorithm (selective alignment, chaining, MEM extraction) but differ in language/rewrite, and you need to quantify the magnitude of discrepancy and trace which algorithmic steps account for residual differences.

When NOT to use

• Input indices are not byte-identical or were built with different parameters (k-mer length, decoy handling); index differences will confound tool-implementation differences.
• Tools are being run with different quantification parameters (e.g., one with bias correction, one without; different library-type inference); compare only under matched parameter sets.
• You are comparing tools on different datasets or read lengths; per-read and per-transcript agreement are dataset-dependent and cannot be generalized across experiments.

Inputs

• byte-identical transcriptome index (same FASTA, same k-mer length, deterministic non-ACGT replacement)
• paired-end RNA-seq reads in FASTQ or gzip format
• quant.sf output files from both tools
• unmapped_names.txt files (per-read mapping status) from both tools

Outputs

• Pearson correlation coefficient for NumReads per transcript
• Pearson correlation coefficient for TPM per transcript
• per-read mapping agreement percentage (% reads both tools agree are mapped or unmapped)
• total assigned NumReads difference as percentage
• breakdown of mapping discrepancies by category (fully unmapped, partially mapped)

How to apply

Build a byte-identical reference index and run both tools on the same sequencing dataset (e.g., ERR188044, 36.35M paired-end reads) using identical parameters (e.g., selective-alignment mode, -l A auto-detect library type, no bias correction). Extract per-transcript quantification columns (NumReads, TPM) from both tools' quant.sf outputs and compute Pearson correlations. For per-read mapping agreement, parse unmapped_names.txt files from both runs and count reads on which both tools agree in mapping status (mapped vs. fully unmapped, coded 'u'). Report total assigned reads difference as a percentage. When residual disagreement exists (~0.15% per-read mapping disagreement), identify the source by comparing parameter defaults (e.g., orphanChainSubThresh, postMergeChainSubThresh chain-pruning thresholds) and trace whether the difference is algorithmic or parametric. Strong agreement (Pearson > 0.998, per-read agreement > 99.8%) indicates the implementations are functionally equivalent despite language differences.

Related tools

• salmon 2.0 (Rust) (primary implementation to validate (generates quant.sf and unmapped_names.txt)) — https://github.com/COMBINE-lab/salmon
• salmon 1.12.0 (C++) (reference implementation for cross-tool comparison) — https://github.com/COMBINE-lab/salmon/tree/cpp
• piscem-rs (underlying mapper in Rust port; exposes orientation behavior in k-mer lookup) — https://github.com/COMBINE-lab/salmon

Examples

salmon index -t GRCh38_cDNA.fa -i idx_salmon2 -k 31 && salmon index -t GRCh38_cDNA.fa -i idx_salmon112 -k 31 && salmon quant -i idx_salmon2 -l A -1 ERR188044_R1.fastq.gz -2 ERR188044_R2.fastq.gz -o quant_rust --no-bias-seq-bias && salmon quant -i idx_salmon112 -l A -1 ERR188044_R1.fastq.gz -2 ERR188044_R2.fastq.gz -o quant_cpp112 --no-bias-seq-bias && paste <(cut -f1,4,5 quant_rust/quant.sf) <(cut -f4,5 quant_cpp112/quant.sf) | Rscript -e 'x <- read.table(stdin()); cor(x[,2],x[,4]); cor(x[,3],x[,5])'

Evaluation signals

• Per-transcript Pearson correlation for NumReads ≥ 0.9985 indicates high quantitative agreement across thousands of transcripts.
• Per-transcript Pearson correlation for TPM ≥ 0.9989 (higher due to normalization reducing noise) indicates abundance ranking is consistent.
• Per-read mapping agreement ≥ 99.8% (i.e., ≤0.2% reads where tools disagree on mapped vs. unmapped status) indicates low residual algorithmic difference.
• Total assigned NumReads difference < 1% indicates tools assign nearly identical total read mass across the transcriptome.
• When disagreement exists, mapping-gap analysis (e.g., '62,812 reads Rust maps but C++ leaves unmapped') can be traced to specific parameter defaults (chain-pruning thresholds account for ~80% of the difference) rather than core algorithmic divergence.

Limitations

• Per-read and per-transcript agreement are sensitive to minor parameter differences (orphanChainSubThresh, postMergeChainSubThresh); identical algorithm with different defaults will show measurable disagreement (up to ~0.2% per-read in this case).
• Correlation metrics alone (Pearson > 0.998) do not reveal systematic bias or directional differences; residual disagreement must be traced by inspecting specific reads and chains to distinguish algorithmic bugs from intentional parameter tuning.
• Index byte-identity is required; any mismatch in k-mer extraction, decoy handling, or bloom-filter configuration will confound the tool comparison and make it impossible to isolate implementation effects.
• This skill validates quantification consistency but does not assess downstream inference (EM, VBEM) convergence or posterior variance; abundance point estimates may agree while credible intervals diverge.

Evidence

• [methods] Per-read mapping agreement between C++ 1.12.0 and Rust is 99.83% on byte-identical index: "per-read mapping agreement | 99.83%"
• [methods] On byte-identical index, salmon 2.0 (Rust) and salmon 1.12.0 (C++) achieve per-transcript Pearson correlations of 0.99854 for NumReads and 0.99897 for TPM: "On a byte-identical index, salmon 2.0 (Rust) and salmon 1.12.0 (C++) achieve per-transcript Pearson correlations of 0.99854 for NumReads and 0.99897 for TPM, with per-read mapping agreement at 99.83%."
• [methods] ~80% of mapping gap is attributable to chain-pruning-threshold defaults, not algorithmic divergence: "~80% (~49k): a chain-sub-optimality default difference (describable, deferred). C++ prunes low-coverage chains/orphans before alignment using orphanChainSubThresh = 0.95 and postMergeChainSubThresh ="
• [methods] Workflow specifies byte-identical index construction and paired parameter control: "Build a byte-identical GRCh38 cDNA index using both salmon 2.0 and C++ salmon 1.12.0 from the same cleaned FASTA with deterministic non-ACGT replacement (193,759 transcripts). 2. Run salmon 2.0 quant"
• [readme] salmon 2.0 is a Rust rewrite and must be validated against C++ reference: "This is salmon 2.0 — a from-scratch Rust rewrite of salmon. It keeps the same workflow (salmon index → salmon quant → quant.sf) and the same output formats downstream tools read, but it is a"