Rnafm

RNA-FM: Foundation Model for Non-Coding RNA

Overview

RNA-FM is a BERT-style transformer pretrained on 23 million non-coding RNA (ncRNA) sequences via masked language modeling. It generates contextual nucleotide embeddings that capture RNA structure and function without labeled data.

Model variants:

• RNA-FM — trained on ncRNA sequences (23M); embedding dim = 640; tokenizes individual nucleotides
• mRNA-FM — trained on 45M mRNA coding sequences (CDS); embedding dim = 1280; tokenizes 3-mers (codons)

Capabilities:

• Sequence embeddings — per-token or pooled representations for any downstream task
• Secondary structure prediction — base-pair contact maps (outperforms LinearFold/SPOT-RNA)
• RNA family clustering — zero-shot separation of RNA families in embedding space
• Functional prediction — UTR function, gene expression, RNA-protein binding

API note: RNA-FM's Python API mirrors ESM2 — (model, alphabet) pair, batch_converter, repr_layers. If you know ESM2, RNA-FM will feel familiar.

Installation

pip install rna-fm

Requirements: Python ≥ 3.8, PyTorch ≥ 1.9. GPU with CUDA 11.1+ recommended.

Model weights download automatically on first use (~1.2 GB for RNA-FM, ~957 MB for mRNA-FM).

Model Checkpoints

Checkpoint	Training data	Embed dim	Tokenization	Best for
rna_fm_t12	23M ncRNA sequences	640	Single nucleotide	ncRNA, structural RNA, general RNA
mrna_fm_t12	45M mRNA CDS sequences	1280	3-mer (codon)	mRNA analysis, codon usage, translation

Core Usage

Load model

import fm

# ncRNA model (default)
model, alphabet = fm.pretrained.rna_fm_t12()

# mRNA model
model, alphabet = fm.pretrained.mrna_fm_t12()

model.eval()

Extract embeddings

import torch
import fm

model, alphabet = fm.pretrained.rna_fm_t12()
model.eval()

batch_converter = alphabet.get_batch_converter()

# Input: list of (label, sequence) tuples
sequences = [
    ("rna1", "GGGUGCGAUCAUACCAGCACUAAUGCCCUCCUGGGAAGUCCUCGUGUUGCACCCCU"),
    ("rna2", "AUGUAAGGCCUUGUAACGCUCUAAACUUCCCCCGCGACGUUUUU"),
]

batch_labels, batch_strs, batch_tokens = batch_converter(sequences)

with torch.no_grad():
    results = model(batch_tokens, repr_layers=[12])

# Per-token embeddings from last layer: (batch, seq_len, 640)
token_embeddings = results["representations"][12]

# Per-sequence mean pooling (exclude BOS/EOS/PAD)
padding_idx = alphabet.padding_idx
for i, label in enumerate(batch_labels):
    mask = (batch_tokens[i] != padding_idx).float()
    seq_emb = (token_embeddings[i] * mask.unsqueeze(-1)).sum(0) / mask.sum()
    print(f"{label}: {seq_emb.shape}")  # (640,)

Secondary structure prediction

# CLI
python launch/predict.py \
  --config="pretrained/ss_prediction.yml" \
  --data_path="sequences.fasta" \
  --save_dir="results/"

Output: contact map in CT format (base-pair predictions).

Extract embeddings via CLI

# Mean-pooled embeddings
python launch/predict.py \
  --config="pretrained/extract_embedding.yml" \
  --data_path="sequences.fasta" \
  --save_dir="results/" \
  --save_embeddings \
  --save_embeddings_format="mean"

# Per-token (raw) embeddings
python launch/predict.py \
  --config="pretrained/extract_embedding.yml" \
  --data_path="sequences.fasta" \
  --save_dir="results/" \
  --save_embeddings \
  --save_embeddings_format="raw"

# BOS token (sequence-level)
python launch/predict.py \
  --config="pretrained/extract_embedding.yml" \
  --data_path="sequences.fasta" \
  --save_dir="results/" \
  --save_embeddings \
  --save_embeddings_format="bos"

Key Parameters

Parameter	Value	Description
repr_layers	[12]	Which transformer layers to extract (0–12; 12 = last)
Max sequence length	1024 nt	Hard limit; truncate longer sequences
Embedding dim	640 (RNA-FM) / 1280 (mRNA-FM)	Output size per token
need_head_weights	False	Return attention weights (memory-intensive)
return_contacts	False	Return contact predictions

Special Tokens

RNA-FM uses ESM-1b-style tokenization:

• <cls> / BOS prepended at position 0
• <eos> appended at end
• <pad> fills shorter sequences in a batch

When extracting per-residue embeddings for downstream use, slice [1:-1] to remove BOS and EOS:

per_residue = token_embeddings[i, 1:-1]  # shape: (seq_len, 640)

For mean pooling, mask out padding tokens using alphabet.padding_idx.

Output

Format	Shape	Description
repr_layers=[12]	(batch, seq_len, 640)	Per-token embeddings (includes BOS/EOS)
Mean pooled	(batch, 640)	Sequence-level representation
BOS token	(batch, 640)	CLS-style representation
logits	(batch, seq_len, vocab_size)	Masked LM prediction logits

Scripts

• scripts/embed.py — embed FASTA sequences, save as .npy or .h5; see scripts/embed.py --help

Related Models (ml4bio)

The ml4bio lab builds an RNA design ecosystem around RNA-FM:

Model	Task	Repo
RhoFold+	RNA 3D structure prediction	ml4bio/RhoFold
RhoDesign	RNA inverse folding (structure → sequence)	ml4bio/RhoDesign
RiboDiffusion	Diffusion-based RNA inverse folding	ml4bio/RiboDiffusion

Resources

• GitHub: https://github.com/ml4bio/RNA-FM
• Paper: Chen et al., Nature Methods 2024 — https://doi.org/10.1038/s41592-024-02487-0

References

• references/api-reference.md — full API reference: BatchConverter, model.forward(), embedding formats, secondary structure, mRNA-FM differences