Prostt5

ProstT5: Protein Sequence ↔ Structure Translation

Overview

ProstT5 is a T5-based protein language model finetuned from ProtT5-XL-U50 to translate between amino acid (AA) sequences and 3Di structural alphabet tokens — the same discrete structural tokens used by Foldseek for fast structure search.

Key capabilities:

• AA → 3Di ("folding"): predict structure as a 3Di string from sequence alone
• 3Di → AA ("inverse folding"): recover amino acid sequences from 3Di structure strings
• Embeddings: high-quality per-residue representations for both AA and 3Di sequences

What 3Di tokens are: Foldseek encodes protein structure as a 20-letter structural alphabet (one token per residue). This is not 3D coordinates — it's a compact structural language. It enables fast structure-based search and comparison without full coordinate files.

When to use ProstT5 vs ESMFold/ESM3:

• ProstT5 outputs 3Di tokens (useful for Foldseek search, structural alphabet analysis)
• ESMFold/ESM3 outputs 3D coordinates (PDB format, for visualization, MD simulation, etc.)

Installation

pip install transformers torch sentencepiece

Model

HF ID	Notes
Rostlab/ProstT5	Full precision
Rostlab/ProstT5_fp16	Half-precision variant (faster, recommended for GPU)

Critical Preprocessing Rules

ProstT5 uses a shared tokenizer for both AA and 3Di sequences:

Sequence type	Format	Prefix token
Amino acids	UPPERCASE, space-separated, UZOB→X	<AA2fold>
3Di tokens	lowercase, space-separated	<fold2AA>

The prefix token tells the model which direction to translate. It is required — omitting it produces wrong outputs.

import re

def preprocess_aa(sequence: str) -> str:
    """Prepare an amino acid sequence for ProstT5."""
    sequence = re.sub(r"[UZOB]", "X", sequence.upper())
    return "<AA2fold> " + " ".join(list(sequence))

def preprocess_3di(structure: str) -> str:
    """Prepare a 3Di sequence for ProstT5."""
    return "<fold2AA> " + " ".join(list(structure.lower()))

Core Workflows

1. Extract Embeddings (AA sequences)

import re, torch
from transformers import T5Tokenizer, T5EncoderModel

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = T5Tokenizer.from_pretrained("Rostlab/ProstT5", do_lower_case=False)
model = T5EncoderModel.from_pretrained("Rostlab/ProstT5").to(device)
model.half() if device.type == "cuda" else model.float()
model.eval()

sequences = ["MKTAYIAKQRQISFVK", "AGLIVHSPQWFYK"]
processed = [preprocess_aa(s) for s in sequences]
lengths = [len(s) for s in sequences]

ids = tokenizer(processed, return_tensors="pt", padding="longest",
                add_special_tokens=True).to(device)

with torch.no_grad():
    output = model(**ids)

# output.last_hidden_state shape: (batch, prefix+seq_len+EOS, 1024)
# Skip prefix token at index 0; residues are at indices [1 : seq_len+1]
for i, seq_len in enumerate(lengths):
    per_residue = output.last_hidden_state[i, 1 : seq_len + 1].float()  # (L, 1024)
    per_protein  = per_residue.mean(dim=0)                               # (1024,)

Note: Unlike ProtT5 (no prefix), ProstT5 adds a prefix token (<AA2fold> or <fold2AA>) at position 0. Start residue slicing at index 1, not 0.

2. AA → 3Di Translation ("Folding")

Predict structure as a 3Di string from a protein sequence:

from transformers import T5Tokenizer, AutoModelForSeq2SeqLM

tokenizer = T5Tokenizer.from_pretrained("Rostlab/ProstT5", do_lower_case=False)
model = AutoModelForSeq2SeqLM.from_pretrained("Rostlab/ProstT5").to(device)
model.half() if device.type == "cuda" else model.float()
model.eval()

sequences = ["MKTAYIAKQRQISFVK", "AGLIVHSPQWFYK"]
processed = [preprocess_aa(s) for s in sequences]
lengths = [len(s) for s in sequences]

ids = tokenizer(processed, return_tensors="pt", padding="longest",
                add_special_tokens=True).to(device)

gen_config = {
    "do_sample": True,
    "num_beams": 3,
    "top_p": 0.95,
    "temperature": 1.2,
    "top_k": 6,
    "repetition_penalty": 1.2,
}

with torch.no_grad():
    translations = model.generate(
        ids.input_ids,
        attention_mask=ids.attention_mask,
        max_length=max(lengths),
        min_length=min(lengths),
        early_stopping=True,
        num_return_sequences=1,
        **gen_config,
    )

decoded = tokenizer.batch_decode(translations, skip_special_tokens=True)
structure_3di = ["".join(s.split()) for s in decoded]  # remove spaces
print(structure_3di)  # list of lowercase 3Di strings, one per input sequence

The output is a list of 3Di strings (e.g., "adwkcvpmaklfdeg...") — one character per residue, all lowercase.

3. 3Di → AA Translation ("Inverse Folding")

Recover amino acid sequences from 3Di structure strings:

structures_3di = ["adwkcvpmaklfdeg", "strmcvpqklf"]
processed = [preprocess_3di(s) for s in structures_3di]
lengths = [len(s) for s in structures_3di]

ids = tokenizer(processed, return_tensors="pt", padding="longest",
                add_special_tokens=True).to(device)

gen_config_inv = {
    "do_sample": True,
    "top_p": 0.90,
    "temperature": 1.1,
    "top_k": 6,
    "repetition_penalty": 1.2,
}

with torch.no_grad():
    translations = model.generate(
        ids.input_ids,
        attention_mask=ids.attention_mask,
        max_length=max(lengths),
        min_length=min(lengths),
        early_stopping=True,
        num_return_sequences=1,
        **gen_config_inv,
    )

decoded = tokenizer.batch_decode(translations, skip_special_tokens=True)
aa_sequences = ["".join(s.split()) for s in decoded]
print(aa_sequences)  # recovered amino acid sequences (uppercase)

Scripts

# AA → 3Di translation from FASTA
python scripts/translate.py sequences.fasta --direction aa2fold --output structures.fasta

# 3Di → AA inverse folding from FASTA
python scripts/translate.py structures.fasta --direction fold2aa --output recovered.fasta

# Generate embeddings from AA sequences
python scripts/translate.py sequences.fasta --direction embed --output embeddings.npy

When to Use ProstT5 vs. Alternatives

Goal	Recommended tool
3Di tokens for Foldseek search	ProstT5
Sequence similarity from structure	ProstT5 (embed 3Di)
3D coordinates / PDB	ESMFold or ESM3
Generative protein design	ESM3
Pure AA embeddings	ProtT5 or ESM-C

Best Practices

• Use ProstT5_fp16 on GPU for ~2× speed with no quality loss
• Translation is slow (0.6–2.5s per protein) because it uses autoregressive decoding; embedding is fast
• The 3Di output can be directly used as input to Foldseek for structure-based search
• For validation, compare ProstT5's 3Di output against AlphaFold2/ESMFold-predicted structures

Resources

• GitHub: https://github.com/agemagician/ProtTrans
• HuggingFace: https://huggingface.co/Rostlab/ProstT5
• Foldseek: https://github.com/steineggerlab/foldseek
• Preprint: Heinzinger et al., 2023 — https://doi.org/10.1101/2023.07.23.550085
• Training data: https://huggingface.co/datasets/Rostlab/ProstT5Dataset

References

• references/prostt5-api.md — detailed generation configs, batched translation, embedding 3Di sequences, integration with Foldseek