Generative design

Version Compatibility

Reference examples tested with: REINVENT 4.0+, RDKit 2024.09+, PyTorch 2.1+, MolMIM (NVIDIA BioNeMo), chemprop 2.0+.

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Generative Molecular Design

Generate novel molecules biased toward desired properties using deep generative models. REINVENT 4 (Loeffler et al. 2024, AstraZeneca) provides four generator families: Reinvent (de novo), Libinvent (scaffold decoration and library design), Linkinvent (linker design), and Mol2Mol (similarity-constrained molecular optimization). These support design tasks including R-group replacement and scaffold hopping and can be used with transfer learning, reinforcement learning, and curriculum learning. For specific niches: MolMIM (NVIDIA BioNeMo) for latent-space property optimization, DiffSMol / DiGress for diffusion-based generation, and JT-VAE for latent-space optimization. The art of generative design is in the scoring function: poorly-designed scoring rewards uninteresting molecules, while well-designed scoring captures both activity and developability.

For QSAR/scoring models that feed generative design, see chemoinformatics/qsar-modeling. For synthetic feasibility, see chemoinformatics/retrosynthesis. For library enumeration as alternative, see chemoinformatics/reaction-enumeration.

Generator Mode Taxonomy

Mode	Input	Output	Use case	Fails when
De novo	Empty seed or training set	Novel molecules	Wide chemical space exploration	Synthetic feasibility weak
Scaffold decoration	Scaffold + attachment points	Decorated molecules	Series expansion	Generation diversity limited by scaffold
Linker design	2 fragments	Linker molecules	PROTAC, ternary complex	Few linker geometric options
R-group replacement	Scaffold + existing R-groups	New R-group set	Optimize one position	Single-position only
Molecular optimization	Lead molecule	Improved analogs	Lead optimization	Improvement window narrow
Constrained generation	Hard constraints (MW, fragments)	Compliant molecules	Patent / IP design	Constraints overly restrictive

Learning Algorithm Taxonomy

Algorithm	Use	Pro	Con
Transfer learning (TL)	Adapt prior model to focused training set	Stable, simple	Limited optimization power
Reinforcement learning (RL)	Reward-driven generation	Powerful for MPO	Reward hacking risk
Curriculum learning (CL)	Gradual constraint introduction	Better convergence	Slower; tuning sensitive

Decision Tree by Scenario

Scenario	Generator	Algorithm	Scoring
New target, no SAR	De novo	Benchmark RL against simpler search	Validated target evidence + developability objectives
Series expansion	Scaffold decoration	TL on series + RL	QSAR ensemble + QED
PROTAC linker	Linker design	Project-specific constrained workflow	Validated geometry/ternary-complex evidence; no generic DC50 surrogate
Lead optimization MPO	Molecular optimization	CL with staged constraints	Multi-task: activity + ADMET
Diverse hit set	De novo with diversity bonus	RL + Tanimoto distance to known	Activity + diversity
Patent space carve-out	Constrained de novo	RL + structural constraints	Activity + novelty
Hit-to-lead	R-group replacement	TL on lead + RL	Activity + Lipinski
ADMET-aware design	De novo or optimization	RL	hERG + CYP + AMES + QED

REINVENT 4 Setup

REINVENT 4 uses a TOML configuration file specifying generator, algorithm, prior model, and scoring functions.

Goal: Configure a reinforcement-learning REINVENT 4 run with a prior, agent, sampling parameters, and a QED scoring component.

Approach: Build a release-matched REINVENT 4 staged-learning TOML config with [parameters] for the prior/agent checkpoints, [[stage]] blocks, and one or more [[stage.scoring.component]] blocks. Validate the config with the installed release because component parameters evolve between versions.

run_type = "staged_learning"
device = "cuda:0"

[parameters]
prior_file = "priors/reinvent.prior"
agent_file = "priors/reinvent.prior"
batch_size = 64
unique_sequences = true

[[stage]]
termination = "simple"
min_steps = 25
max_steps = 500

[stage.scoring]
type = "geometric_mean"

[[stage.scoring.component]]
[stage.scoring.component.QED]

[[stage.scoring.component.QED.endpoint]]
name = "QED"
weight = 1

# The REINVENT 4 CLI binary is `reinvent` (not `reinvent4`).
reinvent -l logfile.log config.toml

Output: a live stage CSV using summary_csv_prefix, plus the configured chkpt_file at stage termination or graceful interruption. Post-process the CSV to select molecules; REINVENT does not emit a checkpoint and SMILES file at every iteration by default.

Scoring Function Design (Most Important Part)

A good scoring function:

• Returns 0-1 (normalized)
• Combines multiple endpoints
• Penalizes pathological generations (PAINS, unstable, unsynthesizable)

Goal: Build a multi-component generative reward that balances predicted activity, drug-likeness, synthesizability, and novelty.

Approach: Combine a QSAR sigmoid on pIC50, QED, SA-score reverse-sigmoid, and Tanimoto-similarity reverse-sigmoid via geometric mean so any zero component zeroes the total.

In REINVENT 4, define these under the active stage's [stage.scoring] section, with each component using the exact component and endpoint tables from the installed release's configuration examples. Do not reuse REINVENT 3 [scoring_function] or [[scoring_function.components]] syntax in a REINVENT 4 config. The accompanying example is deliberately limited to built-in, documented component structure; add predictive-property endpoints only after validating their release-specific model-container parameters.

geometric_mean ensures all components must be reasonably high (one zero -> zero total). arithmetic_mean allows compensation.

Multi-Parameter Optimization (MPO)

Lead optimization commonly involves multiple objectives. The following component types illustrate a project-specific scoring design; weights and transforms must be fit to the actual assays and decision context.

Component	Weight	Transformation
Target activity (predicted pIC50)	0.3	sigmoid 5-8
Selectivity (off-target ratio)	0.2	sigmoid 1-100
QED	0.1	identity
Synthetic accessibility (SA score)	0.1	reverse sigmoid 1-4
hERG predicted prob	0.1	reverse sigmoid 0.3-0.7
AMES predicted prob	0.1	reverse sigmoid 0.3-0.7
Tanimoto novelty vs known	0.1	reverse sigmoid 0.4-0.6

The weights and transformation bounds above are repository starting examples only. Normalize weights as required by the selected aggregation and tune every bound against project assay distributions and prospective behavior.

Reward Hacking (Production Pitfall)

RL agents will find ways to maximize reward without learning the intended behavior:

• Trivial scaffolds that score high on QED
• Repeat structural motifs that game similarity scoring
• Out-of-distribution molecules that exploit QSAR overconfidence
• Trivial SMILES (e.g., "CCC...C") that match generic scoring

Mitigations:

• Include one or more synthesis-aware signals when they have been validated for the project; SA score alone does not establish route feasibility
• Use ensemble QSAR with uncertainty (penalize high-uncertainty predictions)
• Include diversity bonus (Tanimoto to reference)
• Add fingerprint similarity penalty within batch (prevent mode collapse)
• Validate generated samples on held-out QSAR test set

Synthetic Accessibility Scoring

sa_score (Ertl 2009) measures synthetic accessibility: 1 (easy) to 10 (very hard).

from rdkit.Contrib.SA_Score import sascorer
from rdkit import Chem

def sa_score(smi):
    mol = Chem.MolFromSmiles(smi)
    if mol is None:
        return None
    return sascorer.calculateScore(mol)

sascorer is shipped in RDKit Contrib in current RDKit distributions. Use the namespaced import above; do not install an unrelated top-level package.

The score is a 1-to-10 heuristic derived from fragment contributions and molecular complexity, with lower values intended to indicate easier synthesis. It is not a route planner, cost estimate, or calibrated feasibility probability. Use it as one audited reward component or annotation, never as an absolute filter.

Diffusion-Based Generation (Modern Alternatives)

Tool	Approach	Strength	Status
DiGress (Vignac 2023)	Discrete diffusion on graphs	Conditional generation	Public
DiffSMol (Chen 2025)	Equivariant diffusion	3D molecule generation	Public
MolDiff (Peng 2023)	Full-atom diffusion	Joint atom/bond generation	Public
TargetDiff (Guan 2023)	Pocket-conditioned equivariant diffusion	Structure-based design	Public

Diffusion models iteratively denoise molecular representations, whereas REINVENT generators autoregressively construct SMILES. Diversity, validity, and drug-likeness depend on the model, training data, conditioning, and evaluation protocol; compare them on a matched benchmark for the intended task.

Constrained / Goal-Directed Generation

Goal: Enforce hard structural requirements (e.g., must contain hydroxyl) and exclude PAINS without letting constraint satisfaction game the reward.

Approach: Stage transfer learning then RL. In REINVENT 4, CustomAlerts is a global structural-alert filter: a match produces zero and it is applied before score aggregation. MatchingSubstructure is a scoring component (1 for a match and 0.5 otherwise), so it is a soft penalty rather than a hard inclusion constraint. Apply a separate post-generation SMARTS validation step when presence of a feature is mandatory.

[[stage.scoring.component]]
[stage.scoring.component.CustomAlerts]

[[stage.scoring.component.CustomAlerts.endpoint]]
name = "Unwanted SMARTS"
params.smarts = ["PAINS_SMARTS_1", "BRENK_SMARTS_1"]

[[stage.scoring.component]]
[stage.scoring.component.MatchingSubstructure]

[[stage.scoring.component.MatchingSubstructure.endpoint]]
name = "Hydroxyl preference"
weight = 0.1
params.smarts = ["[OX2H]"]

There is no REINVENT 4 filter_only option for these components. Treat structural alerts as triage flags where appropriate, and separately verify any true hard inclusion or exclusion rule on the generated structures.

MolMIM (NVIDIA BioNeMo)

MolMIM encodes SMILES into a learned latent space, uses gradient-free CMA-ES to optimize a user-defined property objective, and decodes candidate molecules.

# Pseudo-code; requires NVIDIA NIM access
# from bionemo.molmim import MolMIMOptimizer
# optimizer = MolMIMOptimizer(model="molmim-property-optimizer")
# optimized = optimizer.optimize(seed_smiles, target_property="logp", target_value=2.0)

Tradeoffs against REINVENT depend on the oracle budget, objective, and implementation; benchmark both under matched constraints when selecting a generator.

Per-Tool Failure Modes

REINVENT RL -- mode collapse

Trigger: The reward, learning strategy, or diversity control favors a narrow chemotype.

Mechanism: Agent finds a high-scoring local maximum and stops exploring.

Symptom: Diversity and scaffold coverage collapse relative to a project-defined baseline while reward continues to rise.

Fix: Add diversity bonus to scoring; reduce sigma; reset agent if collapsed.

REINVENT TL -- overfitting

Trigger: Transfer learning data are too small or homogeneous for the intended generalization task.

Mechanism: Generator memorizes training set; no generalization.

Symptom: Generated molecules near-identical to training set actives.

Fix: Use larger training set; mix with diverse external sample; apply RL after TL.

Generated molecule unsynthesizable

Trigger: SA score missing from reward.

Mechanism: The reward omits synthesis evidence, allowing candidates with no plausible validated route to score well.

Symptom: AiZynthFinder cannot solve route; medchem rejects.

Fix: Combine audited synthesis-aware annotations with reaction- or route-based validation for selected candidates.

PAINS in generation

Trigger: No structural alerts in scoring.

Mechanism: Curcumin / rhodanine / quinone scaffolds optimize for activity (false positives in training data).

Symptom: Generated molecules match PAINS_A.

Fix: Flag relevant structural alerts for orthogonal assay review or apply a project-justified penalty; never reward a PAINS match.

Diffusion model OOD

Trigger: Pocket-conditioned diffusion on novel target family.

Mechanism: Training distribution covered specific protein families; novel targets extrapolate.

Symptom: Generated molecules look like training distribution, not optimized for target.

Fix: Validate on target-family-held-out evaluation; supplement with classical methods.

Validation set leakage

Trigger: Same molecules in training generators and downstream QSAR.

Mechanism: Scoring model has seen the molecule; predictions optimistic.

Symptom: Held-out QSAR validation fails on top generated.

Fix: Use scaffold-split QSAR; ensure scoring model trained on a held-out set vs generation samples.

Reconciliation: REINVENT vs Diffusion

Aspect	REINVENT 4	Diffusion
Speed	Implementation-, hardware-, and oracle-dependent	Implementation-, hardware-, and sampling-step-dependent
Output diversity	Model/training/reward-dependent	Model/training/conditioning-dependent
Drug-likeness of output	Training- and reward-dependent	Training- and conditioning-dependent
Scoring flexibility	Excellent (TOML config)	Method-specific
Production maturity	High	Emerging
When to use	When its priors and scoring system validate well for the task	When a matched benchmark supports the selected diffusion model

Common Errors

Symptom	Cause	Fix
REINVENT generates invalid SMILES	Prior/tokenization/model mismatch or sampling issue	Inspect invalid-token logs, prior compatibility, and release-matched sampling settings; sigma is not a token sampling rate
QSAR score all 0.0	Out-of-domain molecules	Ensemble + uncertainty; reject high-uncertainty
All generations duplicates	unique_sequences=False	Set unique_sequences=true
Generated SMILES too long	Prior/model sequence behavior	Use a release-documented sampling constraint or post-generation project rule; do not invent a staged-learning max_length field
Reward stuck at 0.5	Constraints conflict	Inspect scoring components; reduce constraint count
Diffusion model crashes	Input violates model-specific pocket/size contract	Follow that model release's documented preprocessing and limits
MolMIM cold-start slow	Latent search exhaustiveness	Reduce search budget
Optimization converges trivially	Reward gradient dominated by one term	Use geometric_mean; rebalance weights

References

• Loeffler et al., J. Cheminformatics 16:20 (2024) -- REINVENT 4 framework and four generator families (DOI 10.1186/s13321-024-00812-5).
• Olivecrona M et al., J. Cheminformatics 9:48 (2017) -- REINVENT original (DOI 10.1186/s13321-017-0235-x).
• Vignac et al., ICLR (2023) -- DiGress discrete diffusion.
• Chen H et al., Nat. Mach. Intell. 7:758-770 (2025) -- DiffSMol structure-based 3D molecular generation (DOI 10.1038/s42256-025-01030-w).
• Peng X, Guan J, Liu Q, Ma J. Proc. ICML, PMLR 202:27611-27629 (2023) -- MolDiff full-atom molecular diffusion.
• Guan J et al., ICLR (2023) -- TargetDiff pocket-conditioned 3D equivariant diffusion (OpenReview: kJqXEPXMsE0).
• Reidenbach D, Livne M, Ilango RK, Gill M, Israeli J. MLDD Workshop at ICLR (2023) -- MolMIM and CMA-ES latent-space optimization (OpenReview: iOJlwUTUyrN).
• Jin W, Barzilay R, Jaakkola T. Proc. ICML, PMLR 80:2323-2332 (2018) -- JT-VAE junction-tree.
• Ertl P, Schuffenhauer A. J. Cheminformatics 1:8 (2009) -- SA score (DOI 10.1186/1758-2946-1-8).
• REINVENT 4 official repository and release-matched configs: https://github.com/MolecularAI/REINVENT4
• RDKit SA-score implementation: https://github.com/rdkit/rdkit/tree/master/Contrib/SA_Score

Related Skills

• chemoinformatics/qsar-modeling - Build scoring models for generative
• chemoinformatics/retrosynthesis - Validate synthetic feasibility post-generation
• chemoinformatics/molecular-standardization - Standardize generated SMILES
• chemoinformatics/admet-prediction - ADMET in scoring components
• chemoinformatics/substructure-search - PAINS / BRENK filter for generation
• chemoinformatics/scaffold-analysis - Scaffold-aware generation control
• chemoinformatics/reaction-enumeration - Alternative to generative for combinatorial
• chemoinformatics/virtual-screening - Validate generated against target