Pytorch preference optimization

Skill cxcscmu/SkillLearnBench/skills/b1-one-shot-claude-sonnet-4-6/nlp-paper-reproduction/pytorch-preference-optimization

[COLM'26] SkillLearnBench is the first benchmark for evaluating continual learning methods that automatically generate agent skills.

Install

npx -y skills add cxcscmu/SkillLearnBench --skill pytorch-preference-optimization

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

What its author says it does

Copied from the file, not written here

Guide for implementing preference optimization methods (DPO, SimPO, IPO) in PyTorch. Use when implementing loss functions for RLHF-style training.

SKILL.md

1.2 KB, 265 tokens by cl100k_base, as published. Nobody here has run it

PyTorch Preference Optimization

Common Pattern

All preference optimization methods take:

policy_chosen_logps: log probs for preferred responses
policy_rejected_logps: log probs for rejected responses

DPO Loss

# Requires reference model log probs
logits = (policy_chosen_logps - ref_chosen_logps) - (policy_rejected_logps - ref_rejected_logps)
losses = -F.logsigmoid(beta * logits)

SimPO Loss (Reference-Free)

# No reference model needed - uses length-normalized log probs
pi_logratios = policy_chosen_logps - policy_rejected_logps  # already avg'd per token
logits = pi_logratios - gamma_beta_ratio  # gamma_beta_ratio = gamma/beta
losses = -F.logsigmoid(beta * logits)

Tips

F.logsigmoid(x) is numerically more stable than torch.log(torch.sigmoid(x))
For label smoothing: loss = -logsigmoid(logits)*(1-ls) - logsigmoid(-logits)*ls
Hinge loss: torch.relu(1 - beta * logits)
Always .detach() reward tensors to prevent gradient flow through them