Pytorch preference optimization

Skill cxcscmu/SkillLearnBench/skills/b1-one-shot-claude-sonnet-4-6/nlp-paper-reproduction/pytorch-preference-optimization

Guide for implementing preference optimization methods (DPO, SimPO, IPO) in PyTorch. Use when implementing loss functions for RLHF-style training.From its SKILL.md

Install

npx -y skills add cxcscmu/SkillLearnBench --skill pytorch-preference-optimization

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

SKILL.md

1.2 KB, 265 tokens by cl100k_base, as published. Nobody here has run it

PyTorch Preference Optimization

Common Pattern

All preference optimization methods take:

policy_chosen_logps: log probs for preferred responses
policy_rejected_logps: log probs for rejected responses

DPO Loss

# Requires reference model log probs
logits = (policy_chosen_logps - ref_chosen_logps) - (policy_rejected_logps - ref_rejected_logps)
losses = -F.logsigmoid(beta * logits)

SimPO Loss (Reference-Free)

# No reference model needed - uses length-normalized log probs
pi_logratios = policy_chosen_logps - policy_rejected_logps  # already avg'd per token
logits = pi_logratios - gamma_beta_ratio  # gamma_beta_ratio = gamma/beta
losses = -F.logsigmoid(beta * logits)

Tips

F.logsigmoid(x) is numerically more stable than torch.log(torch.sigmoid(x))
For label smoothing: loss = -logsigmoid(logits)*(1-ls) - logsigmoid(-logits)*ls
Hinge loss: torch.relu(1 - beta * logits)
Always .detach() reward tensors to prevent gradient flow through them

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.