Pytorch preference optimization

Skill cxcscmu/SkillLearnBench/skills/b1-one-shot-claude-opus-4-6/nlp-paper-reproduction/pytorch-preference-optimization

[COLM'26] SkillLearnBench is the first benchmark for evaluating continual learning methods that automatically generate agent skills.

Install

npx -y skills add cxcscmu/SkillLearnBench --skill pytorch-preference-optimization

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

What its author says it does

Copied from the file, not written here

PyTorch patterns for implementing preference optimization losses (DPO, SimPO, etc.) for LLM training.

SKILL.md

1.2 KB, 258 tokens by cl100k_base, as published. Nobody here has run it

PyTorch Preference Optimization Patterns

Key Functions

torch.nn.functional.logsigmoid(x): Numerically stable log-sigmoid, used in Bradley-Terry losses.
torch.relu(x): Used for hinge-loss variants.
torch.FloatTensor: Common type for log probability tensors.

Implementation Pattern

import torch
import torch.nn.functional as F

def preference_loss(chosen_logps, rejected_logps, beta, gamma, label_smoothing=0.0, loss_type="sigmoid"):
    logits = beta * (chosen_logps - rejected_logps) - gamma

    if loss_type == "sigmoid":
        losses = -F.logsigmoid(logits) * (1 - label_smoothing) - F.logsigmoid(-logits) * label_smoothing
    elif loss_type == "hinge":
        losses = torch.relu(1 - logits)

    chosen_rewards = beta * chosen_logps.detach()
    rejected_rewards = beta * rejected_logps.detach()

    return losses, chosen_rewards, rejected_rewards

Environment Setup