Direct Preference Optimization loss (Geron Ch 15, morie.fn grdpo)

L = mean log(1 + exp(-beta * [(lp_w - ref_w) - (lp_l - ref_l)])).

Usage

morie_geron_dpo_loss(logp_w, logp_l, logp_ref_w, logp_ref_l, beta = 0.1)

Arguments

logp_w, logp_l, logp_ref_w, logp_ref_l

Non-positive log-probability vectors of equal length.

beta

Positive inverse temperature.

Value

List with loss, margin, implicit_reward_chosen, implicit_reward_rejected, accuracy, per_pair_loss.

Examples

morie_geron_dpo_loss(logp_w = -1.2, logp_l = -2.5,
                     logp_ref_w = -1.4, logp_ref_l = -2.2, beta = 0.1)
#> $loss
#> [1] 0.6684596
#> 
#> $margin
#> [1] 0.5
#> 
#> $implicit_reward_chosen
#> [1] 0.02
#> 
#> $implicit_reward_rejected
#> [1] -0.03
#> 
#> $accuracy
#> [1] 1
#> 
#> $per_pair_loss
#> [1] 0.6684596
#> 
#> $beta
#> [1] 0.1
#> 
#> $estimate
#> [1] 0.6684596
#> 
#> $n
#> [1] 1
#>