Direct preference optimization (DPO) loss (hmdpo)

Usage

morie_geron_dpo(pi, pi_ref, preferences = NULL, beta = 0.1)

Arguments

pi, pi_ref: Log-probs (B, 2) under policy / reference; col 0 = chosen unless preferences given.
preferences: 0/1 winner index per row (default all 0).
beta: Inverse temperature (>0).

Value

List with loss, per_pair_loss, margin, reward_chosen, reward_rejected, accuracy.

Examples

morie_geron_dpo(matrix(c(-1, -2), 1, 2), matrix(c(-1, -2), 1,
    2))
#> $loss
#> [1] 0.6931472
#> 
#> $per_pair_loss
#> [1] 0.6931472
#> 
#> $margin
#> [1] 0
#> 
#> $reward_chosen
#> [1] 0
#> 
#> $reward_rejected
#> [1] 0
#> 
#> $prob_preferred
#> [1] 0.5
#> 
#> $accuracy
#> [1] 0
#> 
#> $beta
#> [1] 0.1
#> 
#> $estimate
#> [1] 0.6931472
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "DPO loss -log sigmoid(beta*(log pi/pi_ref)_w - beta*(log pi/pi_ref)_l)"
#>