DPO loss (Kamath Eq 5.12)

-mean log sigmoid of the beta-scaled log-ratio margin.

Usage

morie_kamath_ch5_dpo_loss(pi_theta, pi_ref, beta)

Arguments

pi_theta, pi_ref

Lists of two-response policies.

beta

Positive.

Value

List with estimate, margins, implicit rewards.

Examples

morie_kamath_ch5_dpo_loss(list(c(0.6, 0.4), c(0.7, 0.3)), list(c(0.5,
    0.5), c(0.5, 0.5)), 0.2)
#> $estimate
#> [1] 0.6327126
#> 
#> $margins
#> [1] 0.08109302 0.16945957
#> 
#> $per_pair
#> [1] 0.6534225 0.6120027
#> 
#> $implicit_reward_w
#> [1] 0.03646431 0.06729445
#> 
#> $implicit_reward_l
#> [1] -0.04462871 -0.10216512
#> 
#> $beta
#> [1] 0.2
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "DPO loss (Kamath Eq 5.12)"
#>