DPO loss from log-probabilities (Kamath Ch 5)

Bradley-Terry loss on the beta-scaled policy/reference log-ratios; the BT core is morie_alammar_reward_model_bt.

Usage

morie_kamath_dpo_loss(logp_w, logp_l, logp_ref_w, logp_ref_l, beta)

Arguments

logp_w, logp_l, logp_ref_w, logp_ref_l

Log-probabilities (<= 0).

beta

Positive scale.

Value

List with estimate, per_pair, implicit rewards.

Examples

morie_kamath_dpo_loss(-0.5, -1.5, -1, -1, 0.5)
#> $estimate
#> [1] 0.474077
#> 
#> $loss
#> [1] 0.474077
#> 
#> $per_pair
#> [1] 0.474077
#> 
#> $pair_accuracy
#> [1] 1
#> 
#> $implicit_reward_w
#> [1] 0.25
#> 
#> $implicit_reward_l
#> [1] -0.25
#> 
#> $beta
#> [1] 0.5
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "DPO loss (Kamath Ch 5; the Bradley-Terry core reused)"
#>