DPO preference probability (Kamath Eq 5.10-5.11)

Usage

morie_kamath_ch5_dpo_pref_simplified(pi_star, pi_ref, beta)

morie_kamath_ch5_dpo_pref_substituted(pi_star, pi_ref, beta, Z = NULL)

Arguments

pi_star, pi_ref: Two-response policies.
beta: Positive.
Z: Partition function carried through; default 1.

Value

List with estimate, margin, implicit rewards.

Details

km075 -> morie_kamath_ch5_dpo_pref_simplified (Z cancelled), km074 -> morie_kamath_ch5_dpo_pref_substituted, which carries Z explicitly and refuses to return unless the beta log Z terms cancel – the same assertion the Python module makes.

Examples

morie_kamath_ch5_dpo_pref_simplified(c(0.6, 0.4), c(0.5, 0.5),
    0.2)
#> $estimate
#> [1] 0.5202622
#> 
#> $margin
#> [1] 0.08109302
#> 
#> $implicit_reward_w
#> [1] 0.03646431
#> 
#> $implicit_reward_l
#> [1] -0.04462871
#> 
#> $beta
#> [1] 0.2
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "DPO preference, Z cancelled (Kamath Eq 5.11)"
#>