Reward implied by an optimal policy (Kamath Eq 5.7)

r* = beta log(pi*/pi_ref) + beta log Z.

Usage

morie_kamath_ch5_dpo_reward_optimal(pi_star, pi_ref, beta, Z = NULL)

Arguments

pi_star, pi_ref

Distributions with entries in (0, 1].

beta

Strictly positive.

Z

Partition function; default 1.

Value

List with r, log_ratio, offset.

Examples

morie_kamath_ch5_dpo_reward_optimal(c(0.6, 0.4), c(0.5, 0.5),
    0.2, Z = 2)
#> $r
#> [1] 0.17509375 0.09400073
#> 
#> $log_ratio
#> [1]  0.1823216 -0.2231436
#> 
#> $beta
#> [1] 0.2
#> 
#> $Z
#> [1] 2
#> 
#> $offset
#> [1] 0.1386294
#> 
#> $estimate
#> [1] 0.1750937
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "reward implied by an optimal policy (Kamath Eq 5.7)"
#>