Optimal KL-regularised policy (Kamath Eq 5.6)

pi_r = pi_ref exp(r/beta) / Z. Z is computed, and a supplied Z is checked rather than trusted.

Usage

morie_kamath_ch5_rlhf_optimal_policy(pi_ref, r, beta, Z = NULL)

Arguments

pi_ref

Reference distribution.

r

Rewards.

beta

Strictly positive.

Z

Optional partition function.

Value

List with pi, Z, argmax (0-based).

Examples

morie_kamath_ch5_rlhf_optimal_policy(c(0.5, 0.5), c(1, 0), 0.5)
#> $pi
#> [1] 0.8807971 0.1192029
#> 
#> $Z
#> [1] 4.194528
#> 
#> $beta
#> [1] 0.5
#> 
#> $estimate
#> [1] 0.8807971
#> 
#> $argmax
#> [1] 0
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "optimal KL-regularised policy (Kamath Eq 5.6)"
#>