KL-shaped RLHF reward (Kamath Ch 5)

r_phi - beta * KL; a negative KL or a negative beta is refused.

Usage

morie_kamath_kl_reward_shaping(r_phi, kl_divergence, beta)

Arguments

r_phi

Rewards.

kl_divergence

KL values (scalar broadcasts).

beta

Non-negative weight.

Value

List with shaped, estimate, mean_kl, penalty.

Examples

morie_kamath_kl_reward_shaping(r_phi = c(1, 2, 3, 4, 5, 6, 7, 8),
  kl_divergence = c(1, 2, 3, 4, 5, 6, 7, 8), beta = 0.5)
#> $shaped
#> [1] 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
#> 
#> $estimate
#> [1] 2.25
#> 
#> $mean_reward
#> [1] 4.5
#> 
#> $mean_kl
#> [1] 4.5
#> 
#> $penalty
#> [1] 2.25
#> 
#> $beta
#> [1] 0.5
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "KL-shaped RLHF reward r - beta * KL"
#>