KL-penalised RLHF reward (Kamath Eq 5.2)

R = r_theta(x, y) - beta log(pi_RL / pi_SFT).

Usage

morie_kamath_ch5_reward_kl_penalty(x, y, pi_RL, pi_SFT, beta, r_theta = NULL)

Arguments

x, y

Prompt and response.

pi_RL, pi_SFT

Probabilities in (0, 1].

beta

Penalty weight.

r_theta

Reward or function (x, y).

Value

List with estimate, penalised_reward, penalty.

Examples

morie_kamath_ch5_reward_kl_penalty("x", "y", c(0.5, 0.25), c(0.25,
    0.5), 0.2, r_theta = 1)
#> $estimate
#> [1] 1
#> 
#> $penalised_reward
#> [1] 0.8613706 1.1386294
#> 
#> $raw_reward
#> [1] 1
#> 
#> $penalty
#> [1]  0.1386294 -0.1386294
#> 
#> $beta
#> [1] 0.2
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "KL-penalised RLHF reward (Kamath Eq 5.2)"
#>