RLHF objective and PPO loss (Kamath Eq 5.4-5.5)

E[r] - beta KL(pi_theta || pi_ref), maximised.

Usage

morie_kamath_ch5_rlhf_objective(pi_theta, pi_ref, r_phi, beta)

morie_kamath_ch5_ppo_loss(phi, x, y, r_theta, beta, pi_ref = NULL)

Arguments

pi_theta, pi_ref

Response distributions.

r_phi

Rewards.

beta

KL weight.

phi

List of per-prompt policy distributions.

x

Prompts.

y

Per-prompt response name vectors.

r_theta

Function (x, y) -> reward.

Value

List with estimate, expected_reward, kl, penalty.

Details

One core serves both Python modules exactly as km068 calls km069 per prompt and negates the mean: km069 -> morie_kamath_ch5_rlhf_objective, km068 -> morie_kamath_ch5_ppo_loss.

Examples

morie_kamath_ch5_rlhf_objective(c(y1 = 0.6, y2 = 0.4), c(y1 = 0.5,
    y2 = 0.5), c(1, 0), 0.1)
#> $estimate
#> [1] 0.5979864
#> 
#> $expected_reward
#> [1] 0.6
#> 
#> $kl
#> [1] 0.02013551
#> 
#> $penalty
#> [1] 0.002013551
#> 
#> $beta
#> [1] 0.1
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "RLHF objective (Kamath Eq 5.5)"
#>