PPO-RLHF objective (Kamath Ch 5)

mean(r - beta (log pi_theta - log pi_ref)).

Usage

morie_kamath_ppo_rlhf_objective(rewards, logp_theta, logp_ref, beta)

Arguments

rewards

Rewards.

logp_theta, logp_ref

Log-probabilities (<= 0).

beta

Non-negative weight.

Value

List with estimate, mean_reward, kl_estimate, per_sample.

Examples

morie_kamath_ppo_rlhf_objective(c(1, 2), c(-1, -2), c(-1.5, -1.5),
    0.5)
#> $estimate
#> [1] 1.5
#> 
#> $objective
#> [1] 1.5
#> 
#> $mean_reward
#> [1] 1.5
#> 
#> $kl_estimate
#> [1] 0
#> 
#> $penalty
#> [1] 0
#> 
#> $per_sample
#> [1] 0.75 2.25
#> 
#> $beta
#> [1] 0.5
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "PPO-RLHF objective E[r] - beta * E[log pi/pi_ref]"
#>