RLHF reward minus KL objective (Geron Ch 15, morie.fn grrlhf)

J = mean r - beta mean(log pi - log pi_ref).

Usage

morie_geron_rlhf_reward_kl_objective(
  rewards,
  policy_logprobs,
  ref_logprobs,
  beta = 0.1
)

Arguments

rewards

Per-sample rewards.

policy_logprobs, ref_logprobs

Non-positive log-probabilities.

beta

Non-negative KL weight.

Value

List with objective, mean_reward, kl, kl_terms, per_sample.

Examples

if (morie_crypto_liboqs_available()) {
  morie_geron_rlhf_reward_kl_objective(c(1, 0.5, -0.2), c(-0.5,
      -1, -0.3), c(-0.8, -0.9, -0.5), 0.2)
}