Summarisation from human feedback (Kamath Ch 5)

Combines morie_kamath_reward_model_training_loss on the summary pairs with morie_kamath_ppo_rlhf_objective on the policy batch.

Usage

morie_kamath_summarize_from_feedback(
  preferences,
  rewards,
  pi_logprobs,
  ref_logprobs,
  beta
)

Arguments

preferences

List of (score_chosen, score_rejected).

rewards, pi_logprobs, ref_logprobs

PPO batch.

beta

KL weight.

Value

List with loss_rm, rm_accuracy, objective, kl_estimate.

Examples

morie_kamath_summarize_from_feedback(list(c(2, 1), c(1, 2)),
    c(1, 2), c(-1, -2), c(-1.5, -1.5), 0.5)
#> $loss_rm
#> [1] 0.8132617
#> 
#> $rm_accuracy
#> [1] 0.5
#> 
#> $rm_mean_margin
#> [1] 0
#> 
#> $objective
#> [1] 1.5
#> 
#> $mean_reward
#> [1] 1.5
#> 
#> $kl_estimate
#> [1] 0
#> 
#> $beta
#> [1] 0.5
#> 
#> $n_preferences
#> [1] 2
#> 
#> $estimate
#> [1] 1.5
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "Summarisation from human feedback (kmrmloss + kmppok)"
#>