RLHF pipeline SFT -> RM -> PPO (Kamath Ch 5)

The KL reference is the SFT policy, and that identity is reported.

Usage

morie_kamath_rlhf_pipeline(
  demos,
  preferences,
  pi0,
  sft = NULL,
  train_rm = NULL,
  ppo = NULL
)

Arguments

demos

Demonstrations.

preferences

Preference pairs.

pi0

Base policy.

sft, train_rm, ppo

Stage functions.

Value

List with policy, policy_sft, reward_model, stages.

Examples

morie_kamath_rlhf_pipeline(list("d"), list(c("a", "b")), "pi0",
    sft = function(p, d) "sft", train_rm = function(p) function(y) 1,
    ppo = function(p, r, ref) "rlhf")
#> $policy
#> [1] "rlhf"
#> 
#> $policy_sft
#> [1] "sft"
#> 
#> $reward_model
#> function (y) 
#> 1
#> <environment: 0x562184747aa0>
#> 
#> $kl_reference
#> [1] "sft"
#> 
#> $kl_reference_is_sft
#> [1] TRUE
#> 
#> $stages
#> [1] "sft"          "reward_model" "ppo"         
#> 
#> $n_demos
#> [1] 1
#> 
#> $n_preferences
#> [1] 1
#> 
#> $estimate
#> [1] 3
#> 
#> $n
#> [1] 3
#> 
#> $method
#> [1] "RLHF pipeline SFT -> RM -> PPO (KL anchored to SFT)"
#>