Preference-based reward learning (Kamath Ch 5)

Scores trajectory segment pairs with r_phi and hands the Bradley-Terry loss to morie_alammar_reward_model_bt.

Usage

morie_kamath_christiano_deep_rl_feedback(trajectory_pairs, r_phi)

Arguments

trajectory_pairs

List of (sigma_w, sigma_l) pairs.

r_phi

Return function.

Value

List with estimate (summed loss), mean_loss, pair_accuracy.

Examples

morie_kamath_christiano_deep_rl_feedback(list(c("w1", "l1"),
    c("w2", "l2")), function(s) if (startsWith(s, "w")) 2 else 0.5)
#> $estimate
#> [1] 0.4028266
#> 
#> $mean_loss
#> [1] 0.2014133
#> 
#> $losses
#> [1] 0.2014133 0.2014133
#> 
#> $pair_accuracy
#> [1] 1
#> 
#> $returns_preferred
#> [1] 2 2
#> 
#> $returns_rejected
#> [1] 0.5 0.5
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "preference-based reward learning (Kamath Ch 5)"
#>