RLAIF Bradley-Terry reward fit (Kamath Ch 5)

MM iteration for the BT strengths over the AI preference graph; a graph that is not strongly connected has no finite MLE and is refused. The pairwise loss comes from morie_kamath_reward_model_training_loss.

Usage

morie_kamath_rlaif_objective(ai_preferences, max_iter = 1000, tol = 1e-12)

Arguments

ai_preferences

List of (winner, loser) pairs.

max_iter, tol

Iteration controls.

Value

List with items, strengths, scores, loss, accuracy.

Examples

morie_kamath_rlaif_objective(list(c("a", "b"), c("b", "c"), c("c",
    "a"), c("a", "c")))
#> $items
#> [1] "a" "b" "c"
#> 
#> $strengths
#> [1] 0.4786203 0.3145962 0.2067835
#> 
#> $scores
#> [1] -0.7368477 -1.1564653 -1.5760830
#> 
#> $loss
#> [1] 0.6419534
#> 
#> $accuracy
#> [1] 0.75
#> 
#> $n_preferences
#> [1] 4
#> 
#> $estimate
#> [1] 0.6419534
#> 
#> $n
#> [1] 3
#> 
#> $method
#> [1] "RLAIF Bradley-Terry reward fit (loss via kmrmloss)"
#>