Pairwise reward-model losses (Kamath Eq 5.1, 5.3)

-mean log sigmoid(r_chosen - r_rejected).

Usage

morie_kamath_ch5_reward_loss_pairwise(r_theta, x, y_0, y_1, i)

morie_kamath_ch5_rm_bradley_terry(x, y_w, y_l, r_theta)

Arguments

r_theta

Function (x, y) -> reward.

x

Prompts.

y_0, y_1

Response pairs.

i

Choice, 0 or 1 per pair.

y_w, y_l

Winning and losing responses.

Value

List with estimate, margins, per_pair.

Details

One core serves both Python modules exactly as km067 calls km065: km065 -> morie_kamath_ch5_reward_loss_pairwise (choice vector i, 0 = y_0 wins), km067 -> morie_kamath_ch5_rm_bradley_terry (winner fixed, so i is all zeros).

Examples

A <- matrix(c(1, 0, 2, 1), 2)
B <- diag(2)
x <- c(1, 2, 3, 4)
y <- morie_kamath_ch4_krona_efficient(A, B, x)$y
morie_kamath_ch5_reward_loss_pairwise(function(x, y) c(a = 2,
    b = 0.5)[[y]], list("p1", "p2"), list("a", "b"), list("b",
    "a"), c(0, 1))
#> $estimate
#> [1] 0.2014133
#> 
#> $margins
#> [1] 1.5 1.5
#> 
#> $per_pair
#> [1] 0.2014133 0.2014133
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "pairwise reward-model loss (Kamath Eq 5.1)"
#>