Top-1 prompt selection accuracy (Kamath Eq 3.8)

A(t) = mean over (x, y) of 1[y = argmax_y' P_LM(y'|x, t)]. A proportion, never a mean of the inputs.

Usage

morie_kamath_ch3_top1_prompt_metric(R, t, P_LM)

Arguments

R

List of list(x, y) pairs.

t

Template.

P_LM

Function (x, t) -> named probability vector.

Value

List with estimate, n_correct, correct.

Examples

morie_kamath_ch3_top1_prompt_metric(list(list("a", "pos"), list("b",
    "neg")), "T1", function(x, t) c(pos = 0.9, neg = 0.1))
#> $estimate
#> [1] 0.5
#> 
#> $n_correct
#> [1] 1
#> 
#> $correct
#> [1] 1 0
#> 
#> $template
#> [1] "T1"
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "top-1 prompt selection accuracy (Kamath Eq 3.8)"
#>