FActScore (Kamath Eq 6.1)

Mean supported-fact fraction over the prompts the model ANSWERED; a NULL response is skipped, not scored 0.

Usage

morie_kamath_ch6_factscore(M, X, A_y, C)

Arguments

M

Function prompt -> response or NULL.

X

Prompts.

A_y

Function response -> atomic facts.

C

Support predicate or a vector of supported facts.

Value

List with estimate, per_prompt, response_rate.

Examples

x <- c(1, 2, 3, 4)
morie_kamath_ch6_factscore(function(x) if (x == "p2") NULL else paste0("r",
    x), list("p1", "p2", "p3"), function(r) c("f1", "f2"), c("f1"))
#> $estimate
#> [1] 0.5
#> 
#> $per_prompt
#> [1] 0.5 0.5
#> 
#> $n_responded
#> [1] 2
#> 
#> $response_rate
#> [1] 0.6666667
#> 
#> $n
#> [1] 3
#> 
#> $method
#> [1] "FActScore (Kamath Eq 6.1)"
#>