WordPiece merge score (Geron Ch 16, morie.fn grwpc)

score(A, B) = count(AB) / (count(A) count(B)); BPE takes frequency, WordPiece takes surprise. Ties in the ranking break on the string form of the pair.

Usage

morie_geron_wordpiece_tokenizer_score(counts, pairs)

Arguments

counts

Named numeric vector or list of symbol counts.

pairs

List of list(a, b, count) triples (or a named list keyed "a|b").

Value

List with scores, best_pair, best_score, ranking.

Examples

morie_geron_wordpiece_tokenizer_score(list(a = 10, b = 4, c = 6),
    list(list("a", "b", 3), list("b", "c", 4), list("a", "c",
        5)))
#> $scores
#> $scores$`('a', 'b')`
#> [1] 0.075
#> 
#> $scores$`('b', 'c')`
#> [1] 0.1666667
#> 
#> $scores$`('a', 'c')`
#> [1] 0.08333333
#> 
#> 
#> $best_pair
#> [1] "b" "c"
#> 
#> $best_score
#> [1] 0.1666667
#> 
#> $ranking
#> ('b', 'c') ('a', 'c') ('a', 'b') 
#> 0.16666667 0.08333333 0.07500000 
#> 
#> $estimate
#> [1] 0.1666667
#> 
#> $n
#> [1] 3
#>