Skip to contents

Learn a fixed number of BPE merges (Geron Ch 14, morie.fn grbpe)

Source: R/geron_ml_native.R
morie_geron_bpe_merge.Rd

Ties are broken by the LEXICOGRAPHICALLY SMALLEST pair – note this is a different rule from morie_geron_bpe_tokenizer(), which uses first appearance; both are deterministic, and they can disagree. Merging stops early when no adjacent pair occurs more than once.

Usage

morie_geron_bpe_merge(corpus, n_merges)

Arguments

corpus

Character vector of words, or a named count vector.

n_merges

Maximum merges, non-negative.

Value

List with merges, vocab, splits, merge_counts, n_tokens_before, n_tokens_after, compression.

Examples

morie_geron_bpe_merge(corpus = c(1, 2, 3, 4, 5, 6, 7, 8), n_merges = 5L)
#> $merges
#> list()
#> 
#> $vocab
#> [1] "1"    "2"    "3"    "4"    "5"    "6"    "7"    "8"    "</w>"
#> 
#> $splits
#> $splits$`1`
#> [1] "1"    "</w>"
#> 
#> $splits$`2`
#> [1] "2"    "</w>"
#> 
#> $splits$`3`
#> [1] "3"    "</w>"
#> 
#> $splits$`4`
#> [1] "4"    "</w>"
#> 
#> $splits$`5`
#> [1] "5"    "</w>"
#> 
#> $splits$`6`
#> [1] "6"    "</w>"
#> 
#> $splits$`7`
#> [1] "7"    "</w>"
#> 
#> $splits$`8`
#> [1] "8"    "</w>"
#> 
#> 
#> $merge_counts
#> numeric(0)
#> 
#> $n_tokens_before
#> [1] 16
#> 
#> $n_tokens_after
#> [1] 16
#> 
#> $compression
#> [1] 1
#> 
#> $estimate
#> [1] 0
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Byte-pair encoding merges"
#>