Byte-pair encoding to a target vocabulary size (Geron Ch 14, hmbpet)

Ties on pair frequency are broken by FIRST APPEARANCE in corpus order, so the merge sequence is deterministic. Merging stops at the target vocabulary size or when no pair occurs more than once; when the target is below the base-symbol count no merge is performed at all.

Usage

morie_geron_bpe_tokenizer(corpus, vocab_size = 100)

Arguments

corpus: A single string, a character vector, or a named integer vector / list of word -> count.
vocab_size: Target size, counting base symbols.

Value

List with merges (list of c(a, b)), vocab, tokenize, word_tokens, n_merges.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_bpe_tokenizer(V)
#> $merges
#> list()
#> 
#> $vocab
#>    1 </w>    2    3    4    5    6    7    8 
#>    1    8    1    1    1    1    1    1    1 
#> 
#> $tokenize
#> function (word) 
#> {
#>     ss <- c(strsplit(as.character(word), "")[[1]], EOW)
#>     for (mg in ordered_merges) {
#>         a <- mg[1]
#>         b <- mg[2]
#>         out <- character(0)
#>         i <- 1L
#>         while (i <= length(ss)) {
#>             if (i + 1L <= length(ss) && ss[i] == a && ss[i + 
#>                 1L] == b) {
#>                 out <- c(out, paste0(a, b))
#>                 i <- i + 2L
#>             }
#>             else {
#>                 out <- c(out, ss[i])
#>                 i <- i + 1L
#>             }
#>         }
#>         ss <- out
#>     }
#>     ss
#> }
#> <environment: 0x56214a675490>
#> 
#> $word_tokens
#> $word_tokens$`1`
#> [1] "1"    "</w>"
#> 
#> $word_tokens$`2`
#> [1] "2"    "</w>"
#> 
#> $word_tokens$`3`
#> [1] "3"    "</w>"
#> 
#> $word_tokens$`4`
#> [1] "4"    "</w>"
#> 
#> $word_tokens$`5`
#> [1] "5"    "</w>"
#> 
#> $word_tokens$`6`
#> [1] "6"    "</w>"
#> 
#> $word_tokens$`7`
#> [1] "7"    "</w>"
#> 
#> $word_tokens$`8`
#> [1] "8"    "</w>"
#> 
#> 
#> $n_merges
#> [1] 0
#> 
#> $estimate
#> [1] 9
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Byte-pair encoding with greedy most-frequent-pair merges"
#>