Train a BPE model: merge the most frequent adjacent pair greedily — train_bpe • rmorie

Train a BPE model: merge the most frequent adjacent pair greedily

Usage

train_bpe(corpus, vocab_size, add_prefix = TRUE)

Arguments

corpus: See Usage.
vocab_size: See Usage.
add_prefix: See Usage.

Examples

bpe <- train_bpe(c("low lower lowest", "new newer newest"), vocab_size = 30)
names(bpe)
#> [1] "merges"     "vocab"      "vocab_size" "requested"  "algorithm" 
#> [6] "note"