Encode text with a trained BPE model
Usage
encode_bpe(text, model, add_prefix = TRUE)
Arguments
- text
See Usage.
- model
See Usage.
- add_prefix
See Usage.
Examples
bpe <- train_bpe(c("low lower lowest", "new newer newest"), vocab_size = 30)
encode_bpe("lowest newer", bpe)
#> [1] "▁lowest" "▁newer"
