Skip to contents

Construct a MORIE tokenizer

Source: R/tokenizer.R
morie_tokenizer_new.Rd

Returns an environment holding tokenizer state (vocab, scores, merges, special-token IDs, optional SentencePiece processor).

Usage

morie_tokenizer_new(
  vocab = NULL,
  scores = NULL,
  merges = NULL,
  bos_id = 1L,
  eos_id = 2L,
  gguf_metadata = NULL,
  sp_model_path = NULL
)

Arguments

vocab

Character vector of vocabulary tokens. Optional.

scores

Numeric vector of vocab scores. Optional.

merges

Character vector of "a b" BPE merge lines. Optional.

bos_id

Beginning-of-sequence integer ID. Default 1.

eos_id

End-of-sequence integer ID. Default 2.

gguf_metadata

Optional named list from a GGUF file (keys tokenizer.ggml.tokens, .scores, .merges, .bos_token_id, .eos_token_id).

sp_model_path

Optional .model file path; loads SentencePiece via reticulate::import("sentencepiece").

Value

Tokenizer environment of class morie_tokenizer.

Examples

vocab <- c("<unk>", "<s>", "</s>", "\u2581", "hi", "world")
morie_tokenizer_new(vocab = vocab, bos_id = 1L, eos_id = 2L)
#> morie_tokenizer(vocab_size=6, source=gguf/vocab)