SentencePiece unigram trainer (Kamath Ch 2) — morie_kamath_sentencepiece

Seed vocabulary, EM through morie_kamath_unigram_lm_tokenizer, then prune by the EXACT likelihood loss of dropping each multi-character piece.

Usage

morie_kamath_sentencepiece_tokenizer(
  corpus,
  vocab_size,
  max_piece_len = 8,
  seed_multiplier = 4,
  shrink = 0.75,
  max_iter = 50
)

Arguments

corpus: Character vector.
vocab_size: Target size.
max_piece_len: Longest seed piece.
seed_multiplier: Seed factor.
shrink: Fraction retained per prune round.
max_iter: EM cap.

Value

List with vocab, probs, log_likelihood, n_prune_rounds.

Examples

morie_kamath_sentencepiece_tokenizer(corpus = c("a", "b", "c"), vocab_size = 5L)
#> $vocab
#> [1] "a" "b" "c"
#> 
#> $probs
#> $probs$a
#> [1] 0.3333333
#> 
#> $probs$b
#> [1] 0.3333333
#> 
#> $probs$c
#> [1] 0.3333333
#> 
#> 
#> $vocab_size
#> [1] 3
#> 
#> $log_likelihood
#> [1] -3.295837
#> 
#> $segmentations
#> $segmentations[[1]]
#> [1] "a"
#> 
#> $segmentations[[2]]
#> [1] "b"
#> 
#> $segmentations[[3]]
#> [1] "c"
#> 
#> 
#> $n_prune_rounds
#> [1] 0
#> 
#> $estimate
#> [1] 3
#> 
#> $n
#> [1] 3
#> 
#> $method
#> [1] "SentencePiece unigram: seed, EM, exact-loss prune"
#>