Unigram LM tokenizer by EM (Kamath Ch 2)

Forward-backward expected counts over every segmentation, then Viterbi segmentations at the fitted piece probabilities.

Usage

morie_kamath_unigram_lm_tokenizer(corpus, vocab, max_iter = 100, tol = 1e-12)

morie_kamath_unigram_loglik(corpus, probs)

morie_kamath_viterbi_segment(text, probs)

Arguments

corpus

Character vector.

vocab

Candidate pieces.

max_iter, tol

EM controls.

probs

Named list of piece probabilities.

text

One string to segment.

Value

List with probs, log_likelihood, segmentations.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_kamath_unigram_lm_tokenizer(V, V)
#> $probs
#> $probs$`1`
#> [1] 0.125
#> 
#> $probs$`2`
#> [1] 0.125
#> 
#> $probs$`3`
#> [1] 0.125
#> 
#> $probs$`4`
#> [1] 0.125
#> 
#> $probs$`5`
#> [1] 0.125
#> 
#> $probs$`6`
#> [1] 0.125
#> 
#> $probs$`7`
#> [1] 0.125
#> 
#> $probs$`8`
#> [1] 0.125
#> 
#> 
#> $log_likelihood
#> [1] -16.63553
#> 
#> $log_likelihood_history
#> [1] -16.63553 -16.63553
#> 
#> $n_iterations
#> [1] 1
#> 
#> $segmentations
#> $segmentations[[1]]
#> [1] "1"
#> 
#> $segmentations[[2]]
#> [1] "2"
#> 
#> $segmentations[[3]]
#> [1] "3"
#> 
#> $segmentations[[4]]
#> [1] "4"
#> 
#> $segmentations[[5]]
#> [1] "5"
#> 
#> $segmentations[[6]]
#> [1] "6"
#> 
#> $segmentations[[7]]
#> [1] "7"
#> 
#> $segmentations[[8]]
#> [1] "8"
#> 
#> 
#> $vocab_size
#> [1] 8
#> 
#> $estimate
#> [1] -16.63553
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Unigram LM EM (forward-backward expected counts)"
#>