WordPiece subword tokenizer — Wpiece • rmorie

Trains subword merges by unigram-likelihood gain: the pair A, B maximising freq(AB) / (freq(A) freq(B)) is merged, continuation pieces carry the double-hash prefix, and segmentation is greedy longest-match-first. There is exactly one implementation: this function delegates to morie_geron_wordpiece_tokenizer.

Usage

Wpiece(corpus, vocab_size = 50)

Arguments

corpus: Character vector of training text.
vocab_size: Target vocabulary size. Default 50.

Value

List with vocab, merges, scores, tokenize, alphabet, estimate, n, method.

References

Schuster, M. and Nakajima, K. (2012), Japanese and Korean voice search, IEEE ICASSP 2012, 5149-5152. Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2019), BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL-HLT 2019, arXiv:1810.04805. Wu, Y. et al. (2016), Googles Neural Machine Translation System, arXiv:1609.08144, Section 4.1.

Examples

Wpiece("hug hug hugs pug pun", vocab_size = 14)$estimate
#> [1] 14