chronos_tokenize

A step of the chronos_native implementation. Called by morie_chronos. See the file header for the source the module follows. the source it follows.

Usage

chronos_tokenize(x, bins, context = NULL, add_eos = TRUE, pad_to = NULL)

Arguments

x

Passed to chronos_mean_scale.

bins

A list; the body reads $n_bins from it.

context

Passed to chronos_mean_scale.

add_eos

A flag; the body branches on it. Defaults to TRUE.

pad_to

Optional; may be NULL. Coerced to integer by the body, with as.integer.

Value

A list with estimate, tokens, scale, n_clipped, clipped_fraction, vocab_size, method, ignores.

Examples

bins <- chronos_uniform_bins(lo = -15, hi = 15, n_bins = 64L)
chronos_tokenize(c(10, 12, 9, 11, 13), bins)$tokens
#> [1] 33 34 33 34 34 -2