T5 span-corruption masking (encoder input / decoder target / spans)

T5 span-corruption masking (encoder input / decoder target / spans)

Usage

morie_geron_span_corrupt(tokens, noise_density = 0.15, mean_span = 3, seed = 0)

Arguments

tokens

Character vector of tokens.

noise_density

Fraction of tokens to mask, in (0,1).

mean_span

Mean span length (>=1).

seed

LCG seed for span placement.

Value

list(inputs, target, spans) where spans is a list of c(start0, length) pairs (0-based start).

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_span_corrupt(V)
#> $inputs
#> [1] "1"            "<extra_id_0>" "3"            "4"            "5"           
#> [6] "6"            "7"            "8"           
#> 
#> $target
#> [1] "<extra_id_0>" "2"            "<extra_id_1>"
#> 
#> $spans
#> $spans[[1]]
#> [1] 1 1
#> 
#>