T5 span corruption (Kamath Ch 2) — morie_kamath_t5_span

Spans replaced by one sentinel each. The span layout is drawn from the SAME linear congruential generator Python uses (.morie_al_lcg), so a given seed reproduces the Python output draw for draw.

Usage

morie_kamath_t5_span_corruption(
  tokens,
  mean_span_len = 3,
  corruption_rate = 0.15,
  seed = 0,
  sentinel = "<extra_id_{}>"
)

Arguments

tokens: Token vector.
mean_span_len: Mean span length.
corruption_rate: Fraction masked.
seed: LCG seed.
sentinel: Sentinel template containing %d-style {}.

Value

List with input, target, spans, n_masked, n_spans.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_kamath_t5_span_corruption(V)
#> $input
#> [1] "1"            "2"            "3"            "4"            "5"           
#> [6] "6"            "7"            "<extra_id_0>"
#> 
#> $target
#> [1] "<extra_id_0>" "8"            "<extra_id_1>"
#> 
#> $spans
#> $spans[[1]]
#> [1] "8"
#> 
#> 
#> $n_masked
#> [1] 1
#> 
#> $n_spans
#> [1] 1
#> 
#> $span_lengths
#> [1] 1
#> 
#> $compression
#> [1] 1
#> 
#> $estimate
#> [1] 1
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "T5 span corruption with sentinel tokens"
#>