Skip to contents

BART text-infilling corruption and reconstruction score (Geron Ch 15, hmbart)

Source: R/geron_ml_native.R
morie_geron_bart.Rd

Contiguous spans collapse to a SINGLE <mask> token, so the decoder must recover both content and length. With no model the scorer is an add-one-smoothed bigram LM estimated on the target, giving a genuine cross-entropy in nats per token. spans entries are 0-based c(start, length).

Usage

morie_geron_bart(
  src,
  tgt,
  mask_ratio = 0.3,
  mean_span = 3,
  permute = FALSE,
  model = NULL,
  seed = 0
)

Arguments

src, tgt

Token vectors.

mask_ratio

Fraction in (0, 1).

mean_span

Mean geometric span length, >= 1.

permute

Also rotate the corrupted token list.

model

Optional function(corrupted, tgt) -> per-token log-probs.

seed

LCG seed.

Value

List with corrupted, spans, n_masked, n_spans, loss, perplexity, token_logprobs.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_bart(V, V)
#> $corrupted
#> [1] "1"      "<mask>" "3"      "<mask>" "5"      "6"      "7"      "8"     
#> 
#> $spans
#> $spans[[1]]
#> [1] 1 1
#> 
#> $spans[[2]]
#> [1] 3 1
#> 
#> 
#> $n_masked
#> [1] 2
#> 
#> $n_spans
#> [1] 2
#> 
#> $loss
#> [1] 1.504077
#> 
#> $perplexity
#> [1] 4.5
#> 
#> $token_logprobs
#> [1] -1.504077 -1.504077 -1.504077 -1.504077 -1.504077 -1.504077 -1.504077
#> [8] -1.504077
#> 
#> $estimate
#> [1] 1.504077
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "BART text-infilling corruption scored by seq2seq reconstruction cross-entropy"
#>