Skip to contents

Masked language modelling objective (Geron Ch 15, morie.fn hmmlm)

Source: R/geron_w4b_native.R
morie_geron_masked_lm.Rd

Mask positions via the shared LCG stream, not numpy PCG64 choice.

Usage

morie_geron_masked_lm(
  X,
  mask_frac = 0.15,
  model = NULL,
  vocab_size = NULL,
  seed = 0,
  mask_token = -1
)

Arguments

X

Integer token ids (0-based), vector or matrix.

mask_frac

Fraction masked.

model

Optional function(masked_X, positions) -> probability matrix.

vocab_size

Optional vocab size.

seed

Seed.

mask_token

Substituted value.

Value

List with loss, baseline_loss, perplexity, masked_positions, targets, probabilities, n_masked, estimate, n, method.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_masked_lm(V)
#> $loss
#> [1] 2.772589
#> 
#> $baseline_loss
#> [1] 2.772589
#> 
#> $perplexity
#> [1] 16
#> 
#> $masked_positions
#> [1] 0
#> 
#> $masked_input
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]   -1    2    3    4    5    6    7    8
#> 
#> $targets
#> [1] 1
#> 
#> $probabilities
#>        [,1]   [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]  [,9]
#> [1,] 0.0625 0.0625 0.125 0.125 0.125 0.125 0.125 0.125 0.125
#> 
#> $n_masked
#> [1] 1
#> 
#> $vocab_size
#> [1] 9
#> 
#> $estimate
#> [1] 2.772589
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Masked language modelling objective"
#>