RoBERTa pretraining with dynamic masking (Geron Ch 15, morie.fn hmbrob)

Same encoder stack as BERT with two deliberate differences: no NSP head (so n_params is d_model*2 smaller than n_params_with_nsp), and a fresh mask drawn every epoch instead of frozen in preprocessing. masks records sequence 0's positions per epoch, 0-based.

Usage

morie_geron_roberta(
  X,
  n_layers = 2,
  n_heads = 2,
  d_model = 8,
  vocab_size = NULL,
  d_ff = NULL,
  mask_prob = 0.15,
  epochs = 4,
  seed = 0
)

Arguments

X: 0-based token ids.
n_layers, n_heads, d_model, d_ff: Geometry.
vocab_size: Optional.
mask_prob: Fraction in (0, 1).
epochs: Dynamic-masking passes.
seed: LCG seed.

Value

List with hidden, mlm_loss, epoch_losses, masks, dynamic_masking, has_nsp_head, n_params, n_params_with_nsp, n_masked_per_sequence.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_roberta(V)
#> $hidden
#> , , 1
#> 
#>          [,1]       [,2]     [,3]     [,4]      [,5]       [,6]      [,7]
#> [1,] 1.069966 -0.7569928 1.021947 1.627556 0.8491147 -0.5698388 0.8043986
#>           [,8]
#> [1,] -1.873071
#> 
#> , , 2
#> 
#>           [,1]      [,2]      [,3]      [,4]     [,5]      [,6]       [,7]
#> [1,] 0.5474662 0.8162053 0.5157833 0.1780007 2.167051 0.7581321 -0.6873425
#>            [,8]
#> [1,] -0.1482597
#> 
#> , , 3
#> 
#>          [,1]      [,2]      [,3]      [,4]      [,5]     [,6]     [,7]
#> [1,] -0.33544 0.6509668 0.1834938 -1.670379 -0.626739 1.014835 1.787928
#>           [,8]
#> [1,] 0.2261342
#> 
#> , , 4
#> 
#>           [,1]      [,2]       [,3]       [,4]      [,5]      [,6]     [,7]
#> [1,] 0.6407387 -1.203375 -0.6187581 -0.7195226 -1.240238 -1.337315 1.021377
#>          [,8]
#> [1,] 1.252404
#> 
#> , , 5
#> 
#>           [,1]     [,2]      [,3]        [,4]       [,5]      [,6]       [,7]
#> [1,] -1.676341 1.288473 -1.881617 0.009789788 -0.0248345 0.5351912 -0.7137213
#>            [,8]
#> [1,] -0.3902007
#> 
#> , , 6
#> 
#>            [,1]       [,2]      [,3]     [,4]       [,5]     [,6]       [,7]
#> [1,] -0.5412649 -0.5075518 -1.033252 1.292222 -0.7320585 1.483954 -0.6213946
#>           [,8]
#> [1,] 0.7252707
#> 
#> , , 7
#> 
#>           [,1]      [,2]     [,3]       [,4]        [,5]      [,6]      [,7]
#> [1,] -1.053759 -1.354164 1.012233 -0.5637267 -0.05767872 -1.034818 -1.284992
#>            [,8]
#> [1,] -0.9329709
#> 
#> , , 8
#> 
#>          [,1]     [,2]    [,3]       [,4]       [,5]       [,6]      [,7]
#> [1,] 1.348634 1.066438 0.80017 -0.1539398 -0.3346173 -0.8501418 -0.306253
#>          [,8]
#> [1,] 1.140693
#> 
#> 
#> $mlm_loss
#> [1] 2.272476
#> 
#> $epoch_losses
#> [1] 2.189068 2.349325 2.349325 2.272476
#> 
#> $masks
#> $masks[[1]]
#> [1] 5
#> 
#> $masks[[2]]
#> [1] 1
#> 
#> $masks[[3]]
#> [1] 1
#> 
#> $masks[[4]]
#> [1] 3
#> 
#> 
#> $dynamic_masking
#> [1] TRUE
#> 
#> $has_nsp_head
#> [1] FALSE
#> 
#> $n_params
#> [1] 1216
#> 
#> $n_params_with_nsp
#> [1] 1232
#> 
#> $n_masked_per_sequence
#> [1] 1
#> 
#> $estimate
#> [1] 2.272476
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "RoBERTa: NSP-free encoder pretraining with per-epoch dynamic masking"
#>