BERT masked-language-model loss (Geron Ch 16, morie.fn grmlm)

Cross-entropy on the masked positions only, the selected rows handed to morie_geron_gpt_autoregressive_loss().

Usage

morie_geron_bert_mlm_loss(logits, targets, mask)

Arguments

logits

(T, V) matrix.

targets

0-based token ids.

mask

Logical (or 0/1) selector of masked positions.

Value

List with loss, mean_loss, perplexity, per_token_loss, masked_positions (0-based), n_masked, mask_rate.

Examples

lg <- matrix(c(2, 0.1, 0.1, 0.1, 2, 0.1), 2, 3, byrow = TRUE)
morie_geron_bert_mlm_loss(lg, targets = c(1L, 2L), mask = c(1, 1))
#> $loss
#> [1] 4.323401
#> 
#> $mean_loss
#> [1] 2.1617
#> 
#> $perplexity
#> [1] 8.685894
#> 
#> $per_token_loss
#> [1] 2.1617 2.1617
#> 
#> $masked_positions
#> [1] 0 1
#> 
#> $n_masked
#> [1] 2
#> 
#> $mask_rate
#> [1] 1
#> 
#> $estimate
#> [1] 4.323401
#> 
#> $n
#> [1] 2
#>