PET loss (Kamath Ch 3)

Verbalizer cross entropy + alpha * the km022 MLM loss.

Usage

morie_kamath_pet_loss(
  verbalizer_logits,
  y_true,
  mlm_logits,
  mlm_targets,
  alpha,
  ignore_index = -100L
)

Arguments

verbalizer_logits

Class logits.

y_true

0-BASED true class.

mlm_logits

(T, V) logits.

mlm_targets

Ids, -100 ignored.

alpha

Non-negative weight.

ignore_index

Ignored id.

Value

List with estimate, loss_ce, loss_mlm, n_masked.

Examples

morie_kamath_pet_loss(c(1, 0), 0, matrix(c(1, 0, 0, 2), 2), c(0,
    -100), 0.5)
#> $estimate
#> [1] 0.4698925
#> 
#> $loss
#> [1] 0.4698925
#> 
#> $loss_ce
#> [1] 0.3132617
#> 
#> $loss_mlm
#> [1] 0.3132617
#> 
#> $alpha
#> [1] 0.5
#> 
#> $n_masked
#> [1] 1
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "PET loss = verbalizer CE + alpha * MLM (via km022)"
#>