DeiT hard-label distillation loss (Geron Ch 16, morie.fn grdeit)

L = (1 - alpha) CE(cls, y) + alpha CE(dist, argmax teacher). Labels are 0-based; teacher_labels comes back 0-based too.

Usage

morie_geron_deit_distillation_loss(
  logits_cls,
  logits_dist,
  y,
  teacher_preds,
  alpha = 0.5
)

Arguments

logits_cls, logits_dist

Matrices (B, C).

y

0-based labels.

teacher_preds

Matrix (B, C) or vector of labels.

alpha

Distillation weight in [0, 1].

Value

List with loss, loss_cls, loss_dist, teacher_labels, teacher_agreement, accuracy_cls, accuracy_dist.

Examples

morie_geron_deit_distillation_loss(matrix(c(1, 0, 0, 2), 2, 2,
    byrow = TRUE), matrix(c(0.5, 0.5, 1, 0), 2, 2, byrow = TRUE),
    c(0, 1), matrix(c(1, 0, 1, 0), 2, 2, byrow = TRUE), alpha = 0.3)
#> $loss
#> [1] 0.3050277
#> 
#> $loss_cls
#> [1] 0.2200948
#> 
#> $loss_dist
#> [1] 0.5032044
#> 
#> $teacher_labels
#> [1] 0 0
#> 
#> $teacher_agreement
#> [1] 0.5
#> 
#> $accuracy_cls
#> [1] 1
#> 
#> $accuracy_dist
#> [1] 1
#> 
#> $alpha
#> [1] 0.3
#> 
#> $estimate
#> [1] 0.3050277
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "DeiT hard-label distillation loss"
#>