Knowledge distillation loss (Geron Ch 19, morie.fn grkdl) — morie_geron_knowledge_distillation

L = (1-a) CE(student, y) + a T^2 KL(soft student || soft teacher).

Usage

morie_geron_knowledge_distillation_loss(
  student_logits,
  teacher_logits,
  y,
  alpha,
  T
)

Arguments

student_logits, teacher_logits: (m, K) logit matrices.
y: 0-based hard labels, length m.
alpha: Soft/hard mix in [0, 1].
T: Positive temperature.

Value

List with loss, ce_hard, kl_soft, kl_student_teacher, kl_teacher_student, soft_targets, teacher_entropy.

Details

PORTED AS THE PYTHON HAS IT: the soft term is the LITERAL reverse KL sum q (log q - log t) with the STUDENT as the left argument, which is the mirror image of Hinton's KL(teacher || student). Both directions are returned (kl_student_teacher, kl_teacher_student) but only the student-first one enters loss.

Examples

sl <- matrix(c(2, 0.5, 0.1, 0.2, 1.5, 0.3), 2, 3, byrow = TRUE)
tl <- matrix(c(1.8, 0.6, 0.2, 0.1, 1.7, 0.4), 2, 3, byrow = TRUE)
morie_geron_knowledge_distillation_loss(sl, tl, y = c(1L, 2L),
                                        alpha = 0.5, T = 2)
#> $loss
#> [1] 0.8721935
#> 
#> $ce_hard
#> [1] 1.735112
#> 
#> $kl_soft
#> [1] 0.009274652
#> 
#> $kl_student_teacher
#> [1] 0.002318663
#> 
#> $kl_teacher_student
#> [1] 0.002305834
#> 
#> $soft_targets
#>           [,1]      [,2]      [,3]
#> [1,] 0.5004653 0.2746612 0.2248735
#> [2,] 0.2279267 0.5072602 0.2648131
#> 
#> $teacher_entropy
#> [1] 1.035057
#> 
#> $alpha
#> [1] 0.5
#> 
#> $T
#> [1] 2
#> 
#> $estimate
#> [1] 0.8721935
#> 
#> $n
#> [1] 2
#>