Temperature-scaled knowledge-distillation loss

p_i = exp(z_i/T)/sum_j exp(z_j/T) (eq. 1); the soft cross entropy CE(p, q) is multiplied by T^2 and blended with the hard-label cross entropy taken at T = 1.

Usage

Distilkl(teacher, student, temperature = 2, label = NULL, alpha = 0.5)

Arguments

teacher: Teacher logits, length k.
student: Student logits, length k.
temperature: T > 0.
label: One-based index of the correct class, or NULL.
alpha: Weight on the soft objective, in [0, 1].

Value

List with softce, kl, hardce, total, teacherprob, studentprob, temperature, k.

References

Hinton, Vinyals and Dean (2015), arXiv:1503.02531, Sect. 2 and Equation (1). Read from the ar5iv rendering; the same paper is in the local corpus.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Distilkl(V, V)
#> $softce
#> [1] 1.610384
#> 
#> $kl
#> [1] 0
#> 
#> $hardce
#> [1] NA
#> 
#> $total
#> [1] 6.441537
#> 
#> $teacherprob
#> [1] 0.01210343 0.01995518 0.03290052 0.05424379 0.08943290 0.14744992 0.24310382
#> [8] 0.40081044
#> 
#> $studentprob
#> [1] 0.01210343 0.01995518 0.03290052 0.05424379 0.08943290 0.14744992 0.24310382
#> [8] 0.40081044
#> 
#> $temperature
#> [1] 2
#> 
#> $k
#> [1] 8
#> 
#> $method
#> [1] "Temperature-scaled distillation loss (Hinton et al. 2015 Sect. 2)"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"