DINO self-distillation loss (Geron Ch 16, morie.fn grdino) — morie_geron_dino_self

L = -mean_i sum_k P_teacher(k) log P_student(k), teacher sharpened by tau_t and optionally centred.

Usage

morie_geron_dino_self_distillation(
  student_logits,
  teacher_logits,
  tau_s,
  tau_t,
  center = NULL
)

Arguments

student_logits, teacher_logits: Numeric matrices (m, K); a vector is treated as one row.
tau_s, tau_t: Positive temperatures.
center: Optional length-K centring vector subtracted from the teacher logits.

Value

List with loss, teacher_probs, student_probs, teacher_entropy, student_entropy, max_teacher_prob, is_sharpened, per_sample_loss.

Examples

sl <- matrix(c(2, 0.5, 0.1, 0.2, 1.5, 0.3), 2, 3, byrow = TRUE)
tl <- matrix(c(1.8, 0.6, 0.2, 0.1, 1.7, 0.4), 2, 3, byrow = TRUE)
morie_geron_dino_self_distillation(sl, tl, tau_s = 0.1, tau_t = 0.04)
#> $loss
#> [1] 4.358007e-06
#> 
#> $teacher_probs
#>              [,1]         [,2]         [,3]
#> [1,] 1.000000e+00 9.357623e-14 4.248354e-18
#> [2,] 4.248354e-18 1.000000e+00 7.681205e-15
#> 
#> $student_probs
#>              [,1]         [,2]         [,3]
#> [1,] 9.999997e-01 3.059022e-07 5.602795e-09
#> [2,] 2.260310e-06 9.999916e-01 6.144161e-06
#> 
#> $teacher_entropy
#> [1] 1.579259e-12
#> 
#> $student_entropy
#> [1] 5.826248e-05
#> 
#> $max_teacher_prob
#> [1] 1
#> 
#> $is_sharpened
#> [1] TRUE
#> 
#> $per_sample_loss
#> [1] 3.115065e-07 8.404507e-06
#> 
#> $estimate
#> [1] 4.358007e-06
#> 
#> $n
#> [1] 2
#>