DINO self-distillation cross-entropy

Formula: P_s = softmax(s/tau_s), P_t = softmax((t - C)/tau_t), loss = -(1/B) sum_i sum_k P_t(i,k) log P_s(i,k)

Usage

Dinoloss(s_logits, t_logits, tau_s = 0.1, tau_t = 0.04, center = NULL)

Arguments

s_logits: See Usage.
t_logits: See Usage.
tau_s: See Usage.
tau_t: See Usage.
center: See Usage.

Value

List with loss, per_view, p_s, p_t, B, K.

References

Caron, Touvron, Misra, Jegou, Mairal, Bojanowski and Joulin (2021), Emerging Properties in Self-Supervised Vision Transformers, ICCV/arXiv:2104.14294. Verified against the paper: equation (1) for the temperature softmax, equation (4) for the centre update, and Algorithm 1's pseudocode for the order of centre-then-sharpen.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Dinoloss(V, V)
#> $loss
#> [1] 0
#> 
#> $per_view
#> [1] 0 0 0 0 0 0 0 0
#> 
#> $p_s
#>      [,1]
#> [1,]    1
#> [2,]    1
#> [3,]    1
#> [4,]    1
#> [5,]    1
#> [6,]    1
#> [7,]    1
#> [8,]    1
#> 
#> $p_t
#>      [,1]
#> [1,]    1
#> [2,]    1
#> [3,]    1
#> [4,]    1
#> [5,]    1
#> [6,]    1
#> [7,]    1
#> [8,]    1
#> 
#> $B
#> [1] 8
#> 
#> $K
#> [1] 1
#> 
#> $method
#> [1] "DINO self-distillation loss"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"