DINO teacher centring and sharpening

Formula: P_t = softmax((g_t - C)/tau_t); C <- m C + (1-m) (1/B) sum_i g_t(x_i)

Usage

Dinocenter(g_t, center = NULL, m = 0.9, tau_t = 0.04)

Arguments

g_t

See Usage.

center

See Usage.

m

See Usage.

tau_t

See Usage.

Value

List with p_t, center, center_old, batch_mean, B, K.

References

Caron, Touvron, Misra, Jegou, Mairal, Bojanowski and Joulin (2021), Emerging Properties in Self-Supervised Vision Transformers, ICCV/arXiv:2104.14294. Verified against the paper: equation (1) for the temperature softmax, equation (4) for the centre update, and Algorithm 1's pseudocode for the order of centre-then-sharpen.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Dinocenter(V)
#> $p_t
#>      [,1]
#> [1,]    1
#> [2,]    1
#> [3,]    1
#> [4,]    1
#> [5,]    1
#> [6,]    1
#> [7,]    1
#> [8,]    1
#> 
#> $center
#> [1] 0.45
#> 
#> $center_old
#> [1] 0
#> 
#> $batch_mean
#> [1] 4.5
#> 
#> $B
#> [1] 8
#> 
#> $K
#> [1] 1
#> 
#> $method
#> [1] "DINO teacher centring and sharpening"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"