Scaled cosine attention (Swin Transformer V2)

Liu et al. (2022), "Swin Transformer V2: Scaling Up Capacity and Resolution", CVPR 2022, arXiv:2111.09883, Section 3.2: Sim(q_i, k_j) = cos(q_i, k_j) / tau + B_ij, with tau a learnable scalar constrained above 0.01 and B the relative position bias.

Usage

Vitscn(q, k, v, tau = 0.1, B = NULL)

Arguments

q

Query matrix (n_q x d).

k

Key matrix (n_k x d).

v

Value matrix (n_k x d_v).

tau

Temperature, must exceed 0.01 (paper constraint).

B

Optional (n_q x n_k) bias added after the scaled cosine.

Value

List with output, weights, similarities, tau, estimate, n, method.

References

Liu, Z. et al. (2022), CVPR 2022, arXiv:2111.09883, Section 3.2.

Examples

Vitscn(q = 0.5, k = c(1, 2, 3, 4, 5, 6, 7, 8), v = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $output
#>      [,1]
#> [1,]  4.5
#> 
#> $weights
#>       [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]
#> [1,] 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125
#> 
#> $similarities
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]   10   10   10   10   10   10   10   10
#> 
#> $tau
#> [1] 0.1
#> 
#> $estimate
#> [1] 4.5
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "scaled cosine attention cos(q,k)/tau + B (Liu et al. 2022, Sec 3.2)"
#>