Sparse (masked) scaled dot-product attention

softmax(mask_S(Q K' / sqrt(d))) V, with the score set to -Inf wherever the connectivity matrix S is zero.

Usage

Sparseattn(Q, K, V, S = NULL)

Arguments

Q

Queries, nq x d.

K

Keys, nk x d.

V

Values, nk x dv.

S

Connectivity mask, nq x nk; NULL is dense attention.

Value

List with out, weight, score, nq, nk, d, dv, density.

References

Child, Gray, Radford and Sutskever (2019), arXiv:1904.10509, Sects. 3-4 (in the local corpus); Beltagy, Peters and Cohan (2020), arXiv:2004.05150; scaled dot-product attention is Vaswani et al. (2017), Equation (1).

Examples

Sparseattn(Q = 0.5, K = c(1, 2, 3, 4, 5, 6, 7, 8), V = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $out
#>          [,1]
#> [1,] 6.607765
#> 
#> $weight
#>            [,1]       [,2]       [,3]       [,4]      [,5]      [,6]      [,7]
#> [1,] 0.01210343 0.01995518 0.03290052 0.05424379 0.0894329 0.1474499 0.2431038
#>           [,8]
#> [1,] 0.4008104
#> 
#> $score
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]  0.5    1  1.5    2  2.5    3  3.5    4
#> 
#> $nq
#> [1] 1
#> 
#> $nk
#> [1] 8
#> 
#> $d
#> [1] 1
#> 
#> $dv
#> [1] 1
#> 
#> $density
#> [1] 1
#> 
#> $method
#> [1] "Sparse scaled dot-product attention (Child et al. 2019)"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"