Scaled dot-product attention (Vaswani et al. 2017)

softmax(Q K^T / sqrt(d_k)) V; mask is additive (0 keep, -Inf drop) or logical (TRUE keep). Rows of the attention matrix sum to 1 by construction, and the parity tests assert it.

Usage

morie_alammar_sdp_attention(Q, K, V, mask = NULL)

Arguments

Q, K, V

Matrices.

mask

Optional matrix.

Value

List with output, attention.

Examples

morie_alammar_sdp_attention(Q = 0.5, K = c(1, 2, 3, 4, 5, 6, 7, 8), V = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $output
#>          [,1]
#> [1,] 6.607765
#> 
#> $attention
#>            [,1]       [,2]       [,3]       [,4]      [,5]      [,6]      [,7]
#> [1,] 0.01210343 0.01995518 0.03290052 0.05424379 0.0894329 0.1474499 0.2431038
#>           [,8]
#> [1,] 0.4008104
#> 
#> $d_k
#> [1] 1
#> 
#> $n
#> [1] 1
#> 
#> $estimate
#> [1] 6.607765
#> 
#> $method
#> [1] "Scaled dot-product attention (Vaswani et al. 2017)"
#>