Scaled dot-product attention (Geron Ch 16, morie.fn grsdpa)

The shared kernel behind grsa, grmha, grflash, grswin, grpvt, grpio, grteb and grtdb here.

Usage

morie_geron_scaled_dot_product_attention(Q, K, V, mask = NULL)

Arguments

Q, K, V

Query, key and value matrices.

mask

Optional logical mask, TRUE = attend.

Value

List with output, weights, scores, d_k.

Examples

morie_geron_scaled_dot_product_attention(Q = 0.5, K = 5L, V = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $output
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    1    2    3    4    5    6    7    8
#> 
#> $weights
#>      [,1]
#> [1,]    1
#> 
#> $scores
#>      [,1]
#> [1,]  2.5
#> 
#> $d_k
#> [1] 1
#> 
#> $estimate
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    1    2    3    4    5    6    7    8
#> 
#> $n
#> [1] 1
#>