Self-attention, single head (Geron Ch 16, morie.fn grsa)

SA(X) = softmax(XWq (XWk)^T / sqrt(d_k)) XWv through the shared attention kernel.

Usage

morie_geron_self_attention(X, WQ, WK, WV, mask = NULL)

Arguments

X

(T, d_model) tokens.

WQ, WK, WV

Projections with d_model rows.

mask

Optional attention mask.

Value

List with output, weights, Q, K, V.

Examples

morie_geron_self_attention(X = 5L, WQ = c(1, 2, 3, 4, 5, 6, 7, 8),
  WK = c(1, 2, 3, 4, 5, 6, 7, 8), WV = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $output
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    5   10   15   20   25   30   35   40
#> 
#> $weights
#>      [,1]
#> [1,]    1
#> 
#> $Q
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    5   10   15   20   25   30   35   40
#> 
#> $K
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    5   10   15   20   25   30   35   40
#> 
#> $V
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    5   10   15   20   25   30   35   40
#> 
#> $estimate
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    5   10   15   20   25   30   35   40
#> 
#> $n
#> [1] 1
#>