Self-attention: Q=K=V come from same input

Projects X into Q, K, V with W_Q/W_K/W_V and delegates the kernel to morie_geron_scaled_dot_product.

Usage

morie_geron_self_attention_modules(X, W_Q, W_K, W_V, mask = NULL)

Arguments

X

Sequence (T, d_model).

W_Q, W_K

Projections (d_model, d_k); same width required.

W_V

Projection (d_model, d_v).

mask

Optional mask passed through to the kernel.

Value

list with Y, attention, Q, K, V, estimate, n, method.

Examples

X <- matrix(c(1, 0, 0, 1), 2, 2, byrow = TRUE)
I2 <- diag(2)
morie_geron_self_attention_modules(X, I2, I2, I2)
#> $Y
#>           [,1]      [,2]
#> [1,] 0.6697615 0.3302385
#> [2,] 0.3302385 0.6697615
#> 
#> $attention
#>           [,1]      [,2]
#> [1,] 0.6697615 0.3302385
#> [2,] 0.3302385 0.6697615
#> 
#> $scores
#>           [,1]      [,2]
#> [1,] 0.7071068 0.0000000
#> [2,] 0.0000000 0.7071068
#> 
#> $Q
#>      [,1] [,2]
#> [1,]    1    0
#> [2,]    0    1
#> 
#> $K
#>      [,1] [,2]
#> [1,]    1    0
#> [2,]    0    1
#> 
#> $V
#>      [,1] [,2]
#> [1,]    1    0
#> [2,]    0    1
#> 
#> $estimate
#> [1] 0.6697615
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "Self-attention: shared-source projections into hmsdp"
#>