Scaled dot-product attention

Att(Q,K,V) = softmax(Q K^T / sqrt(d_k)) V. Masked entries are set to -Inf before the row-wise softmax; a query row with no visible key is an error.

Usage

morie_geron_scaled_dot_product(Q, K, V, d_k = NULL, mask = NULL)

Arguments

Q

Queries (T_q, d_k); a 1-D vector is read as one row.

K

Keys (T_k, d_k).

V

Values (T_k, d_v).

d_k

Scaling dimension; defaults to ncol(K).

mask

Optional (T_q, T_k) or length-T_k logical/0-1 mask; TRUE/1 = visible.

Value

list with Y, attention, scores, d_k, estimate, n, method.

Examples

morie_geron_scaled_dot_product(Q = 0.5, K = 5L, V = 5L)
#> $Y
#>      [,1]
#> [1,]    5
#> 
#> $output
#>      [,1]
#> [1,]    5
#> 
#> $attention
#>      [,1]
#> [1,]    1
#> 
#> $scores
#>      [,1]
#> [1,]  2.5
#> 
#> $d_k
#> [1] 1
#> 
#> $estimate
#> [1] 1
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "Scaled dot-product attention with pre-softmax masking"
#>