Scaled dot-product cross-attention (Geron Ch 15, morie.fn grca) — morie_geron_cross

softmax(X_dec WQ (X_enc WK)^T / sqrt(d_k)) X_enc WV. mask marks DISALLOWED pairs with TRUE, matching Python.

Usage

morie_geron_cross_attention(X_dec, X_enc, WQ, WK, WV, mask = NULL)

Arguments

X_dec, X_enc: Decoder and encoder token matrices.
WQ, WK, WV: Projection matrices.
mask: Optional logical matrix.

Value

List with output, attention_weights, logits, scale, d_k.

Examples

morie_geron_cross_attention(matrix(c(1, 2, 0, 1), 2, 2, byrow = TRUE),
    matrix(c(1, 0, 0, 2, 1, 1), 3, 2, byrow = TRUE), WQ = diag(2),
    WK = matrix(c(0.5, 0, 0, 1), 2, 2, byrow = TRUE), WV = matrix(c(1,
        2), ncol = 1))
#> $output
#>          [,1]
#> [1,] 3.581412
#> [2,] 3.295917
#> 
#> $attention_weights
#>            [,1]      [,2]      [,3]
#> [1,] 0.05884618 0.6991048 0.2420491
#> [2,] 0.14002925 0.5759753 0.2839954
#> 
#> $logits
#>           [,1]     [,2]      [,3]
#> [1,] 0.3535534 2.828427 1.7677670
#> [2,] 0.0000000 1.414214 0.7071068
#> 
#> $scale
#> [1] 0.7071068
#> 
#> $d_k
#> [1] 2
#> 
#> $estimate
#> [1] 3.438665
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "Scaled dot-product cross-attention"
#>