Cross-attention with per-query entropy (Geron Ch 15, morie.fn hmcatt) — morie_geron_cross_attention

The attention itself is DELEGATED to morie_geron_cross_attention (the grca core); this adds the per-query entropy in bits, its log2(T_enc) ceiling and the 0-based argmax. Queries come from the decoder and keys/values from the encoder, so no causal mask is needed.

Usage

morie_geron_cross_attention_report(dec_h, enc_h, W_Q, W_K, W_V, mask = NULL)

Arguments

dec_h: Decoder states (T_dec, d).
enc_h: Encoder states.
W_Q, W_K, W_V: Projection matrices.
mask: Optional mask.

Value

List with context, output, attention_weights, logits, scale, d_k, entropy, max_entropy, argmax (0-based), estimate, n.

Examples

a <- 0.5
morie_geron_resnet(c(1, 2, 3), function(a) a * 0.5)
#> $y
#> [1] 1.5 3.0 4.5
#> 
#> $output
#> [1] 1.5 3.0 4.5
#> 
#> $residual
#> [1] 0.5 1.0 1.5
#> 
#> $skip
#> [1] 1 2 3
#> 
#> $residual_fraction
#> [1] 0.3333333
#> 
#> $estimate
#> [1] 1.5 3.0 4.5
#> 
#> $n
#> [1] 3
#> 
#> $method
#> [1] "Residual block y = F(x) + x (optionally projected skip)"
#> 
morie_geron_cross_attention_report(dec_h = a, enc_h = a, W_Q = a, W_K = a, W_V = a)
#> $context
#>      [,1]
#> [1,] 0.25
#> 
#> $output
#>      [,1]
#> [1,] 0.25
#> 
#> $attention_weights
#>      [,1]
#> [1,]    1
#> 
#> $logits
#>        [,1]
#> [1,] 0.0625
#> 
#> $scale
#> [1] 1
#> 
#> $d_k
#> [1] 1
#> 
#> $entropy
#> [1] 0
#> 
#> $max_entropy
#> [1] 0
#> 
#> $argmax
#> [1] 0
#> 
#> $estimate
#> [1] 0.25
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "scaled dot-product cross-attention (delegated to grca) with per-query entropy"
#>