Flamingo tanh-gated cross-attention (Geron Ch 17, morie.fn grflam)

h_new = h + tanh(alpha) * crossattn(h, visual), the cross-attention coming from morie_geron_cross_attention().

Usage

morie_geron_flamingo_cross_modal_attn(
  h,
  visual_features,
  alpha,
  weights,
  mask = NULL
)

Arguments

h

(T, d_model) language hidden states.

visual_features

(S, d_v) visual tokens.

alpha

Finite gate parameter; 0 leaves the LM untouched.

weights

List or named list with WQ, WK, WV.

mask

Optional attention mask.

Value

List with h_new, gate, attention_output, attention_weights, delta_norm, is_identity.

Examples

set.seed(1)
h <- matrix(rnorm(12), 3, 4)
vf <- matrix(rnorm(8), 2, 4)
W <- diag(4)
r <- morie_geron_flamingo_cross_modal_attn(h, vf, alpha = 0.5,
                                           weights = list(WQ = W, WK = W,
                                                          WV = W))
str(r, max.level = 1)
#> List of 9
#>  $ h_new            : num [1:3, 1:4] -1.251 -0.443 -1.711 1.867 0.6 ...
#>  $ gate             : num 0.462
#>  $ alpha            : num 0.5
#>  $ attention_output : num [1:3, 1:4] -1.352 -1.356 -1.894 0.588 0.586 ...
#>  $ attention_weights: num [1:3, 1:2] 0.541 0.539 0.201 0.459 0.461 ...
#>  $ delta_norm       : num 1.49
#>  $ is_identity      : logi FALSE
#>  $ estimate         : num [1:3, 1:4] -1.251 -0.443 -1.711 1.867 0.6 ...
#>  $ n                : int 3