Formula: h <- h + tanh(g) * CrossAttn(h, vision).
Usage
Flmgcr(x, vision, gate, weights = NULL, mask = NULL)
Arguments
- x
Language hidden states, T x d_model.
- vision
Visual features, Tv x d_model.
- gate
Gate parameter g; tanh(g) scales the branch, so g = 0 is
exactly the identity on x.
- weights
Optional list (WQ, WK, WV); identity by default.
- mask
Optional attention mask, passed through.
Value
The list returned by
morie_geron_flamingo_cross_modal_attn, plus estimate
(mean of the updated hidden states), n and method.
Details
The same method as morie_geron_flamingo_cross_modal_attn; this
is a thin alias. When weights is omitted the three projections
default to the identity, making the gated branch plain scaled
dot-product cross-attention on the raw features.
References
Alayrac et al (2022), NeurIPS 35:23716-23736 (Flamingo).
Examples
set.seed(5)
x <- matrix(rnorm(12), 3, 4); v <- matrix(rnorm(8), 2, 4)
identical(Flmgcr(x, v, gate = 0)$h, x) # tanh(0) = 0: the identity
#> [1] TRUE
Flmgcr(x, v, gate = 1)$estimate
#> [1] -0.5331944
