Flamingo: perceiver resampler + tanh-gated cross-attention (hmflmg)

Both attention stages delegated to morie_geron_cross_attention.

Usage

morie_geron_flamingo(
  images,
  text,
  latents = NULL,
  W_Q = NULL,
  W_K = NULL,
  W_V = NULL,
  gate = 0,
  image_index = NULL
)

Arguments

images: Visual features (n_features, d).
text: Frozen LM hidden states (T, d).
latents: Optional perceiver latents (default one row of ones).
W_Q, W_K, W_V: Optional projections (default identity).
gate: Pre-tanh gate value (default 0).
image_index: Optional per-position image index (0 or -1).

Value

List with output, visual_tokens, cross_attention, gate_value, is_identity_at_init.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_flamingo(V, V)
#> $output
#>      [,1]
#> [1,]    1
#> [2,]    2
#> [3,]    3
#> [4,]    4
#> [5,]    5
#> [6,]    6
#> [7,]    7
#> [8,]    8
#> 
#> $visual_tokens
#>      [,1]
#> [1,]  4.5
#> 
#> $cross_attention
#>      [,1]
#> [1,]  4.5
#> [2,]  4.5
#> [3,]  4.5
#> [4,]  4.5
#> [5,]  4.5
#> [6,]  4.5
#> [7,]  4.5
#> [8,]  4.5
#> 
#> $attention_weights
#>      [,1]
#> [1,]    1
#> [2,]    1
#> [3,]    1
#> [4,]    1
#> [5,]    1
#> [6,]    1
#> [7,]    1
#> [8,]    1
#> 
#> $resampler_weights
#>       [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]
#> [1,] 0.125 0.125 0.125 0.125 0.125 0.125 0.125 0.125
#> 
#> $gate_value
#> [1] 0
#> 
#> $gate
#> [1] 0
#> 
#> $is_identity_at_init
#> [1] TRUE
#> 
#> $n_visual_tokens
#> [1] 1
#> 
#> $n_image_features
#> [1] 8
#> 
#> $delta_norm
#> [1] 0
#> 
#> $estimate
#> [1] 0
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "perceiver resampler plus tanh-gated cross-attention; attention delegated to grca"
#>