BLIP-2 Q-Former bridge (Geron Ch 16, morie.fn hmblp2)

Learned queries cross-attend frozen image patch features and are projected to the LLM width. Only the Q-Former is trainable, which is what trainable_params makes explicit.

Usage

morie_geron_blip2(
  image,
  text,
  n_query = 4,
  d_query = 8,
  d_llm = NULL,
  temperature = 1,
  seed = 0
)

Arguments

image: Patch features (n_patches, d_v).
text: Embedding or token matrix.
n_query, d_query: Q-Former geometry.
d_llm: LLM width.
temperature: Positive temperature.
seed: LCG seed.

Value

List with query_output, llm_input, attention, similarity, query_similarities, trainable_params.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_blip2(V, V)
#> $query_output
#>             [,1]      [,2]        [,3]        [,4]      [,5]      [,6]
#> [1,] 0.009941283 0.1869136 -0.07143448 -0.07969451 -0.161877 0.0624544
#> [2,] 0.009941283 0.1869136 -0.07143448 -0.07969451 -0.161877 0.0624544
#> [3,] 0.009941283 0.1869136 -0.07143448 -0.07969451 -0.161877 0.0624544
#> [4,] 0.009941283 0.1869136 -0.07143448 -0.07969451 -0.161877 0.0624544
#>            [,7]      [,8]
#> [1,] -0.1302496 0.1618281
#> [2,] -0.1302496 0.1618281
#> [3,] -0.1302496 0.1618281
#> [4,] -0.1302496 0.1618281
#> 
#> $llm_input
#>             [,1]        [,2]       [,3]       [,4]        [,5]         [,6]
#> [1,] 0.005204421 -0.01270032 -0.0094739 0.01270676 -0.02144889 0.0003156865
#> [2,] 0.005204421 -0.01270032 -0.0094739 0.01270676 -0.02144889 0.0003156865
#> [3,] 0.005204421 -0.01270032 -0.0094739 0.01270676 -0.02144889 0.0003156865
#> [4,] 0.005204421 -0.01270032 -0.0094739 0.01270676 -0.02144889 0.0003156865
#>              [,7]        [,8]
#> [1,] -0.004588101 0.005533675
#> [2,] -0.004588101 0.005533675
#> [3,] -0.004588101 0.005533675
#> [4,] -0.004588101 0.005533675
#> 
#> $attention
#>      [,1]
#> [1,]    1
#> [2,]    1
#> [3,]    1
#> [4,]    1
#> 
#> $similarity
#> [1] 0.5760189
#> 
#> $query_similarities
#> [1] 0.5760189 0.5760189 0.5760189 0.5760189
#> 
#> $trainable_params
#> [1] 352
#> 
#> $frozen_encoders
#> [1] "image" "text" 
#> 
#> $d_llm
#> [1] 8
#> 
#> $estimate
#> [1] 0.5760189
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "BLIP-2: learned queries cross-attending frozen image features, projected to the LLM width"
#>