LLaVA visual instruction assembly (Kamath Ch 9) — morie_kamath_llava_visual

[W ViT(image); text] soft tokens; with lm_head and targets the causal cross entropy is added. targets are 0-BASED with -100 ignored.

Usage

morie_kamath_llava_visual_instruction(
  image,
  W,
  visual_encoder,
  text_tokens,
  lm_head = NULL,
  targets = NULL,
  ignore_index = -100L
)

Arguments

image: Image.
W: Projection (d, d_v).
visual_encoder: Function image -> (n_patches, d_v).
text_tokens: Text embedding rows.
lm_head: Optional head.
targets: Optional target ids.
ignore_index: Ignored id.

Value

List with visual_tokens, inputs, optionally loss.

Examples

morie_kamath_llava_visual_instruction("im", diag(2), function(i) matrix(c(1,
    2), 1), matrix(c(3, 4), 1))
#> $visual_tokens
#>      [,1] [,2]
#> [1,]    1    2
#> 
#> $inputs
#>      [,1] [,2]
#> [1,]    1    2
#> [2,]    3    4
#> 
#> $n_visual
#> [1] 1
#> 
#> $n_text
#> [1] 1
#> 
#> $d_model
#> [1] 2
#> 
#> $estimate
#> [1] 1
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "LLaVA visual instruction assembly"
#>