BLIP multi-task loss with captioning head (Geron Ch 16, morie.fn grblip)

ITC (symmetric InfoNCE) + ITM (pairwise BCE over all B^2 pairs, diagonal = match) + LM (token cross-entropy, negative targets are padding). Caption targets are 0-based token ids.

Usage

morie_geron_blip_itm_itc(
  image_emb,
  text_emb,
  caption_logits,
  caption_targets,
  tau = 0.07,
  lam_itc = 1,
  lam_itm = 1,
  lam_lm = 1,
  normalize = TRUE
)

Arguments

image_emb, text_emb

Matrices (B, d).

caption_logits

3-D array (B, L, V).

caption_targets

Matrix (B, L) of 0-based ids; negatives are padding.

tau

Temperature.

lam_itc, lam_itm, lam_lm

Task weights.

normalize

Cosine similarity.

Value

List with loss, itc, itm, lm, similarity, itm_accuracy, lm_perplexity.

Examples

cl <- array(0, dim = c(2, 2, 2))
morie_geron_blip_itm_itc(matrix(c(1, 0, 0, 1), 2, 2, byrow = TRUE),
    matrix(c(1, 0.2, 0.1, 1), 2, 2, byrow = TRUE), cl, matrix(c(0,
        1, 1, -1), 2, 2, byrow = TRUE), tau = 0.5)
#> $loss
#> [1] 1.356111
#> 
#> $itc
#> [1] 0.1715338
#> 
#> $itm
#> [1] 0.4914298
#> 
#> $lm
#> [1] 0.6931472
#> 
#> $similarity
#>           [,1]       [,2]
#> [1,] 0.9805807 0.09950372
#> [2,] 0.1961161 0.99503719
#> 
#> $itm_accuracy
#> [1] 0.5
#> 
#> $lm_perplexity
#> [1] 2
#> 
#> $tau
#> [1] 0.5
#> 
#> $weights
#> [1] 1 1 1
#> 
#> $estimate
#> [1] 1.356111
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "BLIP multi-task objective (ITC + ITM + LM)"
#>