BLIP pretraining objectives on paired embeddings (Geron Ch 16, hmblip)

ITC is symmetric InfoNCE over the cosine matrix / tau; ITM is binary cross-entropy of the matched pairs against the batch rotated by one; LM is the mean negative caption log-probability when supplied.

Usage

morie_geron_blip(images, texts, temperature = 1, caption_logprobs = NULL)

Arguments

images, texts: Row-aligned embedding matrices (n, d), n >= 2.
temperature: Positive temperature.
caption_logprobs: Optional per-caption mean token log-probs.

Value

List with itc_loss, itm_loss, lm_loss, total_loss, similarity, retrieval_acc.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_blip(V, V)
#> $itc_loss
#> [1] 2.079442
#> 
#> $itc_i2t
#> [1] 2.079442
#> 
#> $itc_t2i
#> [1] 2.079442
#> 
#> $itm_loss
#> [1] 0.8132617
#> 
#> $lm_loss
#> NULL
#> 
#> $total_loss
#> [1] 2.892703
#> 
#> $similarity
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    1    1    1    1    1    1    1    1
#> [2,]    1    1    1    1    1    1    1    1
#> [3,]    1    1    1    1    1    1    1    1
#> [4,]    1    1    1    1    1    1    1    1
#> [5,]    1    1    1    1    1    1    1    1
#> [6,]    1    1    1    1    1    1    1    1
#> [7,]    1    1    1    1    1    1    1    1
#> [8,]    1    1    1    1    1    1    1    1
#> 
#> $retrieval_acc
#> [1] 0.125
#> 
#> $temperature
#> [1] 1
#> 
#> $estimate
#> [1] 2.892703
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "BLIP: contrastive + matching (+ captioning) objectives on paired embeddings"
#>