Image-text contrastive loss (Kamath Ch 9)

Symmetric InfoNCE over cosine(I, T) / tau; the batch must hold at least two pairs.

Usage

morie_kamath_image_text_contrastive(I_emb, T_emb, tau)

Arguments

I_emb, T_emb

Tower outputs.

tau

Temperature.

Value

List with estimate, loss_i2t, loss_t2i, similarity.

Examples

morie_kamath_image_text_contrastive(I_emb = c(1, 2, 3, 4, 5, 6, 7, 8),
  T_emb = c(1, 2, 3, 4, 5, 6, 7, 8), tau = 0.5)
#> $estimate
#> [1] 2.079442
#> 
#> $loss
#> [1] 2.079442
#> 
#> $loss_i2t
#> [1] 2.079442
#> 
#> $loss_t2i
#> [1] 2.079442
#> 
#> $similarity
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    1    1    1    1    1    1    1    1
#> [2,]    1    1    1    1    1    1    1    1
#> [3,]    1    1    1    1    1    1    1    1
#> [4,]    1    1    1    1    1    1    1    1
#> [5,]    1    1    1    1    1    1    1    1
#> [6,]    1    1    1    1    1    1    1    1
#> [7,]    1    1    1    1    1    1    1    1
#> [8,]    1    1    1    1    1    1    1    1
#> 
#> $tau
#> [1] 0.5
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Image-text contrastive (symmetric InfoNCE)"
#>