CLIP symmetric contrastive loss (Geron Ch 16, morie.fn grclp)

Mean of the image->text and text->image InfoNCE terms with diagonal targets. accuracy_* compare 0-based argmax to the diagonal.

Usage

morie_geron_clip_contrastive_loss(
  image_embeddings,
  text_embeddings,
  tau = 0.07,
  normalize = TRUE
)

Arguments

image_embeddings, text_embeddings

Paired matrices (B, d).

tau

Positive temperature.

normalize

L2-normalise first.

Value

List with loss, loss_i2t, loss_t2i, similarity, accuracy_i2t, accuracy_t2i, chance_loss.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_clip_contrastive_loss(V, V)
#> $loss
#> [1] 2.079442
#> 
#> $loss_i2t
#> [1] 2.079442
#> 
#> $loss_t2i
#> [1] 2.079442
#> 
#> $similarity
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    1    1    1    1    1    1    1    1
#> [2,]    1    1    1    1    1    1    1    1
#> [3,]    1    1    1    1    1    1    1    1
#> [4,]    1    1    1    1    1    1    1    1
#> [5,]    1    1    1    1    1    1    1    1
#> [6,]    1    1    1    1    1    1    1    1
#> [7,]    1    1    1    1    1    1    1    1
#> [8,]    1    1    1    1    1    1    1    1
#> 
#> $accuracy_i2t
#> [1] 0.125
#> 
#> $accuracy_t2i
#> [1] 0.125
#> 
#> $chance_loss
#> [1] 2.079442
#> 
#> $tau
#> [1] 0.07
#> 
#> $estimate
#> [1] 2.079442
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "CLIP symmetric contrastive loss"
#>