CLIP pretraining with zero-shot prompt matching (Geron Ch 16, morie.fn hmclip)

The symmetric InfoNCE is DELEGATED to morie_geron_clip_contrastive_loss (the grclp core). The batch's off-diagonal pairs are the negatives, so a bigger batch is a harder task and the chance loss log(B) rises with it.

Usage

morie_geron_clip(
  images,
  texts,
  tau = 0.07,
  normalize = TRUE,
  class_prompts = NULL
)

Arguments

images, texts

Paired embedding matrices of the same shape.

tau

Positive temperature.

normalize

Cosine-normalise.

class_prompts

Optional (C, d) prompt embeddings.

Value

List with loss, loss_i2t, loss_t2i, similarity, matched_similarity, accuracy_i2t, accuracy_t2i, chance_loss, zero_shot, tau, estimate, n.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_clip(V, V)
#> $loss
#> [1] 2.079442
#> 
#> $loss_i2t
#> [1] 2.079442
#> 
#> $loss_t2i
#> [1] 2.079442
#> 
#> $similarity
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    1    1    1    1    1    1    1    1
#> [2,]    1    1    1    1    1    1    1    1
#> [3,]    1    1    1    1    1    1    1    1
#> [4,]    1    1    1    1    1    1    1    1
#> [5,]    1    1    1    1    1    1    1    1
#> [6,]    1    1    1    1    1    1    1    1
#> [7,]    1    1    1    1    1    1    1    1
#> [8,]    1    1    1    1    1    1    1    1
#> 
#> $matched_similarity
#> [1] 1 1 1 1 1 1 1 1
#> 
#> $accuracy_i2t
#> [1] 0.125
#> 
#> $accuracy_t2i
#> [1] 0.125
#> 
#> $chance_loss
#> [1] 2.079442
#> 
#> $zero_shot
#> NULL
#> 
#> $tau
#> [1] 0.07
#> 
#> $estimate
#> [1] 2.079442
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "CLIP symmetric InfoNCE (delegated to grclp) plus zero-shot prompt matching"
#>