CLIP contrastive losses (Kamath Eq 9.5-9.7)

Usage

morie_kamath_ch9_clip_image_to_text(V, L, sigma, N = NULL)

morie_kamath_ch9_clip_text_to_image(L, V, sigma, N = NULL)

morie_kamath_ch9_clip_contrastive_total(L_i2t, L_t2i)

Arguments

V: Image embeddings.
L: Text embeddings.
sigma: Temperature.
N: Optional batch size.
L_i2t, L_t2i: The two directional losses.

Value

List with estimate, per_pair, logits.

Details

One row-wise InfoNCE serves both directional Python modules exactly as km134 calls km133 with the towers swapped: km133 -> morie_kamath_ch9_clip_image_to_text, km134 -> morie_kamath_ch9_clip_text_to_image, km135 -> morie_kamath_ch9_clip_contrastive_total (their sum).

Examples

morie_kamath_ch9_clip_image_to_text(V = c(1, 2, 3, 4, 5, 6, 7, 8),
  L = c(1, 2, 3, 4, 5, 6, 7, 8), sigma = 0.5)
#> $estimate
#> [1] 21.02085
#> 
#> $per_pair
#> [1] 1.414541e+01 2.401849e+01 3.000248e+01 3.200034e+01 3.000005e+01
#> [6] 2.400001e+01 1.400000e+01 1.125352e-07
#> 
#> $logits
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    2    4    6    8   10   12   14   16
#> [2,]    4    8   12   16   20   24   28   32
#> [3,]    6   12   18   24   30   36   42   48
#> [4,]    8   16   24   32   40   48   56   64
#> [5,]   10   20   30   40   50   60   70   80
#> [6,]   12   24   36   48   60   72   84   96
#> [7,]   14   28   42   56   70   84   98  112
#> [8,]   16   32   48   64   80   96  112  128
#> 
#> $temperature
#> [1] 0.5
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "CLIP image-to-text contrastive loss (Kamath Eq 9.5)"
#>