CLIP image-text similarity

Formula: cos(I_emb, T_emb) / tau

Usage

Clipsi(I_emb, T_emb, tau = 0.01)

Arguments

I_emb

An n x d matrix of image embeddings.

T_emb

An n x d matrix of text embeddings, paired row by row.

tau

Temperature, strictly positive.

Value

List with estimate, logits, cosine, retrieved, accuracy, n, d, method.

Details

Both embeddings are L2-normalised first, so the inner product IS the cosine and the logit is bounded by +/- 1/tau. The diagonal of the logit matrix holds the matched pairs; retrieval takes the row argmax, and the reported accuracy is how often that argmax is the diagonal.

References

Radford et al. (2021), Learning Transferable Visual Models From Natural Language Supervision, ICML 139:8748-8763.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Clipsi(V, V)
#> $estimate
#> [1] 1
#> 
#> $logits
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]  100  100  100  100  100  100  100  100
#> [2,]  100  100  100  100  100  100  100  100
#> [3,]  100  100  100  100  100  100  100  100
#> [4,]  100  100  100  100  100  100  100  100
#> [5,]  100  100  100  100  100  100  100  100
#> [6,]  100  100  100  100  100  100  100  100
#> [7,]  100  100  100  100  100  100  100  100
#> [8,]  100  100  100  100  100  100  100  100
#> 
#> $cosine
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    1    1    1    1    1    1    1    1
#> [2,]    1    1    1    1    1    1    1    1
#> [3,]    1    1    1    1    1    1    1    1
#> [4,]    1    1    1    1    1    1    1    1
#> [5,]    1    1    1    1    1    1    1    1
#> [6,]    1    1    1    1    1    1    1    1
#> [7,]    1    1    1    1    1    1    1    1
#> [8,]    1    1    1    1    1    1    1    1
#> 
#> $retrieved
#> [1] 0 0 0 0 0 0 0 0
#> 
#> $accuracy
#> [1] 0.125
#> 
#> $n
#> [1] 8
#> 
#> $d
#> [1] 1
#> 
#> $method
#> [1] "CLIP image-text cosine similarity"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"