Contrastive loss that scores each pair on its own

CLIP softmax normalises over the whole batch, so every pair loss depends on every other pair and the batch must be huge and globally synchronised. A sigmoid per pair removes that coupling: the loss decomposes and needs no all-gather. The learnable bias exists because a batch is overwhelmingly negatives.

Usage

Siglip(image_emb, text_emb, t_prime = 1, bias = 0)

Arguments

image_emb: Image embeddings; L2-normalised here.
text_emb: Text embeddings; L2-normalised here.
t_prime: Logit scale.
bias: Logit bias.

Value

List with estimate, logits, acc, n.

Details

Formula: L = -(1/|B|) sum_ij log sigmoid(z_ij (t x_i . y_j + b)), z_ij = 1 on the diagonal, -1 elsewhere.

References

Zhai, Mustafa, Kolesnikov & Beyer (2023). ICCV 2023, equation (3).

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Siglip(V, V)
#> $estimate
#> [1] 9.506094
#> 
#> $logits
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    1    1    1    1    1    1    1    1
#> [2,]    1    1    1    1    1    1    1    1
#> [3,]    1    1    1    1    1    1    1    1
#> [4,]    1    1    1    1    1    1    1    1
#> [5,]    1    1    1    1    1    1    1    1
#> [6,]    1    1    1    1    1    1    1    1
#> [7,]    1    1    1    1    1    1    1    1
#> [8,]    1    1    1    1    1    1    1    1
#> 
#> $acc
#> [1] 0.125
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "SigLIP pairwise sigmoid loss"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"