SimCSE contrastive sentence embedding

Formula: InfoNCE on dropout-augmented pairs

Usage

Contse(sentences, tau = 0.05, dropout = 0.1, seed = 42)

Arguments

sentences: An n x d matrix of sentence embeddings.
tau: Temperature, strictly positive.
dropout: Dropout rate in [0, 1) used to build the two views.
seed: Seed of the deterministic stream.

Value

List with estimate, loss, per_item, alignment, uniformity, n, d, method.

Details

l_i = -log exp(sim(h_i, h_i+)/tau) / sum_j exp(sim(h_i, h_j+)/tau). The positive is the SAME sentence encoded twice under different dropout masks, so the objective needs no labelled pairs at all. When every similarity is identical the loss is exactly log N, the value it must return on a degenerate batch.

References

Gao, Yao & Chen (2021), SimCSE, EMNLP 2021:6894-6910.

Examples

M <- matrix(c(1, 2, 3, 4, 5, 6), nrow = 2)
Contse(M)
#> $estimate
#> [1] 1.026205
#> 
#> $loss
#> [1] 1.026205
#> 
#> $per_item
#> [1] 1.4187904 0.6336187
#> 
#> $alignment
#> [1] 0.4654775
#> 
#> $uniformity
#> [1] -1.633568
#> 
#> $n
#> [1] 2
#> 
#> $d
#> [1] 3
#> 
#> $method
#> [1] "SimCSE contrastive sentence objective"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"