Formula: ViT-L/14 or ResNet-50x16
Usage
Clipxi(image, backbone = "vit-l/14", seed = 42)
Arguments
- image
An H x W matrix of pixel values; H and W must be
multiples of the backbone patch size.
- backbone
One of vit-l/14, vit-b/16, vit-b/32.
- seed
Seed of the deterministic stream.
Value
List with estimate, embedding,
n_patches, grid, patch, width,
embed_dim, norm, method.
Details
The image is cut into non-overlapping patch x patch squares, each
flattened and linearly projected to the transformer width, a class
token is prepended and a learned position embedding added; the class
token is then projected to the joint embedding space and
L2-normalised, which is what makes the cosine of two encodings a
similarity. The projection weights come from the deterministic
stream rather than trained checkpoints, so the geometry is exact and
reproducible but the semantics are not learned.
References
Radford et al. (2021), ICML 139:8748-8763;
Dosovitskiy et al. (2021), ICLR 2021.
Examples
# \donttest{
img <- matrix(runif(32 * 32), 32, 32)
e <- Clipxi(img, backbone = "vit-b/32")
c(e$n_patches, length(e$embedding))
#> [1] 1 32
# }
