ViT patch embedding (Geron Ch 17, morie.fn grvit)

Row-major patches (top-to-bottom, left-to-right), each flattened ROW-MAJOR over (row, col, channel), projected by E, with the CLS token prepended and the positional table added.

Usage

morie_geron_vit_patch_embedding(
  image,
  patch_size,
  E,
  E_pos = NULL,
  cls_token = NULL
)

Arguments

image

(H, W) or (H, W, C) array.

patch_size

Divisor of both H and W.

E

(p*p*C, d_model) projection.

E_pos

Optional (n_patches + 1, d_model) positional table.

cls_token

Optional length-d_model CLS vector; default zeros.

Value

List with embeddings, patches, n_patches, d_model.

Examples

set.seed(4)
img <- array(rnorm(8 * 8), c(8, 8))
E <- matrix(rnorm(16 * 4, 0, 0.3), 16, 4)
r <- morie_geron_vit_patch_embedding(img, patch_size = 4, E = E)
str(r, max.level = 1)
#> List of 6
#>  $ embeddings: num [1:5, 1:4] 0 -0.535 -0.655 -0.192 0.682 ...
#>  $ patches   : num [1:4, 1:16] 0.217 -0.754 1.636 -0.227 1.897 ...
#>  $ n_patches : int 4
#>  $ d_model   : int 4
#>  $ estimate  : num [1:5, 1:4] 0 -0.535 -0.655 -0.192 0.682 ...
#>  $ n         : int 5