Vision Transformer (ViT): transformer on image patches — morie_geron_vision

Patchify image, linear-embed patches, prepend [CLS], add sinusoidal position encodings, run morie_geron_transformer, linear head on the [CLS] row.

Usage

morie_geron_vision_transformer(
  image,
  patch_size,
  n_layers = 2,
  d_model = 8,
  n_heads = 2,
  n_classes = 2,
  seed = 0
)

Arguments

image: (H,W) or (H,W,C) array.
patch_size: Patch side length.
n_layers, d_model, n_heads, n_classes, seed: As in Python original.

Value

list with logits, cls, tokens, patches, n_patches, seq_len, patch_dim, grid, encoder_params, total_params, predicted, estimate, n, method.

Examples

img <- matrix(0:15, 4, 4, byrow = TRUE)
morie_geron_vision_transformer(img, patch_size = 2, n_layers = 1,
    d_model = 4, n_heads = 2, n_classes = 3)
#> $logits
#> [1]  0.20726938 -0.00897518 -0.05603381
#> 
#> $cls
#> [1] -1.1010744  0.9215063 -0.8905985  1.0701666
#> 
#> $tokens
#>            [,1]       [,2]       [,3]     [,4]
#> [1,] -1.1010744  0.9215063 -0.8905985 1.070167
#> [2,] -0.2572567  0.4762396 -1.4665102 1.247527
#> [3,] -0.3421350 -0.2950716 -1.0216993 1.658906
#> [4,] -1.1234178 -0.0713179 -0.4073986 1.602134
#> [5,] -1.3636944  0.2083964 -0.2677838 1.423082
#> 
#> $patches
#>      [,1] [,2] [,3] [,4]
#> [1,]    0    1    4    5
#> [2,]    2    3    6    7
#> [3,]    8    9   12   13
#> [4,]   10   11   14   15
#> 
#> $n_patches
#> [1] 4
#> 
#> $seq_len
#> [1] 5
#> 
#> $patch_dim
#> [1] 4
#> 
#> $grid
#> [1] 2 2
#> 
#> $encoder_params
#> [1] 228
#> 
#> $total_params
#> [1] 287
#> 
#> $predicted
#> [1] 0
#> 
#> $estimate
#> [1] 287
#> 
#> $n
#> [1] 4
#> 
#> $method
#> [1] "ViT: patch embedding + [CLS] + sinusoidal positions + transformer encoder (hmtfm)"
#>