DeiT: ViT architecture + distillation token, resolved concretely (hmdeit)

Block costs delegated to morie_geron_block_params; the distillation objective (when logits are supplied) to morie_geron_deit_distillation_loss.

Usage

morie_geron_deit(
  image,
  patch_size = 16,
  n_layers = 12,
  teacher = NULL,
  d_model = 384,
  n_heads = 6,
  n_classes = 1000,
  in_channels = 3,
  logits_cls = NULL,
  logits_dist = NULL,
  y = NULL,
  alpha = 0.5
)

Arguments

image

(H,W) or (C,H,W).

patch_size, n_layers, d_model, n_heads, n_classes, in_channels

int.

teacher

Logits or function(image).

logits_cls, logits_dist, y, alpha

Loss inputs.

Value

List with n_patches, n_tokens, total_params, distillation_overhead, loss, teacher_agreement.

Examples

img <- array(0, dim = c(3, 32, 32))
morie_geron_deit(img, patch_size = 16, n_layers = 1, d_model = 8,
    n_heads = 2, n_classes = 4)
#> $n_patches
#> [1] 4
#> 
#> $n_tokens
#> [1] 6
#> 
#> $patch_grid
#> [1] 2 2
#> 
#> $total_params
#> [1] 7176
#> 
#> $patch_embed_params
#> [1] 6152
#> 
#> $position_params
#> [1] 48
#> 
#> $block_params
#> [1] 872
#> 
#> $head_params
#> [1] 72
#> 
#> $distillation_overhead
#> [1] 44
#> 
#> $d_head
#> [1] 4
#> 
#> $loss
#> NULL
#> 
#> $loss_cls
#> NULL
#> 
#> $loss_dist
#> NULL
#> 
#> $teacher_agreement
#> NULL
#> 
#> $alpha
#> [1] 0.5
#> 
#> $estimate
#> [1] 7176
#> 
#> $n
#> [1] 6
#> 
#> $method
#> [1] "DeiT architecture resolved concretely; blocks via hmdctr, distillation loss via grdeit"
#>