Decoder-only transformer (GPT family), resolved to exact param counts (hmdctr)

Usage

morie_geron_decoder_only(
  X,
  n_layers = 12,
  n_heads = 12,
  d_model = 768,
  vocab_size = 50257,
  max_len = 1024,
  d_ff = NULL,
  tie_embeddings = TRUE
)

Arguments

X: Token ids, used for length only.
n_layers, n_heads: int.
d_model, vocab_size, max_len: int.
d_ff: FFN width (default 4*d_model).
tie_embeddings: Reuse token embedding as output head.

Value

List with total_params, block_params, per_block, embedding_params, d_head, mask, seq_len.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_decoder_only(V)
#> $total_params
#> [1] 124439808
#> 
#> $block_params
#> [1] 7087872
#> 
#> $per_block
#> $per_block$self_attention
#> [1] 2362368
#> 
#> $per_block$ffn
#> [1] 4722432
#> 
#> $per_block$layer_norms
#> [1] 3072
#> 
#> $per_block$total
#> [1] 7087872
#> 
#> 
#> $embedding_params
#> [1] 39383808
#> 
#> $output_head_params
#> [1] 0
#> 
#> $d_head
#> [1] 64
#> 
#> $mask
#>       [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]
#> [1,] FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [2,] FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [3,] FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [4,] FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE
#> [5,] FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE
#> [6,] FALSE FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE
#> [7,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE
#> [8,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> 
#> $seq_len
#> [1] 8
#> 
#> $n_layers
#> [1] 12
#> 
#> $n_heads
#> [1] 12
#> 
#> $d_model
#> [1] 768
#> 
#> $d_ff
#> [1] 3072
#> 
#> $vocab_size
#> [1] 50257
#> 
#> $max_len
#> [1] 1024
#> 
#> $tie_embeddings
#> [1] TRUE
#> 
#> $flops_per_token
#> [1] 248879616
#> 
#> $estimate
#> [1] 124439808
#> 
#> $n
#> [1] 12
#> 
#> $method
#> [1] "decoder-only transformer resolved to exact parameter counts and a causal mask"
#>