GPT-1 architecture (delegated to hmdctr) plus the causal LM objective (hmgpt1)

Usage

morie_geron_gpt1(
  X,
  n_layers = NULL,
  n_heads = NULL,
  logits = NULL,
  targets = NULL,
  ...
)

Arguments

X: Token ids.
n_layers, n_heads: Optional overrides.
logits: Optional (T, V) logits.
targets: Optional next-token targets (default shifted X).
...: Other decoder_only args.

Value

List with total_params, config, loss, perplexity, n_predicted.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_gpt1(V)
#> $total_params
#> [1] 116536320
#> 
#> $config
#> $config$n_layers
#> [1] 12
#> 
#> $config$n_heads
#> [1] 12
#> 
#> $config$d_model
#> [1] 768
#> 
#> $config$vocab_size
#> [1] 40478
#> 
#> $config$max_len
#> [1] 512
#> 
#> $config$d_ff
#> [1] 3072
#> 
#> 
#> $block_params
#> [1] 7087872
#> 
#> $embedding_params
#> [1] 31480320
#> 
#> $d_head
#> [1] 64
#> 
#> $mask
#>       [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]
#> [1,] FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [2,] FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [3,] FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [4,] FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE
#> [5,] FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE
#> [6,] FALSE FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE
#> [7,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE
#> [8,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> 
#> $loss
#> NULL
#> 
#> $perplexity
#> NULL
#> 
#> $token_losses
#> NULL
#> 
#> $n_predicted
#> [1] 7
#> 
#> $estimate
#> [1] 116536320
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "GPT-1 architecture delegated to hmdctr, plus the causal LM objective"
#>