GPT-2 released configurations resolved through hmdctr (hmgpt2)

Usage

morie_geron_gpt2(X, n_layers = NULL, n_heads = NULL, size = "small", ...)

Arguments

X: Token ids.
n_layers, n_heads: Optional overrides.
size: "small"/"medium"/"large"/"xl".
...: Other decoder_only args.

Value

List with total_params, non_embedding_params, config, params_vs_small, all_sizes.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_gpt2(V)
#> $total_params
#> [1] 124439808
#> 
#> $non_embedding_params
#> [1] 85056000
#> 
#> $embedding_params
#> [1] 39383808
#> 
#> $block_params
#> [1] 7087872
#> 
#> $config
#> $config$vocab_size
#> [1] 50257
#> 
#> $config$max_len
#> [1] 1024
#> 
#> $config$n_layers
#> [1] 12
#> 
#> $config$n_heads
#> [1] 12
#> 
#> $config$d_model
#> [1] 768
#> 
#> 
#> $size
#> [1] "small"
#> 
#> $d_head
#> [1] 64
#> 
#> $params_vs_small
#> [1] 1
#> 
#> $all_sizes
#> $all_sizes$small
#> [1] 124439808
#> 
#> $all_sizes$medium
#> [1] 354823168
#> 
#> $all_sizes$large
#> [1] 774030080
#> 
#> $all_sizes$xl
#> [1] 1557611200
#> 
#> 
#> $mask
#>       [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]
#> [1,] FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [2,] FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [3,] FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [4,] FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE
#> [5,] FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE
#> [6,] FALSE FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE
#> [7,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE
#> [8,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> 
#> $estimate
#> [1] 124439808
#> 
#> $n
#> [1] 12
#> 
#> $method
#> [1] "GPT-2 released configurations resolved through hmdctr, with scaling comparisons"
#>