Mistral-7B architecture accounting (Geron Ch 15, morie.fn hmmis7)

Usage

morie_geron_mistral7b(
  prompt,
  n_tokens,
  n_layers = 32,
  d_model = 4096,
  n_heads = 32,
  n_kv_heads = 8,
  d_ff = 14336,
  vocab_size = 32000,
  window = 4096,
  dtype_bytes = 2
)

Arguments

prompt: Integer token ids (0-based).
n_tokens: Tokens to generate.
n_layers, d_model, n_heads, n_kv_heads, d_ff, vocab_size, window: Architecture.
dtype_bytes: Cache bytes/elem.

Value

List with total_parameters, parameters_per_layer, breakdown, d_head, kv_cache_bytes, kv_cache_saving, attention_mask, effective_context, estimate, n, method.

Examples

morie_geron_mistral7b(prompt = c(1, 2, 3, 4, 5, 6, 7, 8), n_tokens = 5L)
#> $total_parameters
#> [1] 7241732096
#> 
#> $parameters_per_layer
#> [1] 218112000
#> 
#> $breakdown
#> $breakdown$token_embedding
#> [1] 131072000
#> 
#> $breakdown$output_head
#> [1] 131072000
#> 
#> $breakdown$attention
#> [1] 1342177280
#> 
#> $breakdown$feedforward
#> [1] 5637144576
#> 
#> $breakdown$norms
#> [1] 266240
#> 
#> 
#> $d_head
#> [1] 128
#> 
#> $d_kv
#> [1] 1024
#> 
#> $kv_cache_bytes
#> [1] 1703936
#> 
#> $kv_cache_bytes_mha
#> [1] 6815744
#> 
#> $kv_cache_saving
#> [1] 4
#> 
#> $attention_mask
#>      [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]
#> [1,] TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [2,] TRUE  TRUE FALSE FALSE FALSE FALSE FALSE FALSE
#> [3,] TRUE  TRUE  TRUE FALSE FALSE FALSE FALSE FALSE
#> [4,] TRUE  TRUE  TRUE  TRUE FALSE FALSE FALSE FALSE
#> [5,] TRUE  TRUE  TRUE  TRUE  TRUE FALSE FALSE FALSE
#> [6,] TRUE  TRUE  TRUE  TRUE  TRUE  TRUE FALSE FALSE
#> [7,] TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE FALSE
#> [8,] TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
#> 
#> $window
#> [1] 4096
#> 
#> $effective_context
#> [1] 131072
#> 
#> $n_prompt_tokens
#> [1] 8
#> 
#> $n_generated
#> [1] 5
#> 
#> $estimate
#> [1] 7241732096
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Mistral-7B architecture accounting"
#>