Original encoder-decoder transformer, resolved to exact params + 3 masks (hmencd)

Block costs delegated to morie_geron_block_params; only the decoder block carries the extra cross-attention sub-layer.

Usage

morie_geron_encoder_decoder_transformer(
  src,
  tgt,
  n_layers = 6,
  n_heads = 8,
  d_model = 512,
  vocab_size = 37000,
  max_len = 512,
  d_ff = 2048,
  share_embeddings = TRUE
)

Arguments

src, tgt

Token ids.

n_layers, n_heads, d_model, vocab_size, max_len, d_ff

int.

share_embeddings

Share one embedding matrix between the stacks.

Value

List with total_params, encoder_params, decoder_params, src_mask, tgt_mask, cross_mask.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_encoder_decoder_transformer(V, V)
#> $total_params
#> [1] 82327688
#> 
#> $encoder_params
#> [1] 18914304
#> 
#> $decoder_params
#> [1] 25224192
#> 
#> $encoder_block_params
#> [1] 3152384
#> 
#> $decoder_block_params
#> [1] 4204032
#> 
#> $extra_per_decoder_block
#> [1] 1051648
#> 
#> $embedding_params
#> [1] 19206144
#> 
#> $output_head_params
#> [1] 18981000
#> 
#> $src_mask
#>       [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]
#> [1,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [2,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [3,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [4,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [5,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [6,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [7,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [8,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> 
#> $tgt_mask
#>       [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]
#> [1,] FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [2,] FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [3,] FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE  TRUE
#> [4,] FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE  TRUE
#> [5,] FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE  TRUE
#> [6,] FALSE FALSE FALSE FALSE FALSE FALSE  TRUE  TRUE
#> [7,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE  TRUE
#> [8,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> 
#> $cross_mask
#>       [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]
#> [1,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [2,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [3,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [4,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [5,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [6,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [7,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> [8,] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
#> 
#> $src_len
#> [1] 8
#> 
#> $tgt_len
#> [1] 8
#> 
#> $d_head
#> [1] 64
#> 
#> $n_layers
#> [1] 6
#> 
#> $n_heads
#> [1] 8
#> 
#> $d_model
#> [1] 512
#> 
#> $d_ff
#> [1] 2048
#> 
#> $share_embeddings
#> [1] TRUE
#> 
#> $estimate
#> [1] 82327688
#> 
#> $n
#> [1] 12
#> 
#> $method
#> [1] "original encoder-decoder transformer resolved to exact parameter counts and its three masks"
#>