Exact transformer block parameter count, itemised (hmdctr) — morie_geron_block_params • rmorie

Exact transformer block parameter count, itemised (hmdctr)

Usage

morie_geron_block_params(d_model, d_ff = NULL, cross_attention = FALSE)

Arguments

d_model: Model width.
d_ff: FFN width (default 4*d_model).
cross_attention: Add a cross-attention sub-layer + norm.

Value

List with self_attention, ffn, layer_norms, (cross_attention), total.

Examples

morie_geron_block_params(d_model = 5L)
#> $self_attention
#> [1] 120
#> 
#> $ffn
#> [1] 225
#> 
#> $layer_norms
#> [1] 20
#> 
#> $total
#> [1] 365
#>