Post-LN and pre-LN transformer blocks (Kamath Ch 2)

Usage

morie_kamath_post_ln_transformer(x, attn_fn, ffn_fn, eps = 1e-05)

morie_kamath_pre_ln_transformer(x, attn_fn, ffn_fn, eps = 1e-05)

morie_kamath_layer_norm_rows(x, eps = 1e-05)

Arguments

x: Input rows.
attn_fn, ffn_fn: Shape-preserving sublayers.
eps: LayerNorm epsilon.

Value

List with output, after_attention, placement.

Details

One LayerNorm serves both Python modules exactly as kmprln imports kmpoln's layer_norm: kmpoln -> morie_kamath_post_ln_transformer (LN after the add), kmprln -> morie_kamath_pre_ln_transformer (LN before the sublayer). The variance is the POPULATION variance, matching numpy's var.

Examples

att <- function(v) v * 0.5
ffn <- function(v) v * 0.25
morie_kamath_post_ln_transformer(matrix(c(1, 2, 3), 1), att,
    ffn)
#> $output
#>           [,1] [,2]     [,3]
#> [1,] -1.224741    0 1.224741
#> 
#> $after_attention
#>           [,1] [,2]     [,3]
#> [1,] -1.224741    0 1.224741
#> 
#> $estimate
#> [1] -1.224741
#> 
#> $placement
#> [1] "post-LN"
#> 
#> $eps
#> [1] 1e-05
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "Post-LayerNorm transformer block"
#>