Transformer position-wise feed-forward (Geron Ch 16, morie.fn grffn)

FFN(x) = relu(x W1 + b1) W2 + b2, routed through morie_geron_linear_layer_forward() with the transposed weights.

Usage

morie_geron_transformer_feedforward(x, W1, b1, W2, b2)

Arguments

x

Token vector or (T, d_model) matrix.

W1

(d_model, d_ff), b1 length d_ff.

b1

Hidden bias.

W2

(d_ff, d_model), b2 length d_model.

b2

Output bias.

Value

List with output, hidden, d_model, d_ff, expansion_ratio, sparsity, n_parameters.

Examples

x <- matrix(c(0.5, -0.2, 0.1, 0.4), 2, 2)
W1 <- matrix(0.1, 2, 4); b1 <- rep(0, 4)
W2 <- matrix(0.1, 4, 2); b2 <- rep(0, 2)
morie_geron_transformer_feedforward(x, W1, b1, W2, b2)
#> $output
#>       [,1]  [,2]
#> [1,] 0.024 0.024
#> [2,] 0.008 0.008
#> 
#> $hidden
#>      [,1] [,2] [,3] [,4]
#> [1,] 0.06 0.06 0.06 0.06
#> [2,] 0.02 0.02 0.02 0.02
#> 
#> $d_model
#> [1] 2
#> 
#> $d_ff
#> [1] 4
#> 
#> $expansion_ratio
#> [1] 2
#> 
#> $sparsity
#> [1] 0
#> 
#> $n_parameters
#> [1] 22
#> 
#> $estimate
#>       [,1]  [,2]
#> [1,] 0.024 0.024
#> [2,] 0.008 0.008
#> 
#> $n
#> [1] 2
#>