Position-wise feed-forward used by the transformer blocks — morie_geron_block_feed_forward • rmorie

Position-wise feed-forward used by the transformer blocks

Usage

morie_geron_block_feed_forward(X, weights)

Arguments

X: (T, d_model) matrix.
weights: Named list with W1, W2 and optional b1, b2.

Value

Matrix of the same shape as X.

Examples

set.seed(1)
X <- matrix(rnorm(12), 3, 4)
r <- morie_geron_block_feed_forward(X, list(W1 = matrix(rnorm(32, 0, 0.4), 4, 8),
                                            W2 = matrix(rnorm(32, 0, 0.4), 8, 4)))
str(r, max.level = 1)
#>  num [1:3, 1:4] -0.1204 -0.2236 -0.3522 -0.0678 -0.2362 ...