A step of the mistr_native implementation. Called by mistr_mistral_block.
See the file header for the source the module follows.
Usage
mistr_swiglu(x, W1, W2, W3)
Arguments
- x
A matrix; passed to crossprod.
- W1
A matrix; passed to crossprod.
- W2
A matrix; passed to crossprod.
- W3
A matrix; passed to crossprod.
Value
A vector, from as.numeric.
Examples
set.seed(1)
x <- matrix(rnorm(4), 1, 4)
mistr_swiglu(x, W1 = matrix(rnorm(24), 4, 6), W2 = matrix(rnorm(24), 6, 4), W3 = matrix(rnorm(24), 4, 6))
#> [1] -0.17831424 -0.54536821 0.96380373 -0.04769728
