(x W1 * SiLU(x W3)) W2 (Shazeer 2020): a front-end to
swiglu_activation with W = W3, V = W1, followed by the output
projection W2 when given.
Usage
swiglu(x, W1 = NULL, W2 = NULL, W3 = NULL)
Arguments
- x
Input matrix (rows are tokens).
- W1
Up-projection weights (NULL with W3 for elementwise gating).
- W2
Output projection (NULL returns the hidden layer).
- W3
Gate-projection weights.
Value
List with value (output), output and hidden.
References
Shazeer, N. (2020). GLU variants improve Transformer.
arXiv:2002.05202.
Examples
swiglu(rbind(c(1, -2)))$value
#> [,1] [,2]
#> [1,] 0.7310586 0.4768117
