SwiGLU activation (Kamath Ch 2)

Swish(xW + b) gates (xV + c) elementwise.

Usage

morie_kamath_swiglu_activation(x, W, V, b = NULL, c = NULL)

morie_kamath_swish(z, beta = 1)

Arguments

x

Input.

W, V

Equal-shaped projections.

b, c

Biases.

z

Pre-activation.

beta

Swish sharpness.

Value

List with output, gate, linear.

Examples

morie_kamath_swiglu_activation(x = c(1, 2, 3, 4, 5, 6, 7, 8), W = c(1, 2, 3, 4, 5, 6, 7, 8),
  V = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $output
#> [1] 41616
#> 
#> $gate
#> [1] 204
#> 
#> $linear
#> [1] 204
#> 
#> $estimate
#> [1] 41616
#> 
#> $hidden_dim
#> [1] 1
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "SwiGLU: Swish(xW + b) * (xV + c)"
#>