GELU activation (Geron Ch 11, morie.fn hmgelu)

Exact form z * Phi(z) with Phi the standard normal CDF, plus the tanh approximation. GELU is non-monotone: the derivative Phi(z) + z phi(z) is negative near z = -1, which the parity test checks by finite differences.

Usage

morie_geron_gelu(z, approximate = FALSE)

Arguments

z

Numeric vector.

approximate

Use the tanh form.

Value

List with a, activation, exact, approx, derivative, max_abs_gap, estimate, n.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_gelu(V)
#> $a
#> [1] 0.8413447 1.9544997 2.9959503 3.9998733 4.9999986 6.0000000 7.0000000
#> [8] 8.0000000
#> 
#> $activation
#> [1] 0.8413447 1.9544997 2.9959503 3.9998733 4.9999986 6.0000000 7.0000000
#> [8] 8.0000000
#> 
#> $exact
#> [1] 0.8413447 1.9544997 2.9959503 3.9998733 4.9999986 6.0000000 7.0000000
#> [8] 8.0000000
#> 
#> $approx
#> [1] 0.841192 1.954598 2.996363 3.999930 5.000000 6.000000 7.000000 8.000000
#> 
#> $derivative
#> [1] 1.083315 1.085232 1.011946 1.000504 1.000007 1.000000 1.000000 1.000000
#> 
#> $max_abs_gap
#> [1] 0.000412302
#> 
#> $estimate
#> [1] 4.473958
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "GELU(z) = z * Phi(z)"
#>