Swish / SiLU activation (Geron Ch 11, morie.fn hmswi)

z * sigmoid(beta z), with the sigmoid delegated to morie_geron_sigmoid exactly as the Python module delegates to hmsigm.

Usage

morie_geron_swish(z, beta = 1)

Arguments

z

Numeric vector.

beta

Finite gate scale.

Value

List with a, grad, gate, beta, estimate, n.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_swish(V)
#> $a
#> [1] 0.7310586 1.7615942 2.8577224 3.9280552 4.9665357 5.9851643 6.9936226
#> [8] 7.9973172
#> 
#> $grad
#> [1] 0.9276705 1.0907842 1.0881041 1.0526646 1.0265474 1.0123264 1.0054605
#> [8] 1.0023466
#> 
#> $gate
#> [1] 0.7310586 0.8807971 0.9525741 0.9820138 0.9933071 0.9975274 0.9990889
#> [8] 0.9996646
#> 
#> $beta
#> [1] 1
#> 
#> $estimate
#> [1] 4.402634
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Swish z*sigmoid(beta z) with sigmoid delegated to hmsigm"
#>