Rectifier linear unit activation and its gradient

Formula: g(z) = max(0, z); g'(z) = 1 if z > 0, else 0

Usage

Reluact(z, slope = 0)

Arguments

z

Pre-activation values.

slope

Slope applied below the threshold; 0 gives the plain ReLU, a small positive value gives the leaky ReLU of Sect. 10.3.3.

Value

List with activation, gradient, n.

References

Montesinos Lopez, Montesinos Lopez and Crossa (2022), Multivariate Statistical Machine Learning Methods for Genomic Prediction, Springer, doi:10.1007/978-3-030-89010-0. Chapter 10, Sect. 10.3.2 p. 388: g(z) = max(0, z), flat below the threshold and linear above it; Sect. 10.3.3 p. 389 gives the leaky variant with slope alpha below zero (the figure uses alpha = 0.1). Read from the chapter PDF, not recalled.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Reluact(V)
#> $activation
#> [1] 1 2 3 4 5 6 7 8
#> 
#> $gradient
#> [1] 1 1 1 1 1 1 1 1
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "ReLU activation, MVSML Sect. 10.3.2"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"