Neural network weight initialisation (Philox)

R arm of morie.fn.vctrs, xavir and xvrig; the draws come from the Philox generator, row by row, so both arms give the same matrices. WeightInit: Xavier/Glorot, He/Kaiming and LeCun uniform or normal initialisation and orthogonal initialisation (modified Gram-Schmidt of a Gaussian matrix). Xavir: Xavier matrix fan_in by fan_out. Xvrig: Xavier matrix fan_out by fan_in.

Usage

WeightInit(fan_in, fan_out, method = "xavier_uniform", gain = 1, seed = 42)

Xavir(fan_in, fan_out, seed = 42, uniform = TRUE)

Xvrig(fan_in, fan_out, distribution = "normal", seed = NULL)

Arguments

fan_in, fan_out: Layer sizes.
method: Initialisation method.
gain: Scaling factor.
seed: Philox seed.
uniform: Uniform (TRUE) or normal.
distribution: "normal" or "uniform".

Value

A list with the weights and their moments.

References

Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. AISTATS, 249-256.

He, K., Zhang, X., Ren, S. and Sun, J. (2015). Delving deep into rectifiers. ICCV, 1026-1034.

Saxe, A. M., McClelland, J. L. and Ganguli, S. (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. ICLR.

Examples

WeightInit(3, 2, seed = 1)$value
#>           [,1]      [,2]
#> [1,] 0.8550150 0.8647109
#> [2,] 0.1878161 0.4628652
#> [3,] 0.3768293 0.8196382
crossprod(WeightInit(5, 3, method = "orthogonal")$value)
#>               [,1]          [,2]          [,3]
#> [1,]  1.000000e+00  7.520733e-17 -1.393964e-17
#> [2,]  7.520733e-17  1.000000e+00 -3.770363e-17
#> [3,] -1.393964e-17 -3.770363e-17  1.000000e+00