Apply a dropout mask with inverted scaling

Formula: a_i = x_i * m_i / (1 - rate), m_i in {0, 1} supplied by the caller

Usage

Dropmask(x, mask, rate)

Arguments

x

Activations of the layer being regularized.

mask

Keep/drop indicator per unit: 1 keeps, 0 drops. Supplied by the caller so the result is reproducible.

rate

Dropout rate in [0, 1); the surviving activations are divided by 1 - rate.

Value

List with activation, kept, dropped, rate, n.

References

Montesinos Lopez, Montesinos Lopez and Crossa (2022), Multivariate Statistical Machine Learning Methods for Genomic Prediction, Springer, doi:10.1007/978-3-030-89010-0. Chapter 10, Sect. 10.6 p. 404 describes dropout as setting a random fraction of the weights of the input or hidden neurons to zero, so their contribution is removed on the forward pass and they receive no update on the backward pass; it prints no formula and attributes the method to Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov (2014), Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR 15:1929-1958, which is where the 1/(1 - rate) inverted scaling comes from. The mask is an argument rather than drawn internally so the function is deterministic. Chapter read from the PDF; the scaling is from the paper the book names.

Examples

set.seed(1)
Dropmask(x = rnorm(10), mask = rbinom(10, 1, 0.5), rate = 0.5)
#> $activation
#>  [1] -1.252908  0.000000 -1.671257  0.000000  0.000000  0.000000  0.000000
#>  [8]  0.000000  1.151563  0.000000
#> 
#> $kept
#> [1] 3
#> 
#> $dropped
#> [1] 7
#> 
#> $rate
#> [1] 0.5
#> 
#> $n
#> [1] 10
#> 
#> $method
#> [1] "Inverted dropout, MVSML Sect. 10.6"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"