Vanishing gradients: small gradients shrink through many layers

Reports per-layer ratio ||g_{l+1}||/||g_l|| and the geometric mean ratio (constant per-layer factor); flags "vanishing" if the geometric mean falls below tol.

Usage

morie_geron_vanishing_gradients(grads, tol = 0.5)

Arguments

grads

List of >= 2 per-layer gradients, input side first.

tol

Geometric-mean ratio threshold, in (0,1).

Value

list with norms, ratios, geometric_ratio, attenuation, vanishing, tol, estimate, n, method.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_vanishing_gradients(V)
#> $norms
#> [1] 1 2 3 4 5 6 7 8
#> 
#> $ratios
#> [1] 2.000000 1.500000 1.333333 1.250000 1.200000 1.166667 1.142857
#> 
#> $geometric_ratio
#> [1] 0.7429971
#> 
#> $attenuation
#> [1] 0.125
#> 
#> $vanishing
#> [1] FALSE
#> 
#> $tol
#> [1] 0.5
#> 
#> $estimate
#> [1] 0.7429971
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Per-layer gradient norms with the geometric mean decay factor towards the input"
#>