Exploding gradients: gradients grow through layers

Mirror of morie_geron_vanishing_gradients: geometric-mean amplification per layer towards the input; optional global-norm clipping g <- g * clip_norm / ||g||_global.

Usage

morie_geron_exploding_gradients(grads, tol = 2, clip_norm = NULL)

Arguments

grads

List of >= 2 per-layer gradients, input side first.

tol

Amplification threshold (> 1).

clip_norm

Optional clipping threshold (> 0).

Value

list with norms, ratios, geometric_ratio, amplification, exploding, global_norm, clipped, scale, tol, estimate, n, method.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_exploding_gradients(V)
#> $norms
#> [1] 1 2 3 4 5 6 7 8
#> 
#> $ratios
#> [1] 2.000000 1.500000 1.333333 1.250000 1.200000 1.166667 1.142857
#> 
#> $geometric_ratio
#> [1] 0.7429971
#> 
#> $amplification
#> [1] 0.125
#> 
#> $exploding
#> [1] FALSE
#> 
#> $global_norm
#> [1] 14.28286
#> 
#> $clipped
#> NULL
#> 
#> $scale
#> [1] 1
#> 
#> $tol
#> [1] 2
#> 
#> $estimate
#> [1] 0.7429971
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Per-layer gradient norms with geometric amplification and optional global-norm clipping"
#>