Global-norm gradient clipping (Geron Ch 11, morie.fn grgcl)

g *= min(1, c / (||g|| + 1e-12)) – the norm is capped and the direction preserved, so the cosine with the original is 1.

Usage

morie_geron_gradient_clipping_grgcl(gradients, c)

Arguments

gradients

Numeric object, or a list of them.

c

Positive threshold.

Value

List with clipped, total_norm, clipped_norm, clip_coef, was_clipped, cosine_with_original.

Examples

g <- list(matrix(c(3, -4, 1, 2), 2, 2), c(0.5, -0.5))
morie_geron_gradient_clipping_grgcl(g, c = 1)
#> $clipped
#> $clipped[[1]]
#>            [,1]      [,2]
#> [1,]  0.5432145 0.1810715
#> [2,] -0.7242860 0.3621430
#> 
#> $clipped[[2]]
#> [1]  0.09053575 -0.09053575
#> 
#> 
#> $total_norm
#> [1] 5.522681
#> 
#> $clipped_norm
#> [1] 1
#> 
#> $clip_coef
#> [1] 0.1810715
#> 
#> $was_clipped
#> [1] TRUE
#> 
#> $cosine_with_original
#> [1] 1
#> 
#> $threshold
#> [1] 1
#> 
#> $estimate
#> [1] 1
#> 
#> $n
#> [1] 6
#>