Gradient clipping by global norm (Geron Ch 11, morie.fn hmgcl) — morie_geron_gradient

The rescaling is uniform, so the update direction is unchanged. Pass a numeric vector/matrix for the single-tensor form, or a list of them for the multi-tensor form (which returns a list).

Usage

morie_geron_gradient_clipping(grads, max_norm, norm_type = 2)

Arguments

grads: Numeric vector/matrix, or a list of them.
max_norm: Positive threshold.
norm_type: p, or Inf.

Value

List with clipped, total_norm, scale, was_clipped, new_norm, max_norm, norm_type, estimate, n.

Examples

morie_geron_gradient_clipping(grads = c(1, 2, 3, 4, 5, 6, 7, 8), max_norm = 5L)
#> $clipped
#> [1] 0.35007 0.70014 1.05021 1.40028 1.75035 2.10042 2.45049 2.80056
#> 
#> $total_norm
#> [1] 14.28286
#> 
#> $scale
#> [1] 0.35007
#> 
#> $was_clipped
#> [1] TRUE
#> 
#> $new_norm
#> [1] 5
#> 
#> $max_norm
#> [1] 5
#> 
#> $norm_type
#> [1] 2
#> 
#> $estimate
#> [1] 5
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "global-norm gradient clipping"
#>