RMSNorm (Kamath Ch 2)

x / sqrt(mean(x^2) + eps) * g; no mean subtraction, no bias.

Usage

morie_kamath_rms_norm(x, g = NULL, eps = 1e-06)

Arguments

x

Input rows.

g

Optional gain.

eps

Epsilon.

Value

List with y, tensor, rms.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_kamath_rms_norm(V)
#> $y
#> [1] 0.1980295 0.3960590 0.5940885 0.7921180 0.9901475 1.1881770 1.3862065
#> [8] 1.5842360
#> 
#> $tensor
#> [1] 0.1980295 0.3960590 0.5940885 0.7921180 0.9901475 1.1881770 1.3862065
#> [8] 1.5842360
#> 
#> $rms
#> [1] 5.049753
#> 
#> $estimate
#> [1] 0.1980295
#> 
#> $eps
#> [1] 1e-06
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "RMSNorm x / sqrt(mean(x^2) + eps) * g"
#>