Sinusoidal positional encoding (Geron Ch 16, morie.fn hmpe)

PE[p, 2i] = sin(p / base^(2i/d)), PE[p, 2i+1] = cos(...). The rotation_check field is the max error of expressing PE(p + 1) as a fixed rotation of PE(p) – the property that lets the encoding extrapolate past the training length. It should be ~1e-16.

Usage

morie_geron_positional_encoding(pos, d_model, base = 10000)

Arguments

pos

Position or vector of positions.

d_model

Even width.

base

Frequency base (> 1).

Value

List with pe, wavelengths, rotation_check, d_model, estimate, n.

Examples

morie_geron_positional_encoding(c(0, 1, 2, 7), 6)
#> $pe
#>           [,1]       [,2]       [,3]      [,4]        [,5]      [,6]
#> [1,] 0.0000000  1.0000000 0.00000000 1.0000000 0.000000000 1.0000000
#> [2,] 0.8414710  0.5403023 0.04639922 0.9989230 0.002154433 0.9999977
#> [3,] 0.9092974 -0.4161468 0.09269850 0.9956942 0.004308856 0.9999907
#> [4,] 0.6569866  0.7539023 0.31922465 0.9476791 0.015080471 0.9998863
#> 
#> $wavelengths
#> [1]    6.283185  135.367124 2916.396276
#> 
#> $rotation_check
#> [1] 1.110223e-16
#> 
#> $d_model
#> [1] 6
#> 
#> $estimate
#>           [,1]       [,2]       [,3]      [,4]        [,5]      [,6]
#> [1,] 0.0000000  1.0000000 0.00000000 1.0000000 0.000000000 1.0000000
#> [2,] 0.8414710  0.5403023 0.04639922 0.9989230 0.002154433 0.9999977
#> [3,] 0.9092974 -0.4161468 0.09269850 0.9956942 0.004308856 0.9999907
#> [4,] 0.6569866  0.7539023 0.31922465 0.9476791 0.015080471 0.9998863
#> 
#> $n
#> [1] 4
#> 
#> $method
#> [1] "Sinusoidal positional encoding (Vaswani et al. form)"
#>