Sinusoidal positional encoding (Geron Ch 16, morie.fn grpe)

PE(pos, 2i) = sin(pos / base^(2i/d)), PE(pos, 2i+1) = cos(...).

Usage

morie_geron_sinusoidal_positional_encoding(seq_len, d_model, base = 10000)

Arguments

seq_len

Positions.

d_model

Even model width, at least 2.

base

Frequency base greater than 1.

Value

List with encoding and wavelengths.

Examples

morie_geron_sinusoidal_positional_encoding(5, 6)
#> $encoding
#>            [,1]       [,2]       [,3]      [,4]        [,5]      [,6]
#> [1,]  0.0000000  1.0000000 0.00000000 1.0000000 0.000000000 1.0000000
#> [2,]  0.8414710  0.5403023 0.04639922 0.9989230 0.002154433 0.9999977
#> [3,]  0.9092974 -0.4161468 0.09269850 0.9956942 0.004308856 0.9999907
#> [4,]  0.1411200 -0.9899925 0.13879810 0.9903207 0.006463259 0.9999791
#> [5,] -0.7568025 -0.6536436 0.18459872 0.9828140 0.008617632 0.9999629
#> 
#> $wavelengths
#> [1]    6.283185  135.367124 2916.396276
#> 
#> $estimate
#>            [,1]       [,2]       [,3]      [,4]        [,5]      [,6]
#> [1,]  0.0000000  1.0000000 0.00000000 1.0000000 0.000000000 1.0000000
#> [2,]  0.8414710  0.5403023 0.04639922 0.9989230 0.002154433 0.9999977
#> [3,]  0.9092974 -0.4161468 0.09269850 0.9956942 0.004308856 0.9999907
#> [4,]  0.1411200 -0.9899925 0.13879810 0.9903207 0.006463259 0.9999791
#> [5,] -0.7568025 -0.6536436 0.18459872 0.9828140 0.008617632 0.9999629
#> 
#> $n
#> [1] 5
#>