Rotary positional embedding (Kamath Ch 2)

Rotates feature PAIRS by m * base^(-2i/d) at explicit positions.

Usage

morie_kamath_rotary_positional_embedding(q, positions = NULL, base = 10000)

Arguments

q

(d,) or (seq_len, d) input.

positions

Optional positions.

base

RoPE base.

Value

List with y, angles, theta, positions.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_kamath_rotary_positional_embedding(V)
#> $y
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,]    1    2    3    4    5    6    7    8
#> 
#> $angles
#>      [,1] [,2] [,3] [,4]
#> [1,]    0    0    0    0
#> 
#> $theta
#> [1] 1.000 0.100 0.010 0.001
#> 
#> $positions
#> [1] 0
#> 
#> $estimate
#> [1] 1
#> 
#> $base
#> [1] 10000
#> 
#> $d
#> [1] 8
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "Rotary positional embedding at explicit positions"
#>