Rotary position embedding rescaled to reach past the trained length

Interpolating every frequency by the same factor stretches the high-frequency dimensions too, and those carry local word order, so plain position interpolation blurs what attention needs at short range. The NTK-aware variant scales the base instead: low frequencies stretch a lot, high frequencies barely at all.

Usage

Ropedy(y, q, m, theta = 10000, L_new = NULL, L_train = NULL)

Arguments

y

Unused; keeps the family signature.

q

Query or key vector, d even.

m

Position index.

theta

Base.

L_new, L_train

Target and trained context lengths.

Value

List with estimate, theta_base, freqs, scale, d.

Details

Formula: \(theta' = theta (L_new/L_train)^(d/(d-2))\) inside \(theta_i = theta'^(-2i/d)\), then rotate by m theta_i.

References

Su et al. (2024), RoFormer, Neurocomputing 568:127063. The NTK-aware base rescaling is bloc97 (2023), a LocalLLaMA community note, cited as such because no peer-reviewed source exists.

Examples

Ropedy(y = c(1, 2, 3, 4, 5, 6, 7, 8), q = 0.5, m = 5L)
#> $estimate
#> [1] 0.5
#> 
#> $theta_base
#> [1] 10000
#> 
#> $freqs
#> numeric(0)
#> 
#> $scale
#> [1] 1
#> 
#> $d
#> [1] 1
#> 
#> $method
#> [1] "NTK-aware dynamically scaled RoPE"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"