YaRN NTK-aware RoPE frequency rescaling (Kamath Ch 2)

theta_i / s^(2i/d); the top frequencies move least. With ramp the old and new frequencies are blended linearly across the given band.

Usage

morie_kamath_yarn_context_extrapolation(theta, scale, d, ramp = NULL)

Arguments

theta

RoPE base (scalar) or the d/2 frequencies.

scale

Context multiplier.

d

Embedding width (even).

ramp

Optional (lo, hi) blend band.

Value

List with theta, theta_new, scale_factors.

Examples

morie_kamath_yarn_context_extrapolation(10000, 4, 8)
#> $theta
#> [1] 1.000 0.100 0.010 0.001
#> 
#> $theta_new
#> [1] 1.0000000000 0.0707106781 0.0050000000 0.0003535534
#> 
#> $scale_factors
#> [1] 1.0000000 0.7071068 0.5000000 0.3535534
#> 
#> $effective_context_multiplier
#> [1] 4
#> 
#> $ramp
#> NULL
#> 
#> $estimate
#> [1] 0.0003535534
#> 
#> $d
#> [1] 8
#> 
#> $n
#> [1] 4
#> 
#> $method
#> [1] "YaRN NTK-aware RoPE frequency rescaling"
#>