RWKV time-mixing (Kamath Ch 2)

Decayed normalised value average, computed in log space so long decays never overflow.

Usage

morie_kamath_rwkv_time_mix(k, v, w, u = 0)

Arguments

k, v

Key and value sequences.

w

Decay (non-negative).

u

Current-token bonus.

Value

List with wkv, estimate.

Examples

morie_kamath_rwkv_time_mix(k = c(1, 2, 3, 4, 5, 6, 7, 8), v = c(1, 2, 3, 4, 5, 6, 7, 8), w = 5L)
#> $wkv
#> [1] 1.000000 1.997527 2.997515 3.997515 4.997515 5.997515 6.997515 7.997515
#> 
#> $estimate
#> [1] 7.997515
#> 
#> $w
#> [1] 5
#> 
#> $u
#> [1] 0
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "RWKV time-mixing (log-space stable scan)"
#>