RetNet retention (Kamath Ch 2)

((Q K^T) .* gamma^(i-j) causal) V, with the recurrent form returned alongside as the equality check the parallel form must satisfy.

Usage

morie_kamath_retnet_retention(Q, K, V, gamma)

Arguments

Q, K, V

Sequence matrices.

gamma

Decay in (0, 1].

Value

List with output, decay, recurrent_output.

Examples

morie_kamath_retnet_retention(Q = c(1, 2, 3, 4, 5, 6, 7, 8), K = c(1, 2, 3, 4, 5, 6, 7, 8),
  V = c(1, 2, 3, 4, 5, 6, 7, 8), gamma = 0.5)
#> $output
#>          [,1]
#> [1,]   1.0000
#> [2,]   9.0000
#> [3,]  33.7500
#> [4,]  86.5000
#> [5,] 179.0625
#> [6,] 323.4375
#> [7,] 531.6719
#> [8,] 815.8125
#> 
#> $decay
#>           [,1]     [,2]    [,3]   [,4]  [,5] [,6] [,7] [,8]
#> [1,] 1.0000000 0.000000 0.00000 0.0000 0.000 0.00  0.0    0
#> [2,] 0.5000000 1.000000 0.00000 0.0000 0.000 0.00  0.0    0
#> [3,] 0.2500000 0.500000 1.00000 0.0000 0.000 0.00  0.0    0
#> [4,] 0.1250000 0.250000 0.50000 1.0000 0.000 0.00  0.0    0
#> [5,] 0.0625000 0.125000 0.25000 0.5000 1.000 0.00  0.0    0
#> [6,] 0.0312500 0.062500 0.12500 0.2500 0.500 1.00  0.0    0
#> [7,] 0.0156250 0.031250 0.06250 0.1250 0.250 0.50  1.0    0
#> [8,] 0.0078125 0.015625 0.03125 0.0625 0.125 0.25  0.5    1
#> 
#> $recurrent_output
#>          [,1]
#> [1,]   1.0000
#> [2,]   9.0000
#> [3,]  33.7500
#> [4,]  86.5000
#> [5,] 179.0625
#> [6,] 323.4375
#> [7,] 531.6719
#> [8,] 815.8125
#> 
#> $estimate
#> [1] 815.8125
#> 
#> $gamma
#> [1] 0.5
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "RetNet retention ((QK^T) .* D) V"
#>