Adds the -m (i - j) distance bias to the scores and hands the softmax to
morie_alammar_sdp_attention – one attention core, reused.
Examples
morie_kamath_alibi_bias(Q = 0.5, K = c(1, 2, 3, 4, 5, 6, 7, 8), V = c(1, 2, 3, 4, 5, 6, 7, 8),
slopes = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $estimate
#> [1] 7.712832
#>
#> $output
#> $output[[1]]
#> [,1]
#> [1,] 7.712832
#>
#> $output[[2]]
#> [,1]
#> [1,] 7.910575
#>
#> $output[[3]]
#> [,1]
#> [1,] 7.968862
#>
#> $output[[4]]
#> [,1]
#> [1,] 7.988766
#>
#> $output[[5]]
#> [,1]
#> [1,] 7.995896
#>
#> $output[[6]]
#> [,1]
#> [1,] 7.998494
#>
#> $output[[7]]
#> [,1]
#> [1,] 7.999447
#>
#> $output[[8]]
#> [,1]
#> [1,] 7.999796
#>
#>
#> $attention
#> $attention[[1]]
#> [,1] [,2] [,3] [,4] [,5] [,6]
#> [1,] 2.139237e-05 9.587394e-05 0.0004296772 0.00192568 0.008630297 0.03867831
#> [,7] [,8]
#> [1,] 0.1733442 0.7768746
#>
#> $attention[[2]]
#> [,1] [,2] [,3] [,4] [,5]
#> [1,] 2.304884e-08 2.807923e-07 3.420751e-06 4.167328e-05 0.0005076844
#> [,6] [,7] [,8]
#> [1,] 0.006184863 0.07534705 0.917915
#>
#> $attention[[3]]
#> [,1] [,2] [,3] [,4] [,5]
#> [1,] 2.220591e-11 7.353587e-10 2.435174e-08 8.064187e-07 2.670492e-05
#> [,6] [,7] [,8]
#> [1,] 0.0008843455 0.0292855 0.9698026
#>
#> $attention[[4]]
#> [,1] [,2] [,3] [,4] [,5]
#> [1,] 2.064773e-14 1.858649e-12 1.673103e-10 1.506079e-08 1.355729e-06
#> [,6] [,7] [,8]
#> [1,] 0.0001220388 0.01098559 0.988891
#>
#> $attention[[5]]
#> [,1] [,2] [,3] [,4] [,5]
#> [1,] 1.896199e-17 4.639846e-15 1.135333e-12 2.778068e-10 6.797709e-08
#> [,6] [,7] [,8]
#> [1,] 1.663344e-05 0.00407007 0.9959132
#>
#> $attention[[6]]
#> [,1] [,2] [,3] [,4] [,5]
#> [1,] 1.733595e-20 1.153086e-17 7.669656e-15 5.101408e-12 3.393159e-09
#> [,6] [,7] [,8]
#> [1,] 2.256931e-06 0.001501179 0.9984966
#>
#> $attention[[7]]
#> [,1] [,2] [,3] [,4] [,5]
#> [1,] 1.582339e-23 2.860935e-20 5.172692e-17 9.352447e-14 1.690962e-10
#> [,6] [,7] [,8]
#> [1,] 3.057331e-07 0.0005527785 0.9994469
#>
#> $attention[[8]]
#> [,1] [,2] [,3] [,4] [,5] [,6]
#> [1,] 1.443411e-26 7.09403e-23 3.486552e-19 1.71356e-15 8.42175e-12 4.139095e-08
#> [,7] [,8]
#> [1,] 0.000203427 0.9997965
#>
#>
#> $bias
#> [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8]
#> [1,] 0 1 2 3 4 5 6 7
#>
#> $slopes
#> [1] 1 2 3 4 5 6 7 8
#>
#> $n
#> [1] 1
#>
#> $method
#> [1] "ALiBi biased attention (Kamath Ch 2; softmax core reused)"
#>