MuZero categorical value head and its invertible scaling

Softmax the head over the integer support -support..support, take the expectation y, then invert h(x) = sign(x)(sqrt(|x|+1) - 1 + eps x): h^{-1}(y) = sign(y)[((sqrt(1 + 4 eps(|y| + 1 + eps)) - 1)/(2 eps))^2 - 1].

Usage

Mzvalue(logits, support = 300, epsilon = 0.001)

Arguments

logits: Head outputs over the support, length 2*support + 1.
support: Half-width of the integer support.
epsilon: eps in the transform.

Value

List with value, expected, prob, support, epsilon, k.

References

Schrittwieser et al. (2020), arXiv:1911.08265, Appendix F. Read from the ar5iv rendering of the arXiv source.

Examples

Mzvalue(c(0, 1, 0), support = 1)
#> $value
#> [1] 1.776357e-15
#> 
#> $expected
#> [1] 0
#> 
#> $prob
#> [1] 0.2119416 0.5761169 0.2119416
#> 
#> $support
#> [1] 1
#> 
#> $epsilon
#> [1] 0.001
#> 
#> $k
#> [1] 3
#> 
#> $method
#> [1] "MuZero categorical value head (Schrittwieser et al. 2020 App. F)"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"