MuZero n-step bootstrapped value targets

z_t = sum_{j=0}^{n-1} gamma^j u_{t+j} + gamma^n nu_{t+n}, with the bootstrap term dropped past the end of the trajectory.

Usage

Mznstep(rewards, values, n = 5, gamma = 0.997)

Arguments

rewards: Environment rewards, one per transition.
values: Search values at the same time steps.
n: Bootstrap horizon, >= 1.
gamma: Discount factor.

Value

List with target, T, n, gamma, mean.

References

Schrittwieser et al. (2020), Nature 588, 604-609; arXiv:1911.08265, Methods (training targets) and Equation (3). Read from the ar5iv rendering of the arXiv source.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Mznstep(V, V)
#> $target
#> [1] 20.79094 26.74612 32.70130 29.79067 25.86828 20.93107 14.97600  8.00000
#> 
#> $T
#> [1] 8
#> 
#> $n
#> [1] 5
#> 
#> $gamma
#> [1] 0.997
#> 
#> $mean
#> [1] 22.47555
#> 
#> $method
#> [1] "MuZero n-step value target (Schrittwieser et al. 2020)"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"