MuZero Reanalyze targets and prioritised-replay weights

z_t = sum_{j<n} gamma^j u_{t+j} + gamma^n nu_{t+n}; pi_t(a) = N_t(a)/sum_b N_t(b); p_i = |nu_i - z_i|; P(i) = p_i^alpha / sum_k p_k^alpha; w_i = ((1/N)(1/P(i)))^beta.

Usage

Mzreanal(
  rewards,
  freshvalues,
  visits,
  n = 5,
  gamma = 0.997,
  alpha = 1,
  beta = 1,
  oldvalues = NULL
)

Arguments

rewards

Environment rewards u_1..u_T.

freshvalues

Root values from the fresh search, length T.

visits

Fresh root visit counts, T x A.

n

Bootstrap horizon.

gamma

Discount factor.

alpha, beta

Prioritised-replay exponents.

oldvalues

Values for the priority gap; NULL uses freshvalues.

Value

List with target, policy, priority, prob, weight, T, A, n, gamma.

References

Schrittwieser et al. (2020), arXiv:1911.08265, Appendix H and Methods (training). Read from the ar5iv rendering.

Examples

Mzreanal(rewards = c(1, 2, 3, 4, 5, 6, 7, 8), freshvalues = c(1, 2, 3, 4, 5, 6, 7, 8),
  visits = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $target
#> [1] 20.79094 26.74612 32.70130 29.79067 25.86828 20.93107 14.97600  8.00000
#> 
#> $policy
#>      [,1]
#> [1,]    1
#> [2,]    1
#> [3,]    1
#> [4,]    1
#> [5,]    1
#> [6,]    1
#> [7,]    1
#> [8,]    1
#> 
#> $priority
#> [1] 19.79094 24.74612 29.70130 25.79067 20.86828 14.93107  7.97600  0.00000
#> 
#> $prob
#> [1] 0.13762405 0.17208183 0.20653960 0.17934552 0.14511573 0.10382904 0.05546423
#> [8] 0.00000000
#> 
#> $weight
#> [1] 0.9082715 0.7263986 0.6052108 0.6969786 0.8613815 1.2039021 2.2537047
#> [8]       Inf
#> 
#> $T
#> [1] 8
#> 
#> $A
#> [1] 1
#> 
#> $n
#> [1] 5
#> 
#> $gamma
#> [1] 0.997
#> 
#> $method
#> [1] "MuZero Reanalyze targets (Schrittwieser et al. 2020 App. H)"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"