MuZero pUCT action selection with min-max normalised values

score(a) = Qbar(s,a) + P(s,a) sqrt(sum_b N(s,b)) / (1 + N(s,a)) * (c1 + log((sum_b N(s,b) + c2 + 1)/c2)), with Qbar the tree-wide min-max rescaling of Q onto [0, 1].

Usage

Mzpuct(Q, N, P, c1 = 1.25, c2 = 19652, qmin = NULL, qmax = NULL)

Arguments

Q

Mean action values at this node.

N

Visit counts, same length as Q.

P

Prior policy, same length as Q.

c1, c2

Exploration constants; 1.25 and 19652 in the paper.

qmin, qmax

Tree-wide value bounds; NULL uses range(Q).

Value

List with score, qbar, exploration, best, sumn, k. best is a zero-based action index, matching the Python arm.

References

Schrittwieser et al. (2020), Nature 588, 604-609; arXiv:1911.08265, Equation (2) and the paragraph after Equation (4). Read from the ar5iv rendering of the arXiv source.

Examples

Mzpuct(Q = 0.5, N = 5L, P = 0.5)
#> $score
#> [1] 0.2329806
#> 
#> $qbar
#> [1] 0
#> 
#> $exploration
#> [1] 0.2329806
#> 
#> $best
#> [1] 0
#> 
#> $sumn
#> [1] 5
#> 
#> $k
#> [1] 1
#> 
#> $method
#> [1] "MuZero pUCT selection (Schrittwieser et al. 2020 eq. 2)"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"