Value iteration on the Bellman optimality operator (Geron Ch 19, hmbel)

V*(s) = max_a [R(s,a) + gamma sum_s' P(s'|s,a) V*(s')]. policy is 0-based.

Usage

morie_geron_value_iteration(V, P, R, gamma, tol = 1e-10, max_iter = 10000)

Arguments

V

Initial values (S).

P

Array (S, A, S).

R

Matrix (S, A).

gamma

Discount in [0, 1).

tol, max_iter

Stopping rule.

Value

List with V, policy, Q, iterations, residual, converged.

Examples

P <- array(0, c(2, 2, 2))
P[1, , ] <- rbind(c(0.9, 0.1), c(0.2, 0.8))
P[2, , ] <- rbind(c(0.5, 0.5), c(0.1, 0.9))
R <- matrix(c(1, 0.5, 0.2, 0), 2, 2)
r <- morie_geron_value_iteration(c(0, 0), P, R, gamma = 0.9)
str(r, max.level = 1)
#> List of 10
#>  $ V         : num [1:2] 9.3 8.52
#>  $ policy    : int [1:2] 0 0
#>  $ Q         : num [1:2, 1:2] 9.3 8.52 8 7.73
#>  $ iterations: int 219
#>  $ residual  : num 9.71e-11
#>  $ converged : logi TRUE
#>  $ gamma     : num 0.9
#>  $ estimate  : num 9.3
#>  $ n         : int 2
#>  $ method    : chr "Value iteration on the Bellman optimality operator"