Q-value iteration (Geron Ch 19, morie.fn grbo)

Q*(s,a) = sum_s' T(s,a,s')[R(s,a,s') + gamma max_a' Q*(s',a')]. Rows of transitions may sum to 1 (available) or 0 (unavailable). policy is 0-based.

Usage

morie_geron_q_value_iteration(
  Q,
  transitions,
  rewards,
  gamma,
  max_iter = 1000,
  tol = 1e-10
)

Arguments

Q

Initial table (S, A).

transitions

Array (S, A, S).

rewards

Array (S, A, S) or matrix (S, A).

gamma

Discount.

max_iter, tol

Stopping rule.

Value

List with Q, V, policy, residual, iterations, converged.

Examples

P <- array(0, dim = c(2, 2, 2))
R <- matrix(c(1, 0, 0, 2), 2, 2, byrow = TRUE)
morie_geron_q_value_iteration(matrix(0, 2, 2), P, R, 0.9)
#> $Q
#>      [,1] [,2]
#> [1,]    0    0
#> [2,]    0    0
#> 
#> $V
#> [1] 0 0
#> 
#> $policy
#> [1] 0 0
#> 
#> $residual
#> [1] 0
#> 
#> $iterations
#> [1] 1
#> 
#> $converged
#> [1] TRUE
#> 
#> $gamma
#> [1] 0.9
#> 
#> $estimate
#> [1] 0
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "Q-value iteration (Bellman optimality)"
#>