Tabular Q-learning update (Geron Ch 18, morie.fn hmql)

Off-policy TD control: the max over next actions is what biases the estimate upward. s, a and s_next are 0-based; max_next is NaN on a terminal transition, matching Python.

Usage

morie_geron_q_learning(Q, s, a, r, s_next, alpha, gamma, done = FALSE)

Arguments

Q

(states, actions) table.

s, a

0-based state/action.

r

Reward.

s_next

0-based next state.

alpha

In (0, 1].

gamma

In [0, 1].

done

Terminal.

Value

List with Q, td_error, target, old_value, new_value, max_next, estimate, n.

Examples

Q <- matrix(c(0, 1, 2, -1), nrow = 2, byrow = TRUE)
morie_geron_q_learning(Q, 0, 1, 0.5, 1, 0.5, 0.9)
#> $Q
#>      [,1]  [,2]
#> [1,]    0  1.65
#> [2,]    2 -1.00
#> 
#> $td_error
#> [1] 1.3
#> 
#> $target
#> [1] 2.3
#> 
#> $old_value
#> [1] 1
#> 
#> $new_value
#> [1] 1.65
#> 
#> $max_next
#> [1] 2
#> 
#> $estimate
#> [1] 1.65
#> 
#> $n
#> [1] 4
#> 
#> $method
#> [1] "Tabular Q-learning update (off-policy TD control)"
#>