Q-learning update (Geron Ch 18, morie.fn grql)

Q(s,a) += alpha [r + gamma max_a' Q(s',a') - Q(s,a)]; done drops the bootstrap. State and action indices are 0-based.

Usage

morie_geron_q_learning_update(Q, s, a, r, s_next, alpha, gamma, done = FALSE)

Arguments

Q

(n_states, n_actions) table.

s, a, s_next

0-based indices.

r

Finite reward.

alpha

Step size in (0, 1].

gamma

Discount in [0, 1].

done

Terminal flag.

Value

List with Q, old_value, new_value, target, td_error, max_next.

Examples

Q <- matrix(0.2, 4, 2)
morie_geron_q_learning_update(Q, s = 0L, a = 1L, r = 1, s_next = 2L,
                              alpha = 0.1, gamma = 0.9)
#> $Q
#>      [,1]  [,2]
#> [1,]  0.2 0.298
#> [2,]  0.2 0.200
#> [3,]  0.2 0.200
#> [4,]  0.2 0.200
#> 
#> $old_value
#> [1] 0.2
#> 
#> $new_value
#> [1] 0.298
#> 
#> $target
#> [1] 1.18
#> 
#> $td_error
#> [1] 0.98
#> 
#> $max_next
#> [1] 0.2
#> 
#> $estimate
#> [1] 0.298
#> 
#> $n
#> [1] 4
#>