Double DQN targets (Geron Ch 19, morie.fn grddqn)

y = r + gamma Q_target(s', argmax_a Q_online(s', a)); the vanilla max target and the removed overestimation are returned alongside. selected_action is 0-based.

Usage

morie_geron_double_dqn_target(
  Q_online,
  Q_target,
  s_next,
  r,
  gamma,
  done = NULL
)

Arguments

Q_online, Q_target

Matrices (S, A).

s_next

0-based indices.

r

Rewards.

gamma

Discount.

done

Optional flags.

Value

List with target, selected_action, q_eval, vanilla_target, overestimation_gap.

Examples

morie_geron_double_dqn_target(matrix(c(0, 1, 2, 0), 2, 2, byrow = TRUE),
    matrix(c(10, -10, 1, 3), 2, 2, byrow = TRUE), s_next = c(0,
        1, 0), r = c(0, 1, 2), gamma = 0.9, done = c(FALSE, FALSE,
        TRUE))
#> $target
#> [1] -9.0  1.9  2.0
#> 
#> $selected_action
#> [1] 1 0 1
#> 
#> $q_eval
#> [1] -10   1 -10
#> 
#> $vanilla_target
#> [1] 9.0 3.7 2.0
#> 
#> $overestimation_gap
#> [1] 18.0  1.8  0.0
#> 
#> $gamma
#> [1] 0.9
#> 
#> $estimate
#> [1] -1.7
#> 
#> $n
#> [1] 3
#> 
#> $method
#> [1] "Double DQN target"
#>