Tabular Double Q-learning — morie

Maintains two independent action-value tables and, on each step, updates one of them using the other for evaluation: \(Q_1(s,a) \leftarrow Q_1(s,a) + \alpha[r + \gamma Q_2(s', \arg\max_{a'} Q_1(s',a')) - Q_1(s,a)]\) (and symmetrically). Because the table that selects the action is not the table that evaluates it, the positive bias of the single \(\max\) in morie_qlearn is removed (van Hasselt 2010).

Usage

morie_ddqn(
  P,
  R,
  gamma,
  alpha = 0.1,
  epsilon = 0.1,
  n_episodes = 100L,
  start = 0L,
  terminal = c(),
  max_steps = 1000L,
  seed = 0
)

Arguments

P: List of A transition matrices, each (S, S) with rows summing to 1.
R: Reward, (S, A) matrix or a length-A list of (S, S) matrices; see morie_mdpval.
gamma: Discount factor.
alpha: Step size.
epsilon: Exploration probability.
n_episodes: Number of episodes.
start: Start state, 0-based.
terminal: Terminal states, 0-based.
max_steps: Step cap per episode.
seed: Seed for the reproducible generator shared with the Python arm.

Value

A list with estimate (the averaged (S, A) table), q1, q2, policy, v, n_steps, n_episodes and method.

References

van Hasselt, H. (2010). Double Q-learning. Advances in Neural Information Processing Systems, 23, 2613-2621.

Examples

P <- list(matrix(c(0.9, 0.1, 0, 1), 2, byrow = TRUE), matrix(c(0.1, 0.9, 0, 1), 2, byrow = TRUE))
R <- matrix(c(0, 1, 0, 0), 2, 2)
morie_ddqn(P, R, gamma = 0.9, n_episodes = 50L, terminal = 1L)$policy
#> [1] 0 0