Maintains two tables and, with probability one half per step, applies the update of Sutton and Barto (2018) eq. 6.10, \(Q_1(S,A) \leftarrow Q_1(S,A) + \alpha [R + \gamma Q_2(S', \arg\max_a Q_1(S',a)) - Q_1(S,A)]\) else the same with the tables swapped; behaviour actions are epsilon-greedy in \(Q_1 + Q_2\). This is the boxed algorithm of their Section 6.7, p. 136; the estimator is van Hasselt (2010), and the same selection/evaluation decoupling applied to deep Q-networks is Double DQN, van Hasselt, Guez and Silver (2016), eq. 4. Determinism conventions mirror the Python arm bit-exactly: per step the stream yields one uniform for the epsilon test, one for the random action when exploring, one for the next-state draw (inverse CDF on the transition row), one for the coin (update the first table when u below one half); greedy ties break to the lowest action index.
Usage
Ddqn(
P,
R,
gamma,
alpha = 0.1,
epsilon = 0.1,
n_episodes = 100L,
start = 0L,
terminal = integer(0),
max_steps = 1000L,
seed = 0
)Arguments
- P
List of A transition matrices, each (S, S) row-stochastic.
- R
Matrix (S, A) of expected rewards, or a list of A (S, S) per-transition reward matrices averaged under P.
- gamma
Discount factor in 0 to 1.
- alpha
Step size in 0 to 1.
- epsilon
Exploration probability.
- n_episodes
Number of episodes.
- start
0-based start state.
- terminal
Integer vector of 0-based absorbing terminal states.
- max_steps
Step cap per episode.
- seed
SplitMix64 seed.
Value
List with estimate (average of the two (S, A)
tables), q1, q2, policy (0-based greedy
actions on the average), v, n_steps,
n_episodes, method.
References
Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction, 2nd ed., MIT Press, Section 6.7, boxed algorithm p. 136, eq. 6.10. van Hasselt, H. (2010). Double Q-learning. NeurIPS 23, 2613-2621. van Hasselt, H., Guez, A. and Silver, D. (2016). Deep reinforcement learning with Double Q-learning. AAAI 2016 (arXiv:1509.06461), eq. 4.