Tabular Q-learning — morie

Off-policy temporal-difference control: behaves epsilon-greedily but bootstraps from the greedy action, i.e. the update of Sutton and Barto (2018), Eq. (6.8), \(Q(s,a) \leftarrow Q(s,a) + \alpha[r + \gamma \max_{a'} Q(s',a') - Q(s,a)]\).

Usage

morie_qlearn(
  P,
  R,
  gamma,
  alpha = 0.1,
  epsilon = 0.1,
  n_episodes = 100L,
  start = 0L,
  terminal = c(),
  max_steps = 1000L,
  seed = 0,
  Q0 = NULL
)

Arguments

P: List of A transition matrices, each (S, S) with rows summing to 1.
R: Reward, (S, A) matrix or a length-A list of (S, S) matrices; see morie_mdpval.
gamma: Discount factor.
alpha: Step size.
epsilon: Exploration probability.
n_episodes: Number of episodes.
start: Start state, 0-based.
terminal: Terminal states, 0-based.
max_steps: Step cap per episode.
seed: Seed for the reproducible generator shared with the Python arm.
Q0: Optional (S, A) initial action values; terminal rows are forced to zero.

Value

A list with estimate (the (S, A) Q table), policy (0-based greedy actions), v, n_steps, n_episodes and method.

References

Watkins, C. J. C. H. and Dayan, P. (1992). Q-learning. Machine Learning, 8, 279-292.

Examples

P <- list(matrix(c(0.9, 0.1, 0, 1), 2, byrow = TRUE), matrix(c(0.1, 0.9, 0, 1), 2, byrow = TRUE))
R <- matrix(c(0, 1, 0, 0), 2, 2)
morie_qlearn(P, R, gamma = 0.9, n_episodes = 50L, terminal = 1L)$policy
#> [1] 0 0