The boxed algorithm of Sutton and Barto (2018), Section 6.4, p. 130: after taking A in S and observing R and the next state, the next action is chosen epsilon-greedily first and the update uses the quintuple (S, A, R, S2, A2): \(Q(S,A) \leftarrow Q(S,A) + \alpha [R + \gamma Q(S',A') - Q(S,A)]\) The name and algorithm come from the modified connectionist Q-learning of Rummery and Niranjan (1994). Q(terminal, .) = 0. Determinism conventions match the Python arm bit-exactly: greedy ties to the lowest action index; the epsilon-greedy draw consumes one uniform for the epsilon test plus one for the random action when exploring; the next state consumes one uniform by inverse CDF on the transition row; the initial action of an episode is drawn before the step loop, exactly as in the source box.
Usage
Sarsa(
P,
R,
gamma,
alpha = 0.1,
epsilon = 0.1,
n_episodes = 100L,
start = 0L,
terminal = integer(0),
max_steps = 1000L,
seed = 0,
Q0 = NULL
)Arguments
- P
List of A transition matrices, each (S, S) row-stochastic.
- R
Matrix (S, A) of expected rewards, or a list of A (S, S) per-transition reward matrices averaged under P.
- gamma
Discount factor in 0 to 1.
- alpha
Step size in 0 to 1.
- epsilon
Exploration probability.
- n_episodes
Number of episodes.
- start
0-based start state.
- terminal
Integer vector of 0-based absorbing terminal states.
- max_steps
Step cap per episode.
- seed
SplitMix64 seed.
- Q0
Optional (S, A) initial action values.
Value
List with estimate (the learned (S, A) Q table),
policy (0-based greedy actions), v, n_steps,
n_episodes, method.
References
Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction, 2nd ed., MIT Press, Section 6.4, boxed algorithm p. 130. Rummery, G. A. and Niranjan, M. (1994). On-line Q-learning using connectionist systems. Technical Report CUED/F-INFENG/TR 166, Cambridge University Engineering Department.