Soft policy iteration on an explicit finite MDP

The exact tabular algorithm underlying soft actor-critic (Haarnoja et al. 2018, Section 4.1): soft policy evaluation applies the soft Bellman backup of eqs. (2)-(3), Q <- r + gamma E[V], V = E_pi[Q - temp log pi]; soft policy improvement projects onto softmax(Q/temp) (eq. 4), the exact minimizer for the tabular class, with monotone improvement and convergence per Lemmas 1-2 and Theorem 1. At the fixed point V(s) = temp logsumexp(Q(s,.)/temp) (the test anchor). temp is the entropy weight alpha of eq. (1).

Usage

Sacc(
  P,
  R,
  gamma,
  temp = 1,
  tol = 1e-12,
  max_eval = 100000L,
  max_improve = 1000L
)

Arguments

P

List of A transition matrices, each (S x S).

R

Reward matrix (S x A) or list of A (S x S) matrices.

gamma

Discount factor in [0, 1).

temp

Entropy temperature alpha > 0.

tol

Convergence threshold.

max_eval, max_improve

Iteration caps.

Value

List with estimate (soft V*), policy, q, entropy, n_improve, n_eval, converged, method.

References

Haarnoja, T., Zhou, A., Abbeel, P. and Levine, S. (2018), ICML 2018, arXiv:1801.01290, Section 4.1, eqs. (2)-(4), Lemmas 1-2, Theorem 1.

Examples

Sacc(P = 1, R = 5L, gamma = 0.5)
#> $estimate
#> [1] 10
#> 
#> $policy
#>      [,1]
#> [1,]    1
#> 
#> $q
#>      [,1]
#> [1,]   10
#> 
#> $entropy
#> [1] 0
#> 
#> $n_improve
#> [1] 1
#> 
#> $n_eval
#> [1] 44
#> 
#> $converged
#> [1] TRUE
#> 
#> $method
#> [1] "Soft policy iteration (maximum-entropy RL, exact tabular SAC core)"
#>