Soft actor-critic (SAC): entropy-regularized max reward

Tabular SAC: soft state value V(s) = sum_a pi(a|s)(Q(s,a) - alpha log pi(a|s)), soft critic update Q += lr*(target - Q), closed-form policy improvement pi(.|s) = softmax(Q(s,.)/alpha).

Usage

morie_geron_sac(
  env,
  policy = NULL,
  critic = NULL,
  epochs = 20,
  lr = 0.5,
  alpha = 0.2,
  gamma = 0.9,
  steps = 20,
  seed = 0
)

Arguments

env

List/environment with functions reset(), step(a) returning list(s2, r, done), and integers n_states, n_actions.

policy

Optional initial (n_states, n_actions) probabilities; default uniform.

critic

Optional initial (n_states, n_actions) Q table; default zeros.

epochs, lr, alpha, gamma, steps, seed

As in the Python original.

Value

list with policy, Q, V, entropy, returns, estimate, n, method.

Examples

bandit <- list(n_states = 1L, n_actions = 2L, reset = function() 0L,
    step = function(a) list(0L, as.numeric(a), FALSE))
morie_geron_sac(bandit, epochs = 30, lr = 0.5, alpha = 0.05)
#> $policy
#>              [,1] [,2]
#> [1,] 1.200231e-83    1
#> 
#> $Q
#>            [,1]    [,2]
#> [1,] 0.03094794 9.57755
#> 
#> $V
#> [1] 9.57755
#> 
#> $entropy
#>  [1] 4.318721e-08 1.264378e-15 1.756178e-22 1.090831e-28 2.718816e-34
#>  [6] 2.407727e-39 6.731783e-44 5.322402e-48 1.075594e-51 5.067462e-55
#> [11] 5.120336e-58 1.028947e-60 3.841191e-63 2.504941e-65 2.699552e-67
#> [16] 4.573279e-69 1.164213e-70 4.276346e-72 2.185073e-73 1.502823e-74
#> [21] 1.350571e-75 1.544176e-76 2.192823e-77 3.784741e-78 7.786297e-79
#> [26] 1.876155e-79 5.211838e-80 1.645596e-80 5.830535e-81 2.291626e-81
#> 
#> $returns
#>  [1] 13 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
#> [26] 20 20 20 20 20
#> 
#> $alpha
#> [1] 0.05
#> 
#> $gamma
#> [1] 0.9
#> 
#> $estimate
#> [1] 20
#> 
#> $n
#> [1] 600
#> 
#> $method
#> [1] "Tabular soft actor-critic: soft value backup + closed-form Boltzmann policy improvement"
#>