Advantage actor-critic, A2C (Geron Ch 19, morie.fn hma2c)

Linear softmax policy plus linear value baseline, trained by Monte-Carlo advantage. env is a list with reset() and step(action) (action arrives 0-based, as in Python) returning list(state, reward, done). Actions are sampled from the same integer LCG stream as Python, so runs match token for token.

Usage

morie_geron_a2c(
  env,
  actor,
  critic,
  epochs = 100,
  lr = 0.1,
  gamma = 0.99,
  critic_lr = NULL,
  max_steps = 200,
  seed = 0
)

Arguments

env

List with reset/step.

actor

Matrix (n_actions, n_features).

critic

Value weights.

epochs

Episodes.

lr

Actor step.

gamma

Discount.

critic_lr

Critic step.

max_steps

Episode cap.

seed

LCG seed.

Value

List with actor, critic, returns, policy, value, advantages.

Examples

a <- morie_geron_adagrad(2, s = 4, eta = 0.1, eps = 0)
env <- list(reset = function() 1, step = function(a) list(1,
    if (a == 0) 1 else 0, TRUE))
morie_geron_a2c(env, matrix(c(0, 0), 2, 1), 0, epochs = 25, lr = 0.5,
    seed = 1)
#> $actor
#>           [,1]
#> [1,]  1.598361
#> [2,] -1.598361
#> 
#> $critic
#> [1] 0.998779
#> 
#> $returns
#>  [1] 1 1 1 0 1 1 1 1 1 1 1 1 1 0 1 0 1 1 1 1 1 1 1 1 1
#> 
#> $policy
#> function (state) 
#> {
#>     st <- as.numeric(state)
#>     .morie_gr_need(length(st) == n_feat, "policy: feature count mismatch")
#>     .morie_gr_softmax(as.numeric(A %*% st))
#> }
#> <environment: 0x562164b852a8>
#> 
#> $value
#> function (state) 
#> {
#>     st <- as.numeric(state)
#>     .morie_gr_need(length(st) == n_feat, "value: feature count mismatch")
#>     sum(V * st)
#> }
#> <environment: 0x562164b852a8>
#> 
#> $advantages
#> [1] 0.002441943
#> 
#> $estimate
#> [1] 1
#> 
#> $n
#> [1] 25
#> 
#> $method
#> [1] "A2C with a linear softmax policy and a linear value baseline"
#>