Asynchronous advantage actor-critic, A3C (Geron Ch 19, morie.fn hma3c)

Workers are stepped in deterministic round-robin: each pulls the shared parameters, rolls out one episode, and pushes its gradient into parameters other workers may have already moved – the staleness that makes A3C asynchronous. Worker w uses LCG seed + 7919 w.

Usage

morie_geron_a3c(
  env,
  actor,
  critic,
  n_workers = 4,
  lr = 0.1,
  epochs = 50,
  gamma = 0.99,
  critic_lr = NULL,
  max_steps = 200,
  seed = 0
)

Arguments

env

List with reset/step, or a function(worker_id) returning one.

actor, critic

Initial parameters.

n_workers

Workers.

lr, critic_lr

Step sizes.

epochs

Episodes per worker.

gamma

Discount.

max_steps

Episode cap.

seed

Base seed.

Value

List with actor, critic, returns, worker_returns, policy, updates.

Examples

a <- morie_geron_adagrad(2, s = 4, eta = 0.1, eps = 0)
env <- list(reset = function() 1, step = function(a) list(1,
    if (a == 0) 1 else 0, TRUE))
morie_geron_a3c(env, matrix(c(0, 0), 2, 1), 0, n_workers = 3,
    epochs = 10, lr = 0.5, seed = 3)
#> $actor
#>           [,1]
#> [1,]  2.058641
#> [2,] -2.058641
#> 
#> $critic
#> [1] 0.9999379
#> 
#> $returns
#>  [1] 1.0000000 0.6666667 0.6666667 0.6666667 1.0000000 0.6666667 1.0000000
#>  [8] 1.0000000 1.0000000 1.0000000
#> 
#> $worker_returns
#>      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
#> [1,]    1    1    0    1    1    1    1    1    1     1
#> [2,]    1    0    1    0    1    0    1    1    1     1
#> [3,]    1    1    1    1    1    1    1    1    1     1
#> 
#> $policy
#> function (state) 
#> {
#>     st <- as.numeric(state)
#>     .morie_gr_need(length(st) == n_feat, "policy: feature count mismatch")
#>     .morie_gr_softmax(as.numeric(shared_A %*% st))
#> }
#> <environment: 0x5621319257d8>
#> 
#> $updates
#> [1] 30
#> 
#> $estimate
#> [1] 1
#> 
#> $n
#> [1] 30
#> 
#> $method
#> [1] "A3C: round-robin workers pushing advantage gradients into shared parameters"
#>