Twin delayed DDPG (TD3): two critics + delayed policy updates

target = r + gamma * min(Q1_target, Q2_target)(s', a_tilde). Target policy smoothing perturbs the target action with probability noise; policy/target refresh (argmax + Polyak averaging at rate tau) happens only every policy_delay epochs.

Usage

morie_geron_td3(
  env,
  policy = NULL,
  Q1 = NULL,
  Q2 = NULL,
  epochs = 30,
  lr = 0.5,
  gamma = 0.9,
  steps = 20,
  policy_delay = 2,
  tau = 0.5,
  noise = 0.2,
  seed = 0
)

Arguments

env

List with reset(), step(a), n_states, n_actions.

policy

Optional initial deterministic policy (one 0-based action per state).

Q1, Q2

Optional initial (n_states, n_actions) critics; default zeros.

epochs, lr, gamma, steps, policy_delay, tau, noise, seed

As in Python original.

Value

list with policy, Q1, Q2, Q1_target, Q2_target, returns, overestimation_gap, policy_updates, estimate, n, method.

Examples

bandit <- list(n_states = 1L, n_actions = 2L, reset = function() 0L,
    step = function(a) list(0L, as.numeric(a), FALSE))
morie_geron_td3(bandit, epochs = 40)
#> $policy
#> [1] 1
#> 
#> $Q1
#>          [,1]     [,2]
#> [1,] 4.311675 5.902695
#> 
#> $Q2
#>          [,1]     [,2]
#> [1,] 4.311675 5.902695
#> 
#> $Q1_target
#>          [,1]     [,2]
#> [1,] 4.450581 5.675093
#> 
#> $Q2_target
#>          [,1]     [,2]
#> [1,] 4.450581 5.675093
#> 
#> $returns
#>  [1]  1  2 19 17 17 18 20 16 16 16 17 17 16 16 20 19 16 18 16 16 19 18 15 18 19
#> [26] 15 19 17 18 16 18 18 16 18 17 19 15 19 18 18
#> 
#> $overestimation_gap
#> [1] 0
#> 
#> $policy_updates
#> [1] 20
#> 
#> $estimate
#> [1] 18
#> 
#> $n
#> [1] 800
#> 
#> $method
#> [1] "Tabular TD3: twin critics, clipped target smoothing, delayed Polyak policy/target updates"
#>