IMPALA V-trace off-policy targets — morie_impala

Truncated importance weights \(\rho = \min(\bar\rho, \pi/\mu)\) and \(c = \min(\bar c, \pi/\mu)\) applied to an n-step return.

Usage

morie_impala_vtrace(
  rewards,
  values,
  behavior_logp,
  target_logp,
  gamma = 0.99,
  rho_bar = 1,
  c_bar = 1,
  bootstrap_value = 0
)

Arguments

rewards, values: per-step rewards and value estimates.
behavior_logp, target_logp: log-probabilities under each policy.
gamma: discount.
rho_bar, c_bar: truncation levels.
bootstrap_value: value past the end of the segment.

Value

list with vs, advantage, rho, c, delta, ratio, n_truncated_rho.

Details

The two clips are not interchangeable and confusing them is the standard mistake. rho_bar determines WHAT is learned – the fixed point lies somewhere between the behaviour and target policies, at the behaviour policy when rho_bar is small and the target policy as it grows. c_bar determines HOW FAST the trace decays and does not move the fixed point at all.

References

Espeholt, L. et al. (2018). IMPALA: scalable distributed deep-RL with importance weighted actor-learner architectures. ICML, 1407-1416.

Examples

morie_impala_vtrace(c(1, 0, 1), c(0.5, 0.4, 0.3), c(-1, -1, -1),
                    c(-0.9, -1.2, -1))$vs
#> [1] 1.8742206 0.8830511 1.0000000