Actor-critic, DDPG and beam-search front ends

R arm of morie.fn.a2cv, ddpgc and gptas, thin wrappers of the Geron-shelf implementations. A2cv: advantage actor-critic with linear softmax policy and value baseline (morie_geron_a2c; n_steps caps each rollout). Ddpgc: deep deterministic policy gradient with linear actor and critic, target networks and Ornstein-Uhlenbeck exploration (morie_geron_ddpg). Gptas: beam search decoding of width k (morie_geron_beam_search).

Usage

A2cv(
  env,
  actor,
  critic,
  n_steps = 200,
  epochs = 100,
  lr = 0.1,
  gamma = 0.99,
  critic_lr = NULL,
  seed = 0
)

Ddpgc(
  env,
  actor,
  critic,
  tau = 0.01,
  epochs = 20,
  lr = 0.01,
  gamma = 0.95,
  ou_theta = 0.15,
  ou_sigma = 0.2,
  seed = 0,
  s0 = NULL
)

Gptas(model, prompt, k = 3, max_len = 10, eos = NULL, length_penalty = 0)

Arguments

env

Environment (see the wrapped functions).

actor, critic

Initial actor and critic weights.

n_steps

Rollout cap.

epochs

Training episodes.

lr

Learning rate.

gamma

Discount factor.

critic_lr

Critic learning rate.

seed

Seed.

tau

Soft target-update rate.

ou_theta, ou_sigma

Ornstein-Uhlenbeck parameters.

s0

Initial state.

model

Next-token scorer model(src, prefix).

prompt

Source sequence.

k

Beam width.

max_len

Maximum length.

eos

End-of-sequence token.

length_penalty

Length penalty.

Value

The wrapped function's result.

References

Mnih, V. et al. (2016). Asynchronous methods for deep reinforcement learning. ICML, 1928-1937.

Lillicrap, T. P. et al. (2016). Continuous control with deep reinforcement learning. ICLR.

Sutskever, I., Vinyals, O. and Le, Q. V. (2014). Sequence to sequence learning with neural networks. NeurIPS, 3104-3112.

Examples

m <- function(src, prefix) if (length(prefix) %% 2 == 0) log(c(0.6, 0.4)) else log(c(0.3, 0.7))
Gptas(m, "hi", k = 2, max_len = 3)$sequence
#> [1] 0 1 0