Adversarial Inverse Reinforcement Learning

Fit the AIRL discriminator and return the recovered reward, following Fu, Luo & Levine (2018) eq. 4 and Algorithm 1.

Usage

airl(
  expert_states,
  expert_actions,
  expert_next,
  expert_log_policy,
  policy_states,
  policy_actions,
  policy_next,
  policy_log_policy,
  gamma = 0.99,
  state_only = TRUE,
  lr = 0.1,
  epochs = 500L,
  l2 = 0
)

Arguments

expert_states, expert_actions, expert_next

Expert transitions.

expert_log_policy

log pi(a|s) under the current policy.

policy_states, policy_actions, policy_next

Policy transitions.

policy_log_policy

log pi(a|s) for the policy transitions.

gamma

Discount used in the shaping term.

state_only

TRUE: parameterise g on s alone (Theorem C.1); FALSE: g(s, a).

lr, epochs, l2

Full-batch gradient ascent on the logistic log-likelihood with optional ridge penalty.

Value

List with reward (line 6 of Algorithm 1), g, h, f_policy, f_expert, D_policy, D_expert, accuracy, log_likelihood, gamma, state_only, method.

Examples

# \donttest{
set.seed(19)
es <- sample(0:3, 50, TRUE); ea <- rep(1, 50); en <- pmin(es + 1, 3)
ps <- sample(0:3, 50, TRUE); pa <- rep(0, 50); pn <- pmax(ps - 1, 0)
r <- airl(es, ea, en, rep(log(0.5), 50), ps, pa, pn, rep(log(0.5), 50), epochs = 200L)
r$accuracy
#> [1] 1
# }