TMLE for the long-run value of a policy in a finite Markov decision process

The value is the stationary average reward V^pi = sum_s d^pi(s) r*(s, pi(s)), with d^pi the stationary distribution of the transition matrix induced by pi. The stationary average avoids inventing a discount factor the caller never supplied and is what a single long trajectory identifies.

Usage

Tmlmrk(state, action, reward, policy)

Arguments

state

State label at each step, in time order.

action

Action taken at each step.

reward

Reward received at each step.

policy

Action the evaluated policy takes in each distinct state, in sorted order of the state labels.

Value

List with estimate, se, eps, n_states, n.

Details

The reward model is the cell mean r(s, a); the behaviour policy is the empirical action frequency per state. The clever covariate is the importance ratio H = I(a = pi(s))/b(pi(s)|s) and r*(s, pi(s)) = r(s, pi(s)) + eps/b(pi(s)|s) with eps = sum H (R - r(s, a))/sum H^2. Transitions are counted from consecutive entries, so the input must be one trajectory in time order.

References

Liao, P., Klasnja, P. & Murphy, S. A. (2021). JASA 116(533):382-391, doi:10.1080/01621459.2020.1807993 (the influence curve w(S) H (R + h(S_next) - h(S) - V), S_next the following state); Murphy, S. A. (2003). JRSS B 65(2):331-355; van der Laan, M. J. & Rubin, D. (2006). IJB 2(1):11.

Examples

Tmlmrk(state = c(1, 2, 3, 4, 5, 6, 7, 8), action = c(1, 2, 3, 4, 5, 6, 7, 8),
  reward = c(1, 2, 3, 4, 5, 6, 7, 8), policy = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $estimate
#> [1] 8
#> 
#> $se
#> [1] 0
#> 
#> $eps
#> [1] 0
#> 
#> $n_states
#> [1] 8
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "TMLE for the long-run average reward of a policy in an MDP"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"