Backward-targeted Q-learning for a multi-stage treatment regime

Plain Q-learning is a sequence of regressions and inherits a bias at every stage, because the maximisation is applied to a fitted value and a fitted value is biased in the direction the maximiser looks. Targeting each stage before carrying it back removes the first-order part: at stage t the clever covariate H = I(a = a*(s))/b(a*(s)|s) fluctuates Q_t along the regime being evaluated, and only the targeted V_t(s) = Q*_t(s, a*(s)) is passed back as part of the pseudo-outcome. The behaviour policy is logistic in the state, so actions are assumed binary.

Usage

Tmlqlc(state, action, reward, time)

Arguments

state: State at each subject-stage.
action: Binary action at each subject-stage.
reward: Reward collected at each subject-stage.
time: Stage index of each row.

Value

List with estimate, se, n_stages, n_subj, n.

Details

Rows must be grouped by stage, with the subjects in the same order within each stage and equal counts per stage.

References

Murphy, S. A. (2003). JRSS B 65(2):331-355; Petersen, M. et al. (2014). Journal of Causal Inference 2(2):147-185.

Examples

Tmlqlc(state = c(1, 2, 3, 4, 5, 6, 7, 8), action = c(1, 2, 3, 4, 5, 6, 7, 8),
  reward = c(1, 2, 3, 4, 5, 6, 7, 8), time = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $estimate
#> [1] 36
#> 
#> $se
#> [1] NaN
#> 
#> $n_stages
#> [1] 8
#> 
#> $n_subj
#> [1] 1
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Backward-targeted Q-learning for a multi-stage regime"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"