Iterative Q-learning (backward induction) for a dynamic regime

Q-learning solves the regime backwards. At the final stage the Q-function is fitted by least squares on the stage covariates, the action and their interaction; at earlier stages the same regression is run on the pseudo-outcome R_t + gamma max_a' Q_{t+1}(s', a'). The optimal rule at stage t is 1{Q_t(s,1) > Q_t(s,0)} and the value of the regime is the average of max_a Q_1(S_1, a). Every step is a closed-form least-squares solve; nothing is simulated.

Usage

Itrlrn(state, action, reward, time, gamma = 1)

Arguments

state: Stage covariates, one row per (subject, stage) record.
action: Binary action in that record.
reward: Reward received in that record.
time: Stage index; every stage must hold the same number of records in the same subject order.
gamma: Discount factor.

Value

List with estimate, value, stage_value, coef, share_treated, n_stages, n_subjects, gamma, n, method.

Details

Formula: Q_t(s,a) <- R_t + gamma max_a' Q_{t+1}(s', a').

References

Murphy (2003), Optimal dynamic treatment regimes, Journal of the Royal Statistical Society Series B 65(2):331-355, doi:10.1111/1467-9868.00389 ; Petersen et al (2014), Journal of Causal Inference 2(2):147-185. doi:10.1515/jci-2013-0007

Examples

state <- rep(c(0, 1), 8)
action <- rep(c(1, 0), 8)
reward <- rep(c(1, 0), 8)
time <- rep(1:2, each = 8)
Itrlrn(state, action, reward, time)
#> $estimate
#> [1] 1.888886
#> 
#> $value
#> [1] 1.888886
#> 
#> $stage_value
#> [1] 1.888886 0.833333
#> 
#> $coef
#> [1]  0.8888933 -0.2222274  1.1111067  0.0000000  0.3333341 -0.3333341  0.6666659
#> [8]  0.0000000
#> 
#> $share_treated
#> [1] 1 1
#> 
#> $n_stages
#> [1] 2
#> 
#> $n_subjects
#> [1] 8
#> 
#> $gamma
#> [1] 1
#> 
#> $n
#> [1] 16
#> 
#> $method
#> [1] "Q_t(s,a) <- R_t + gamma max_a' Q_{t+1}(s',a') by least squares, Murphy (2003)"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"