State value function by exact policy evaluation (Geron Ch 18, morie.fn hmvf)

V = (I - gamma P_pi)^-1 r_pi, solved as a linear system. This is the INDEPENDENT route the parity tests check morie_geron_value_iteration against: the two must agree to solver tolerance on the same MDP.

Usage

morie_geron_value_function(s, pi, gamma, P = NULL, R = NULL)

Arguments

s

0-based query state.

pi

Policy.

gamma

In [0, 1).

P

(n_states, n_actions, n_states) array.

R

(n_states, n_actions) or (n_states, n_actions, n_states).

Value

List with V, value, r_pi, P_pi, residual, gamma, state, estimate, n.

Details

pi is either a length-n_states vector of 0-based action indices or an (n_states, n_actions) stochastic matrix; s is 0-based.

Examples

P <- array(0, c(2, 2, 2))
P[1, , ] <- rbind(c(0.9, 0.1), c(0.2, 0.8))
P[2, , ] <- rbind(c(0.5, 0.5), c(0.1, 0.9))
R <- matrix(c(1, 0.5, 0.2, 0), 2, 2)
pi <- rbind(c(0.5, 0.5), c(0.5, 0.5))
r <- morie_geron_value_function(0L, pi, gamma = 0.9, P = P, R = R)
str(r, max.level = 1)
#> List of 10
#>  $ V       : num [1:2] 4.17 3.72
#>  $ value   : num 4.17
#>  $ r_pi    : num [1:2] 0.6 0.25
#>  $ P_pi    : num [1:2, 1:2] 0.55 0.3 0.45 0.7
#>  $ residual: num 4.44e-16
#>  $ gamma   : num 0.9
#>  $ state   : int 0
#>  $ estimate: num 4.17
#>  $ n       : int 2
#>  $ method  : chr "Exact policy evaluation V = (I - gamma P_pi)^-1 r_pi"