State value function V^pi (Geron Ch 18, morie.fn grvpi)

Solves (I - gamma P_pi) V = r_pi exactly; no iteration, no convergence error. state is 0-based.

Usage

morie_geron_state_value_function(state, policy, transitions, rewards, gamma)

Arguments

state

0-based state index.

policy

Deterministic action vector or (S, A) stochastic matrix.

transitions

(S, A, S') array.

rewards

(S, A, S') or (S, A) array.

gamma

Discount in [0, 1).

Value

List with value, values, state.

Examples

P <- array(0, c(2, 2, 2))
P[1, , ] <- rbind(c(0.9, 0.1), c(0.2, 0.8))
P[2, , ] <- rbind(c(0.5, 0.5), c(0.1, 0.9))
R <- array(1, c(2, 2, 2))
R[2, , ] <- 0.2
pi <- rbind(c(0.5, 0.5), c(0.5, 0.5))
morie_geron_state_value_function(0L, pi, P, R, gamma = 0.9)
#> $value
#> [1] 5.819355
#> 
#> $values
#> [1] 5.819355 4.787097
#> 
#> $state
#> [1] 0
#> 
#> $estimate
#> [1] 5.819355
#> 
#> $n
#> [1] 2
#>