Action-value function Q^pi (Geron Ch 18, morie.fn grqpi)

Q = r_sa + gamma sum_s' P V^pi(s'); V from morie_geron_state_value_function()'s evaluation. Indices 0-based.

Usage

morie_geron_action_value_function(
  state,
  action,
  policy,
  transitions,
  rewards,
  gamma
)

Arguments

state, action

0-based indices.

policy, transitions, rewards, gamma

MDP specification.

Value

List with q_value, q_values, values, advantage, greedy_action.

Examples

P <- array(0, c(2, 2, 2))
P[1, , ] <- rbind(c(0.9, 0.1), c(0.2, 0.8))
P[2, , ] <- rbind(c(0.5, 0.5), c(0.1, 0.9))
R <- array(1, c(2, 2, 2))
R[2, , ] <- 0.2
pi <- rbind(c(0.5, 0.5), c(0.5, 0.5))
morie_geron_action_value_function(0L, 1L, pi, P, R, gamma = 0.9)
#> $q_value
#> [1] 5.494194
#> 
#> $q_values
#>          [,1]     [,2]
#> [1,] 6.144516 5.494194
#> [2,] 4.972903 4.601290
#> 
#> $values
#> [1] 5.819355 4.787097
#> 
#> $advantage
#> [1] -0.3251613
#> 
#> $greedy_action
#> [1] 0
#> 
#> $estimate
#> [1] 5.494194
#> 
#> $n
#> [1] 2
#>