Credit assignment over a trajectory (Geron Ch 18, morie.fn hmcrd) — morie_geron_credit

Discounted returns G_t = r_t + gamma G_{t+1} computed backwards, accumulating eligibility traces e_t = gamma lambda e_{t-1} + 1 forwards, and the per-step share of the total return. normalize uses the POPULATION sd (numpy default), not R's sd().

Usage

morie_geron_credit_assignment(
  trajectory,
  gamma = 0.95,
  lam = NULL,
  normalize = FALSE
)

Arguments

trajectory: Numeric rewards, or a list of (state, action, reward) triples.
gamma: Discount in [0, 1].
lam: Trace decay; default gamma.
normalize: Standardise the returns.

Value

List with returns, raw_returns, rewards, eligibility, credit, discounted_rewards, total_return.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_credit_assignment(V)
#> $returns
#> [1] 28.48456 28.93111 28.34854 26.68267 23.87650 19.87000 14.60000  8.00000
#> 
#> $raw_returns
#> [1] 28.48456 28.93111 28.34854 26.68267 23.87650 19.87000 14.60000  8.00000
#> 
#> $rewards
#> [1] 1 2 3 4 5 6 7 8
#> 
#> $eligibility
#> [1] 1.000000 1.902500 2.717006 3.452098 4.115519 4.714256 5.254616 5.742291
#> 
#> $credit
#> [1] 0.03510674 0.06670281 0.09505150 0.12039857 0.14297330 0.16298956 0.18064676
#> [8] 0.19613077
#> 
#> $discounted_rewards
#> [1] 1.000000 1.900000 2.707500 3.429500 4.072531 4.642686 5.145643 5.586698
#> 
#> $total_return
#> [1] 28.48456
#> 
#> $gamma
#> [1] 0.95
#> 
#> $lam
#> [1] 0.95
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "discounted returns, eligibility traces and per-step credit share"
#>