Skip to contents

Bahdanau (additive) attention (Geron Ch 14, morie.fn hmbdn and grbah)

Source: R/geron_ml_native.R
morie_geron_bahdanau_attention.Rd

e_i = v^T tanh(W h_i + U s + b); alpha = softmax(e); context = alpha h. ONE R function serves both Python modules, whose only difference is argument order and payload naming: grbah's geron_bahdanau_attention(decoder_state, encoder_states, Wh, Ws, v) maps onto this call as h = encoder_states, s_prev = decoder_state, W = Ws, U = Wh. Both payload spellings are returned (alpha is hmbdn's, weights grbah's), and argmax is 0-based in both.

Usage

morie_geron_bahdanau_attention(h, s_prev, W, U, v, b = NULL)

Arguments

h

Encoder states (T, d_h).

s_prev

Decoder state (d_s).

W

(d_a, d_h).

U

(d_a, d_s).

v

(d_a).

b

Optional bias (d_a).

Value

List with alpha, weights, scores, context, entropy, argmax.

Examples

morie_geron_bahdanau_attention(h = 0.5, s_prev = 5L, W = c(1, 2, 3, 4, 5, 6, 7, 8),
  U = c(1, 2, 3, 4, 5, 6, 7, 8), v = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $alpha
#> [1] 1
#> 
#> $weights
#> [1] 1
#> 
#> $scores
#> [1] 35.99997
#> 
#> $context
#> [1] 0.5
#> 
#> $entropy
#> [1] 0
#> 
#> $argmax
#> [1] 0
#> 
#> $estimate
#> [1] 1
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "Bahdanau additive attention e = v^T tanh(W h + U s)"
#>