Sequence-to-sequence encoder-decoder architecture

enc(src) -> z (context vector); dec(z, prefix) -> next-token scores. Runs teacher-forced cross-entropy against tgt and a free-running greedy decode.

Usage

morie_geron_seq2seq(src, tgt, encoder, decoder, max_len = NULL, eos = NULL)

Arguments

src

Source sequence, passed to encoder unchanged.

tgt

Integer target token ids (0-based), used for teacher forcing.

encoder

Function(src) -> numeric context vector z.

decoder

Function(z, prefix) -> numeric scores, length V >= 2 (prefix is 0-based ids).

max_len

Greedy decode length; default length(tgt).

eos

Optional token id (0-based) that stops greedy decoding.

Value

list with z, loss, perplexity, token_logprobs, greedy, greedy_logprob, greedy_matches, exposure_bias, vocab_size, estimate, n, method.

Examples

enc <- function(src) as.numeric(src)
dec <- function(z, prefix) {
  k <- length(prefix)
  sc <- rep(-4, 4)
  sc[(k %% 4) + 1] <- 4
  sc
}
r <- morie_geron_seq2seq(c(0.5, 0.1), c(0L, 1L, 2L), enc, dec)
str(r, max.level = 1)
#> List of 12
#>  $ z             : num [1:2] 0.5 0.1
#>  $ loss          : num 0.00101
#>  $ perplexity    : num 1
#>  $ token_logprobs: num [1:3] -0.00101 -0.00101 -0.00101
#>  $ greedy        : int [1:3] 0 1 2
#>  $ greedy_logprob: num -0.00302
#>  $ greedy_matches: int 3
#>  $ exposure_bias : num 0
#>  $ vocab_size    : int 4
#>  $ estimate      : num 0.00101
#>  $ n             : int 3
#>  $ method        : chr "Encoder-decoder: teacher-forced cross-entropy plus greedy free-running decode"