NExT-GPT any-to-any pipeline (Kamath Ch 9)

encoders -> llm -> decoders, with every contract checked. No input may be silently dropped.

Usage

morie_kamath_nextgpt_any2any(
  inputs_by_modality,
  encoders,
  llm,
  decoders,
  output_modalities = NULL
)

Arguments

inputs_by_modality

Named list of inputs.

encoders

Named list.

llm

Function features -> state.

decoders

Named list.

output_modalities

Optional subset.

Value

List with outputs, features, llm_state.

Examples

f <- morie_kamath_ragas_faithfulness(c("the sky is blue", "the sky is green",
    "grass grows"), "the sky is blue today and grass grows")
morie_kamath_nextgpt_any2any(list(img = 1, txt = 2), list(img = function(v) v *
    10, txt = function(v) v * 100), function(f) sum(unlist(f)),
    list(aud = function(s) s + 1))
#> $outputs
#> $outputs$aud
#> [1] 211
#> 
#> 
#> $features
#> $features$img
#> [1] 10
#> 
#> $features$txt
#> [1] 200
#> 
#> 
#> $llm_state
#> [1] 210
#> 
#> $input_modalities
#> [1] "img" "txt"
#> 
#> $output_modalities
#> [1] "aud"
#> 
#> $estimate
#> [1] 1
#> 
#> $n
#> [1] 3
#> 
#> $method
#> [1] "NExT-GPT any-to-any encode / LLM / decode pipeline"
#>