Skip to contents

Compare a fresh copy of a capsule's data with the pinned one

Source: R/capsule.R
morie_capsule_drift.Rd

Reads the CSV the capsule pins (resource$filename in its provenance record) and compares it, column by column, with a freshly downloaded copy: a digest only says the bytes changed, this says whether the distributions did. This is rmoriebricklayer::cpsdrft() (Kolmogorov-Smirnov and PSI for numeric and date columns, a two-sample homogeneity test for categorical ones; added and removed columns listed).

Usage

morie_capsule_drift(
  dir,
  current,
  provenance_file = "data_provenance.json",
  alpha = 0.01,
  psi_threshold = 0.25,
  psi_min_n = 1000L,
  bins = 10L,
  id_levels = 100L,
  seed = 1L
)

Arguments

dir

The capsule directory, a single string.

current

The fresh data: a data frame, or the path to a CSV.

provenance_file

The provenance record, relative to dir.

alpha

Significance level for a column's drifted flag.

psi_threshold

PSI above which a column is flagged on its own once both samples have at least psi_min_n values.

psi_min_n

Minimum size of both samples before the PSI may flag.

bins

Bins for the PSI.

id_levels

A text column with more distinct values than this is an identifier and is not tested.

seed

Seed for the Monte Carlo branch of the homogeneity test; the caller's random stream is left untouched.

Value

A bricklayer_drift list: columns (one row per shared column: column, type, statistic, p_value, psi, jsdiv, unseen_share, drifted, note), added, removed, nref, ncur, alpha and anydrft.

Details

Errors: dir not an existing directory; no provenance record, or one whose resource$filename is missing or not a relative path inside dir; a pinned or current file that cannot be read as a CSV; current neither a data frame nor one path; and the refusals of cpsdrft() (alpha outside (0, 1), psi_min_n negative, id_levels below 2).

See also

Other capsule: morie_capsule_bundle(), morie_capsule_compare(), morie_capsule_manifest(), morie_capsule_sign(), morie_capsule_verify(), morie_capsule_write()

Examples

dir <- file.path(tempdir(), "capsule-drift")
dir.create(dir, showWarnings = FALSE)
set.seed(1)
write.csv(data.frame(x = stats::rnorm(200)), file.path(dir, "data.csv"),
          row.names = FALSE)
morie_capsule_write(dir, "data.csv", retrieved = "2026-10-20")
morie_capsule_drift(dir, data.frame(x = stats::rnorm(200)))$anydrft
#> [1] FALSE
morie_capsule_drift(dir, data.frame(x = stats::rnorm(200, mean = 2)))$anydrft
#> [1] TRUE
unlink(dir, recursive = TRUE)