Changelog • rmoriedata

rmoriedata 0.4.3

hostload() returns a table as its file holds it. It read the file with read.csv()’s defaults, so codes lost their leading zeros (046 became the number 46), column names were rewritten (Offence Description became Offence.Description) and the text NA became a missing value. Now the names are kept as written, only an empty field is missing, and a column becomes logical, integer or double only when every value in it is written the way R writes such a value; any other column stays text. Text is marked UTF-8 in every locale (it was unmarked in a non-UTF-8 locale), and bytes that are not UTF-8 are marked "bytes". The help page says exactly which values count as numbers.
A table already cached and verified loads offline once the cached manifest is more than a day old: when the manifest cannot be fetched again, the cached copy is used with a message, and the table is still checked against the digest it lists. hostload() failed with a network error.
hostload() checks a cached table under the same lock as a download, so it can no longer delete a copy that another R session has just refreshed. A session waiting for another session’s download stops with an error once that download has not grown for options(rmoriedata.lockwait = ) seconds (600 by default); it waited for ever behind a stalled session.
A failed download from data.rmorie.com names the function called (hostcat() or hostload()), the table and the address, without the dependency’s own function name in front (bldl:).
A privacy budget copied within one R session (written with saveRDS() and read back, passed through serialize(), or rebuilt from its fields) is refused like a budget from another session: the copy and the original could each spend the whole budget. dpbudget()’s help says what is and is not prevented.
The Parquet reader refuses a string that is not valid UTF-8, or that holds a NUL byte, with a parquet: error naming the column: it marked such bytes as UTF-8, and nchar() or toupper() failed later. columns = character(0) gives the file’s rows with no columns (it failed with “footer says 3 rows, decoded 0”), and a damaged footer length is reported as damage, not as a file over the row cap.
The native Parquet reader is about 90 times faster and the writer about 8 times (on a 400,000-row table of 13 columns, 3,500 and 4,400 rows a second became 326,000 and 35,000): string values are read and written in compiled code, and the other per-value loops work on whole vectors. The bytes written and the values read are unchanged.
chiload(full = TRUE) reads text as decimal numbers only, so "0x1A" is no longer 26 and "Inf" or "1e400" no longer an infinite coordinate. A value in Socrata’s export that does not fit its column becomes NA with one warning per column, instead of refusing the whole download; a mirror’s or the cache’s file in the sample’s columns must still fit exactly, and the error names that file (it blamed Socrata’s export).
mdask() no longer probes or asks an Ollama server the user did not choose: with the automatic route, a local server at localhost:11434 is left out unless route = "ollama", an OLLAMA_* variable or a saved Ollama setting names it.
cihiget() checks timeout under its own name and takes 1 to 86400 seconds (0.5 was refused as “tmout must be a positive number”), and a failed download names cihiget(), not the dependency’s blfetch(); strings marked "bytes" passed to kanonvrf(), ldivvrf(), mdload(), mdpath(), mddict(), cihiget(), hostload() and mdask() are refused in words, where base R stopped with “translating strings with”bytes” encoding is not allowed”.
mdload()’s help says how the Parquet and CSV copies differ: in five tables one or two doubles of the Parquet copy are the next double to the CSV’s value.
kanonvrf() and ldivvrf() compare every numeric column exactly, whatever its class: difftime, I(), Date, POSIXct, POSIXlt and complex columns were compared to 15 significant digits, so two distinct values (such as two times a microsecond apart) formed one class and a violated k could be reported as satisfied. A fractional Date is now compared by its stored value, not its printed day. Columns that cannot be compared exactly (bit64::integer64, list and matrix columns) are refused with an error naming them.
kanonvrf() and ldivvrf() build classes column by column, so strings holding control characters can no longer join two different rows into one class, and a factor level that is NA is missing while the string "NA" is a value (they were one class). As a sensitive value, an NA factor level is withheld and no longer counts towards diversity.
A privacy budget of delta = 0 refuses every release with delta > 0: a release at delta = 5e-13 was accepted. The check of the totals allows only floating-point rounding, relative to the budget, and never a release more than twice what remains: a budget of epsilon = 1e-13 paid for a release at 1e-12.
A privacy budget saved and read back in another R session is refused even when that session has the same process ID, as every Rscript started as a container’s entrypoint does: each budget carries a random token drawn from the operating system when the package loads. Budgets saved by earlier versions are refused too. dpbudget()’s help says that a release interrupted after it is charged stays charged.
chiload(full = TRUE) reads a text time with an offset or Z (from a mirror) as that instant: 2024-06-01T10:00:00Z was read as 10:00 in Chicago, five hours off. A value that does not fit its column’s type is refused with an error naming the column, from a mirror and from Socrata’s export alike: text "1.5" became the integer 1, 2 became TRUE, and junk in Socrata’s numeric columns became NA.
The Zstandard decoder refuses a block whose sequence count of 0 is followed by more bytes (libzstd does too), and reports one to three stray bytes after the last frame as trailing data rather than as a truncated frame. The zstd, gzip and Snappy decoders refuse, before allocating, a size that this build of R cannot hold.
A hosted table downloaded in parts checks that every part is there and readable before it joins any: a missing or unreadable part stopped with a base-R warning after the earlier parts had been deleted; now all parts are kept and the next run downloads only what is missing. Two sessions joining the same table at once each write a file of their own instead of sharing one.
R sessions downloading the same hosted table at once take turns on an operating-system lock that the system releases when its holder exits or is killed, so it is never left stale: one session downloads, checks the table against the manifest’s SHA-256 and only then moves it into place in one step, and the others wait and use it. The sessions shared one .parts folder and could delete or overwrite each other’s parts, and an unchecked table sat under its final name until its checksum was read. A download that is cut short or killed still resumes from the parts it finished. The manifest is written under a name of its own and renamed, so sessions fetching it at once no longer write into one file.
The Parquet writer refuses matrix, data frame and integer64 columns, naming them: a matrix column was written with more values than rows, which other readers cut short. Dates and times are written down to the whole day and millisecond as R prints them, so a Date of -0.5 (printed 1969-12-31) no longer reads back as 1970-01-01, and dates or times outside what Parquet holds, or infinite, are refused.
The Parquet reader reads ZSTD-compressed pages, such as those of files written with compression = "zstd" by nanoparquet, arrow or DuckDB, with no other package: the Zstandard decoder is in the package, written from RFC 8878 (no dictionaries), and checks every page against its stated size, and against the frame’s checksum when it has one. Such a file stopped with “compression codec 6 not implemented”. The writer still writes GZIP, Snappy or no compression.
A hosted table downloaded in parts no longer comes out damaged when the disk fills up while the parts are joined: the failed write is an error that names the folder and options(rmoriedata.cache_dir = ), where it was a short file that then failed its checksum. Each part is deleted once it is joined, so the download needs the table’s size plus one part on disk, not twice the table’s size.
The title, description and citation put the package’s disclosure-control tools first (differential privacy, k-anonymity and l-diversity checks, cell suppression); the datasets follow. Nothing in the functions changes.
chiload(full = TRUE) returns the bundled sample’s columns, in its order and with its types (16 columns for complaints and 8 for arrests, with date_iso and a POSIXct date), derived from Socrata’s export the way the samples were: it returned Socrata’s raw 22 or 24 columns. Socrata’s export is read as text first, so an IUCR code keeps its leading zero.
chiload(mirror = ) with a URL downloads <type>_full.parquet from the mirror over the package’s checked https transport, and checks it against <type>_full.parquet.sha256 when the mirror has one; a mirror that fails or does not match falls back to Socrata. A URL was taken for a missing local file, so the mirror was never used.
chiload() checks its arguments in words: limit must be a whole number from 1 to 2147483647 (Inf and 1e10 failed with a coercion error), fraction a number above 0 and at most 1, and full and refresh TRUE or FALSE (full = "yes" or NA silently returned the sample).
cellsupp() checks which hidden cells the totals determine with one factorisation for all of them, not one per cell: a 50x50 table took 152 seconds and a 60x60 table did not finish in 17 minutes; both now take well under a second, with the same cells suppressed. Negative counts are refused, and the help says that zero cells are published as they are.
mdload(slug, refresh = TRUE) checks the file against the signed manifest again: a bundled file changed after its first load in a session was still returned.
A cached hosted/manifest.json that does not parse is fetched again, once, with a message; it blocked hostcat(), hostload() and hostload(refresh = TRUE) until it was a day old.
hostload() takes a large table whole when the server answers a byte-range request with the whole file, instead of failing on every run, and a failed run leaves no empty .parts directory.
A cache directory that cannot be written (read-only, or a file in its place) is an error naming that directory; it was reported as a failure of data.rmorie.com.
dplaphist() keeps the names and dimensions of its counts, so a table()’s labels stay on the release.
The Parquet reader reports any damaged file with a parquet: message: about 4% of corrupt files stopped with a base-R message such as “subscript out of bounds”, and a footer of 5000 nested lists overflowed the stack. Its help, and chiload()’s, say that a POSIXct column reads back in UTC (the same instant): Parquet keeps no time zone.
Extreme arguments are refused in words: k or l of Inf or above the integer range in kanonvrf() and ldivvrf(), an epsilon above 2^20 in the counting mechanisms (both stopped with “missing value where TRUE/FALSE needed”), a slug that is not one name in mddict(). mdsha() takes one string (it hashed only the first of several) and hashes it as UTF-8 whatever encoding it is marked with.
dplapcnt(), dplaphist() and dpgmean() draw fresh random bytes in a forked worker (parallel::mclapply()): the workers inherited the parent’s buffer of random bytes and released identical noise, so the difference of two releases was the exact true difference.
chiclr() deletes only what chiload() wrote (the <type>_full.parquet files, their checksum files and unfinished writes): it removed everything in the cache directory, subfolders and the hosted-table cache included.
cellsupp() refuses a table of three or more dimensions (it was flattened to one column and suppressed against the wrong margins). ldivvrf() counts a class’s values in one pass: 200,000 classes took minutes.
A refused download reads the status the server answered: a table whose name holds 401 or 403 no longer turns a 404 into “need no sign-in”, and a 410 names ICPSR only for an ICPSR table.
Needs rmoriebricklayer 1.1.1 or later.
hostlogin() is gone, and no key is read or sent: every table at data.rmorie.com is open, so the sign-in asked for a name, a reason and an email or GitHub account for nothing. hostcat() and hostload() work as before. A key that an earlier version stored stays in its file, unread. The manifest and services fields are read by exact name.
hostload() checks every table against the manifest’s SHA-256 again, whatever the field is called (sha256_gz, sha256 or sha, as rmoriebricklayer reads it): 0.4.2 read sha256 only, so 631 tables listed with sha256_gz were used unchecked and a modified cached copy loaded. A digest field that is not a SHA-256 is an error, and a table listed without any digest loads with a warning, once per session.
Tables whose source terms do not allow redistribution (the ICPSR studies, the NCVS 38604 extract among them, and any table the manifest marks restricted) are left out of hostcat(), with a message counting them, and hostload() refuses them before any request. The A2AJ case law is an ordinary open table.
The messages for a refused download no longer suggest signing in: the tables need no key.
dplapcnt(), dplaphist(), dpgmean() and dpbudget() refuse an infinite epsilon in words; dplapcnt(5, Inf) stopped with “missing value where TRUE/FALSE needed”.
The signing test writes its XMSS used-index record to a temporary file, not to ~/.config/R/rmoriebricklayer.
The vignette’s reference to the integrity checks is no longer empty.
A privacy budget from dpbudget() cannot be spent from a forked worker or another R process: dplapcnt(), dplaphist() and dpgmean() refuse it there with an error. Four workers under parallel::mclapply() each spent 0.9 from one budget of 1 and the parent recorded nothing; give each worker its own budget and add up the epsilons.
kanonvrf() and ldivvrf() compare numeric quasi-identifiers exactly: values that agree to 15 significant digits, such as 0.1 + 0.2 and 0.3, were one class, so a class of one could be reported as a class of five. -0 and 0 are one class, and NaN is missing with NA.
cellsupp() treats an NA cell as unpublished in its audit, as its help says, and gives it complements like any hidden cell: it was treated as known, so a hidden cell recoverable through it was reported as protected. Infinite and fractional counts are refused, and a function or an environment gets the package’s “must be a numeric matrix or table” error.
chiclr() deletes only arrests_full.parquet, complaints_full.parquet, unfinished write-*.parquet files and their .sha256 files: it deleted any file ending in _full.parquet and left the .sha256 of an unfinished write.
chiload(full = TRUE, mirror = ) gives a mirror’s table the bundled sample’s types when its columns have the sample’s names (a date stored as text came back as text), and refuses a column that cannot be read as the sample’s type.
hostload() and mdload() check refresh in words (NA, "yes", NULL and c(TRUE, FALSE) gave base-R errors or acted as TRUE), and a table size in the hosted manifest that is not a whole number from 1 to 2^40 bytes is refused in words: an absurd size stopped in seq().
mdverify() and mdsums() list hidden files in the store too, and the help of mdverify() says what it covers: every file of the data store, not the package code or the lazy-loaded complaint_sample and arrest_sample. mdsums(), siuload() and cihitbls() read the store named by options(rmoriedata.store = ), the one they verify; they used to read the installed copy.
dplapcnt() and dplaphist() refuse a count of 2^53 itself: 2^53 + 1 rounds back to 2^53, so the release could equal the truth.
The Parquet writer keeps NaN (it read back as NA), refuses a data frame with no columns, which its reader could not open, and refuses list, complex, raw and difftime columns in words, naming the column: they were written as text. Every error of the reader starts with parquet:, and a damaged file no longer lets a base-R warning such as “NAs introduced by coercion to integer range” escape before the error.
The help pages, README, vignette and website describe the package by its own purpose, disclosure control first and the datasets second, and no longer as a companion of another package; the README’s function list and the reference index follow that order. The README names the real source of each table’s attribution, the right path for reading the sample with pyreadr and the right hostcat() columns, and drops an out-of-date note on package size.

rmoriedata 0.4.2

CRAN release: 2026-11-12

Works with rmoriebricklayer 1.1.0 and 1.1.1 (where yoy() is renamed yoychg()): rmoriedata no longer refers to yoy().
hostload() downloads a table larger than twice the part size (option rmoriedata.dpart, 90 MB by default) in byte-range parts, each kept once whole, so a broken connection costs at most one part and running it again resumes; the joined file is checked against the manifest SHA-256.
hostcat() and hostload() need no sign-in; a key stored earlier is still sent. ICPSR studies are no longer served by data.rmorie.com (ICPSR’s terms do not allow redistribution); asking for one says where to get it.
hostlogin()’s help names the arguments rmoriebricklayer::dlogin() takes (affil, purp, restr; it listed affiliation, purpose, restricted). A table refused with 403 is reported as a key asked for or refused: no table sits behind accepted terms any more.

rmoriedata 0.4.1

CRAN release: 2026-10-23

The Snappy and gzip page decoders check the size a file claims before allocating it. A damaged or hostile header asking for more than the compressed bytes can hold (a Snappy stream expands at most 64 bytes per 3, deflate about 1032 to 1, and both sizes are 32-bit) is now an R error up front. Before, R was asked for the full amount first: an ordinary build then failed with an error, but the address-sanitizer build CRAN runs (gcc-ASAN) aborted the test run.
A function missing from the installed rmoriebricklayer is now named in the message that asks for an update, and the test of the old-name fallback no longer depends on which rmoriebricklayer is installed (a development build between the 1.0.0 rename and the return of the old names failed it).
R CMD check on R before 4.6 no longer notes rmoriebricklayer as an unused import.

rmoriedata 0.4.0

CRAN release: 2026-10-21

Short names, without underscores, for every exported function and argument, matching rmoriebricklayer 1.0.0: morie_data_load() is mdload(), morie_dp_laplace_count() is dplapcnt(), load_chicago_data() is chiload(), quasi_identifiers is qids, and so on. Each help page gives the full form of its short name. The old names are not kept.
Works with rmoriebricklayer 0.5.1 or later and calls its 1.0.0 names when they are there. Signing in (hostlogin()), mdask() and the downloads that carry the data key need rmoriebricklayer 1.0.0; with an older one they say so, and attaching rmoriedata prints a one-line note on how to update.
The citation gives the CRAN DOI and the r-universe page, and the author’s given names in full (“Ruhela VS”).

rmoriedata 0.3.7

Needs rmoriebricklayer 0.5.14: a new data key from morie_data_hosted_login() needs the data access terms accepted first, then your name, organisation and why you need the data (and, for the restricted tables, a supervisor, department chair, director or coordinator). Pass them as arguments (accept_terms = TRUE, name =, purpose =, …) or answer at the console.

rmoriedata 0.3.6

Needs rmoriebricklayer 0.5.13, whose download bar now redraws in the RStudio, Positron and R.app consoles and stays a plain line in knitr and Jupyter.
Needs rmoriebricklayer 0.5.12, whose language-model layer ask() is built on (0.5.9 was accepted, so a build against an older one ran ask() on the retired hosted tier).
ask() uses a language model you bring (a provider’s key, Ollama, or your own server) through rmoriebricklayer, and the model is given rmoriedata’s and rmoriebricklayer’s help with tools to read it. Its backend argument is now route ("auto", "own", "ollama" or a provider id such as "anthropic"); the hosted MORIE tier and the rmorie command-line fallback are gone.
morie_data_hosted_load() uses a downloaded table only when it matches the SHA-256 the manifest lists (fetching the manifest again once before calling a file bad); a mismatch deletes the copy and says so. A restricted table (the ICPSR studies, NCVS 38604, the a2aj case law) refused with 403 says to accept the terms at https://data.rmorie.com/terms.
morie_data_llm_config() shows and saves the language-model settings (route, and the address, key and model of each route), shared with rmoriebricklayer, rmorie and morie.

rmoriedata 0.3.5

A hex logo (man/figures/logo.png, source data-raw/hex_logo.svg), on the README and the package website, with its favicons; the website indexes morie_dp_budget() and morie_dp_spent().
A garbled download is refused, not padded. The full Chicago export and the tables from data.rmorie.com are read with fill = FALSE: a row with a field missing or one too many was padded into the result (and, for Chicago, written to the cross-session cache); it is now an error, and a damaged hosted table’s local copy is removed so the next call fetches it again. The Chicago tests also pin the 1% row-count tolerance at its edge (495 of 500 accepted, 494 refused) and the request size (the service’s count, at least 5,000,000 rows).
Privacy verifiers (review of 0.3.5). morie_k_anonymity_verify() treats a missing quasi-identifier as a level of its own: stats::aggregate() had dropped every row with an NA key, so a class of one person could pass k = 3; class sizes now always sum to nrow(data). morie_l_diversity_verify() counts only known sensitive values ("HIV", NA, NA is diversity 1, not 2). morie_cell_suppress() adds complementary suppressions to a fixed point and then audits the result exactly: a hidden cell the row and column totals determine (its indicator in the row space of the margin constraints) gets another complement, and the result carries recoverable_mask, n_recoverable and a protected verdict to test before publishing. The 0.3.5 passes left a lone hidden cell in a row or column, from which the primary value followed by two subtractions; zero cells are no longer used as complements. Checked against brute-force enumeration of every margin-consistent fill on 464 random tables: no disagreement.
Differential privacy. Noise comes from the operating system’s CSPRNG (rmoriebricklayer::random_bytes()), not R’s seeded generator, so set.seed() no longer reproduces a release. Counts and histograms use the exact discrete Laplace sampler of Canonne, Kamath and Steinke (2020) in integer arithmetic (whole-number releases; no floating-point artefact of the kind Mironov (2012) showed in the textbook Laplace sampler). The mean uses the analytic Gaussian mechanism of Balle and Wang (2018), valid for every epsilon; the classical bound, valid only for epsilon <= 1, is gone. All three mechanisms use one neighbouring relation (add or remove one record): the mean is a noised sum over a noised count. New morie_dp_budget() / morie_dp_spent() charge releases against a budget (basic composition) and refuse one that would exceed it. Examples no longer call set.seed().
Parquet reader. Every byte access is bounds-checked (an out-of-range raw index had silently read zeros); varints, lists, maps and nesting are capped by the bytes that remain, so the 21-byte file that looped forever is refused at once; the decoded row count must equal the row group’s and the footer’s exactly (no padding with NA, no truncation); a column chunk’s type must match the schema’s; PLAIN values must fill their page exactly; dictionary indices are range-checked; page sizes and footer row counts are capped (options(rmoriedata.parquet_max_page_bytes, rmoriedata.parquet_max_rows)); PageHeader.crc is verified when present and the writer now writes it. tests/testthat/test-parquet-hostile.R fuzzes all three codecs.
Downloads. .rmd_dl(), which carries the MORIE key as a bearer header, goes through rmoriebricklayer::bricklayer_download() (https to public hosts only, no file://, size cap) instead of base url(); requires rmoriebricklayer 0.5.9. load_cihi_data_tables() checks the catalogue against the signed manifest before reading it. morie_data_verify() reports files added to the store (listed = FALSE), and morie_data_checksums() compares each digest with the manifest (in_manifest).
Workflows. The SIU manifest refresh runs only on demand and uploads an artifact instead of committing: the manifest is covered by the XMSS-signed _checksums.csv, whose key never leaves the maintainer’s machine, so it is re-signed locally with data-raw/sign_store.R. A new ASAN/UBSAN job builds the C and runs the suite, including test-codec-hostile.R.
Packaging. utils is declared in Imports; the stale RoxygenNote is dropped (Config/roxygen2/version is the roxygen 8 field); NOTICE moves to inst/NOTICE so it installs, and its data attribution now names the publishers instead of per-file headers that did not exist. The licence text itself stays out of the tarball, as CRAN asks for a standard licence. DESCRIPTION describes the disclosure-control helpers by what they do.
The curated-data service address comes from the signed services document at rmorie.com (through rmoriebricklayer::bricklayer_services() when the installed bricklayer has it; an older bricklayer keeps the default address), so the endpoint can move or be paused without a release. Keys are personal and issued on request at www.rmorie.com/access/; every hint says so, and morie_data_hosted_login(token = ) stores one (the GitHub and emailed-code sign-ins keep working).
ask() follows rmoriebricklayer::bricklayer_llm_ask()’s route order: an endpoint of your own, a local Ollama server, then the hosted MORIE tier as a last resort; backend = "hosted" insists on the hosted tier. The setup hint when nothing answers names all three.

rmoriedata 0.3.4

The Parquet store is compressed: the writer’s Snappy stream was literal-only (valid, but it compressed nothing), so every table’s Parquet copy was as large as its raw pages. Pages are now written with GZIP (zlib, level 9) by default, and Snappy, still available, has a real match-finder; both codecs and the reader are C (src/gzip.c, src/snappy.c). The Parquet copies shrink from 16.1 MB to 3.3 MB (the SIU corpus from 10.0 MB to 2.1 MB) and the installed package from about 27 MB to 14.5 MB; pandas/pyarrow read them as before.
SIU corpus: narrative_summary holds each report’s own account of the incident on 4,605 of 4,613 reports (it was empty but for three values that described the SIU’s investigation). The text is the opening of the report’s “Incident Narrative” / “Description de l’incident” section, or of the “Notification of the SIU” / “Avis à l’UES” section in reports that have no narrative section (2005-2016 and re-opened files), about 900 characters cut at a sentence end. It was read from the SIU’s pages, re-crawled on 2026-10-05 at one request every 3 seconds; data-raw/add_siu_narratives.R rebuilds it.
SIU corpus, round 8 (4,613 rows; every change is a row of data-raw/siu_round8_review.csv, with the reason read from the page, or a rule of data-raw/siu_round8_rules.R):
- six French reports of 2005-2011 filed under /en/ are labelled French and carry the service their page names; three of them held “Guelph Police Service”, which the pages never name (05-TCD-004 is Toronto, 10-PFD-078 the OPP, 10-OOD-009 Ottawa);
- a corrigendum is applied to the report it corrects, in a new column corrigenda, not counted as a report (drids 385, 620/621 and 5065 correct 328, 521/522 and 4561); six second copies of one report (identical page text) are removed;
- case numbers the page body gives: drids 925/926 are 19-TCI-073a, and drid 2209 is the French report of 22-PCI-191 (the OPP in Quinte West), not of 21-OCI-191 as the SIU page’s header says;
- a French report carries its English report’s reviewed case facts (dates, team, counts, the affected person’s age and sex, the charges, the director’s view): they were empty on about 2,240 of 2,304 French rows; the French text fields stay French;
- charges_recommended is "FALSE", "TRUE" or empty (it had 39 spellings), and logical in the typed table (morie_data_load("siu_directors_reports"), CSV and Parquet), as panel_reviewed is; sex_gender_affected is male or female; one name for Kawartha Lakes and for Cornwall (the Cornwall Community Police Service is now the Cornwall Police Service);
- narrative_summary held the SIU’s mandate paragraph or the page title, not a narrative, and is empty except on the three rows that held report text; supplemental_materials lists the legislation and case law a report cites (it held the page’s share and privacy links).
ask() sends the bundled catalogue (slug, kind, rows) with the question, so the model names tables (it was told only to “prefer the bundled catalog”, which it could not see). Through the rmorie binary it calls rmorie agent [--model NAME] QUESTION, the agent’s own options: it passed --backend, which the agent refuses, and -m for the model; backend is "auto", "hosted" or "cli". The examples that need a stored key or the binary are \dontrun; the one that runs switches the hosted tier off (MORIE_HOSTED_BASE_URL = "off"), so it reaches no network whatever key the machine holds.
morie_dp_laplace_count() and morie_dp_laplace_histogram() refuse Inf in words (they warned about integer coercion, then failed in R’s internals) and refuse counts above 2^53: a double that large has no room for Laplace noise of scale 1/epsilon, so morie_dp_laplace_count(1e300, 1) returned exactly 1e300, a “private” release equal to the true count. A whole count beyond R’s integer range but below 2^53 is still a count. morie_core_mean() refuses text instead of averaging NA.
Help: ?rmoriedata describes the package’s functions (it said there were none); ?load_siu_reports lost “% of them” to an Rd escape; “more than 160” databases at data.rmorie.com; a garbled sentence in ?load_chicago_data; the README’s Chicago example is bounded (limit = 200).
The SIU director’s-report corpus: police_service is the service of the subject officials on every row (it was the notifying force on many French rows, and on 24 English rows a custody, requesting or neighbouring service – each corrected after reading the report); one spelling per service; 59 English reports published since the last build, reviewed by the same model panel, and their 60 French reports added; 54 French reports filed under /en/ relabelled; 29 rows that held no director’s report removed.
ask() answers through the hosted MORIE tier with the key morie_data_hosted_login() stores (ollama.com cloud models and additional AI models such as kimi-k2.6:cf or gpt-oss-120b:cf), and only falls back to the optional rmorie command-line agent when no key is stored.
Hosted tables download with a live progress bar (percent, size, rate) in an interactive session; options(morie.quiet = TRUE) silences it.
Requires rmoriebricklayer 0.5.2 or newer (the login flow).
morie_data_hosted_login() signs in to the hosted MORIE tier (GitHub device flow, email code, or a key you hold) and stores the key in the credentials file every MORIE package reads, so rmoriedata alone is enough to use data.rmorie.com.
morie_data_hosted_catalog() and morie_data_hosted_load("db/table"): the 160 databases the MORIE project materialises from BigQuery public datasets, served from the edge at data.rmorie.com and opened by the MORIE key that rmorie, morie or rmoriebricklayer store. Cached under the package cache directory (tempdir() unless options(rmoriedata.cache_dir = )).

rmoriedata 0.3.3

CRAN release: 2026-09-24

Nothing is written outside tempdir() unless the user asks. The Chicago full-dataset cache lives under tempdir() by default and moves to a persistent directory only through options(rmoriedata.cache_dir = ); clear_chicago_cache() empties it. load_chicago_data(as = "parquet_path") also writes under tempdir(): it wrote the bundled sample into the user’s R_user_dir() cache, and a bounded full = TRUE fetch could land on the full-dataset cache path. The schema names the language column of siu_directors_reports and siu_drid_manifest _language, as the files do; it carried the X_language that check.names had made of it, so the typed store and load_siu_reports() disagreed on the name. load_siu_reports() documents that it returns the corpus as text ("" for empty cells) while morie_data_load("siu_directors_reports") returns the typed frame.
Every bundled table ships as Parquet again, next to its CSV (inst/extdata/parquet/<slug>.parquet, plus _catalog.parquet and siu_directors_reports_corpus.parquet). morie_data_load(format = "parquet") and load_siu_reports(format = "parquet") read those copies; the new morie_data_path(slug, format) returns the verified path of either copy, which is the bridge to pandas.read_parquet(). The catalog gains a parquet_path column and the signed manifest covers the new files.
The Parquet writer no longer mangles non-ASCII text in a C locale (enc2utf8() on an unmarked string turned each byte into its <c3><a9> display form, and load_chicago_data(full = TRUE) wrote that into the cross-session cache). load_siu_reports() reads its CSV as UTF-8 like the store does, so both copies agree in every locale; a data page v2 file is reported as such before any decompression; and CI runs the check in a C locale. Every CSV reader in the package now declares UTF-8, so a live Chicago fetch and its cached copy are identical() in any locale. A string that is not valid UTF-8 and carries no encoding mark is refused by column name (a Parquet string is UTF-8; escaping lost the data, passing the bytes through made a file no other reader opens), and duplicate column names are refused on write and kept apart on read. The cross-session Chicago cache carries a SHA-256 sidecar checked on every read. load_cihi_data_tables() rejects an archived_only that is not TRUE or FALSE instead of ignoring it.
morie_data_checksums() gains a path column relative to the extdata root; the bare file basenames it returned were not unique and 20 of them did not resolve from the root (morie_data_verify() already returned path).
ask() quotes every argument it hands to the rmorie CLI. The preamble contains a parenthesis, so with the CLI installed every call had died in the shell before the binary ran; a ; in the question would have run as a command. model and backend are validated, and a blank or NA question is rejected like "" and NULL. A stub binary on PATH now exercises the branch the tests never reached.
morie_data_load() on a dictionary slug says to use morie_data_dictionary() instead of pointing back at the catalog.
load_chicago_data(full = TRUE) no longer caches a response that is not the dataset. A 200 carrying an outage page, a header-only body or an export shorter than the row count the service reports is refused with the reason; before, an outage was written to the cross-session cache and every later session read the empty frame back. The cache is written to a temporary file and renamed, so a concurrent reader never sees a half-written file; a cache that cannot be read is discarded and refetched; and a new refresh = TRUE argument refetches on demand. When the service’s count endpoint is down the data is returned but not cached across sessions.
fetch_cihi_table() checks the downloaded file’s size and signature against the catalogue’s declared format and validates which (a whole number within the catalogue, or a non-empty title substring) and timeout before any request. A live URL that answers 200 with an outage page now falls through to the Wayback copy.
The verifier pins the signing key’s XMSS root and public seed in package code. A store re-signed with another key, even with a consistent manifest and shipped public key, no longer verifies; the signature now means more than a checksum against local tampering.
CSV tables are read with encoding = "UTF-8" instead of fileEncoding = "UTF-8-BOM": in a C locale the latter truncated 22 of the 99 tables (5 to zero rows); the bytes are now left alone and marked as UTF-8 in every locale.
The data store is signed. _checksums.csv lists every shipped file with its SHA-256, and the manifest’s own SHA-256 carries an XMSS (RFC 8391, SHA-256) signature whose public key ships as _signing_key.json. The signature is checked once per session and each file when first read, so a modified or corrupted install errors instead of returning altered data; morie_data_verify() checks the whole store at once. Needs rmoriebricklayer 0.4.0 or newer for the verification.
load_siu_reports(format = "parquet") pointed at the parquet copy this release removed; the option now errors with a note that the gzip CSV is the same corpus. rmorie reads the corpus through the default "csv".
morie_dp_gaussian_mean() now rejects infinite lower or upper with the error its message already promised, instead of returning NaN with a warning from rnorm().
The data store ships every table once. morie_data_load() reads the table’s CSV (the file rmorie reads by name) and applies a bundled column schema, so each table comes back with exactly the names and classes the retired parquet copies had (verified identical for all 99 tables), and keeps it in a session cache: repeated loads are instant. The parquet directory, a duplicate samples/ directory and a second copy of the SIU reports are gone, which takes the source tarball from 6.8 MB to under the 5 MB CRAN cap. The ten Victoria tables that existed only as parquet now ship as CSV under vic/. morie_data_catalog() has n_rows and n_cols for every table; morie_data_dictionary() returns the shipped JSON.
Documentation is generated with markdown roxygen; backticks in the help pages are now \code{} (they rendered as stray opening quotes in the PDF manual).

rmoriedata 0.3.2 - 2026-09-08

CRAN release: 2026-09-17

describe_corpus.Rds ships here again

The narrative corpus behind morie_describe() was excluded from the 0.3.1 tarball to get this package under 5 MB, which left rmorie carrying the only shipped copy. rmorie has no room for it either, so the corpus is back where it belongs, and both packages sit in the 5-10 MB range CRAN accepts with a written justification.

rmoriedata 0.3.1 - 2026-09-08

Source tarball down to 4.77 MB, under CRAN’s 5 MB guideline

The package shipped 9.2 MB, and most of the excess was duplication:

rmoriedata.sqlite (15 MB uncompressed) was retired by the SQLite-to-Parquet migration and no shipped code reads it. Kept in the repository, excluded from the tarball.
The SIU director’s-report corpus was bundled three times – as a top-level Parquet file, again inside the Parquet store, and as a .csv.gz. load_siu_reports(format = "parquet") now reads the store copy that morie_data_load() already serves, so the top-level duplicate is gone. The CSV remains: it is the default format and a genuinely different encoding, not a third copy of the same one.
describe_corpus.Rds (1.6 MB, and it does not compress) is referenced by no R code, Rd page or test. Excluded.

Verified after the change: the Parquet path, the CSV path and the siu_directors_reports store slug all return the same 5,157 x 65 corpus.

rmoriedata 0.3.0 - 2026-09-08

Native Parquet codec: one fewer hard dependency

nanoparquet is gone from Imports. The bundled store is now read and written by this package’s own codec (R/aaa_parquet.R), so a plain install no longer pulls a compiled Parquet dependency, and load_siu_reports(format = "parquet") works on a machine that never had one – previously it turned a missing optional package into a hard stop, despite the corpus being bundled in exactly that format.

Victorian crime data

Ten Victorian (Australia) crime tables added to the bundled Parquet store, from the Crime Statistics Agency’s “Latest Victorian crime data” release (year ending March 2026, CC BY 4.0): criminal incidents, recorded offences, victim reports, alleged offender incidents, family incidents, the LGA cuts of each, and the two Indigenous-status breakdowns. Reach them as morie_data_load("vic_<key>"); they appear in morie_data_catalog() like any other slug.
The workbooks are .xlsx and were read with rmorie’s native reader, so the bundled data comes through the same code path a user hits – no readxl/openxlsx dependency and no second parser to disagree with the first. Rebuild with data-raw/build_vic_tables.R.
This closes a real gap rather than adding a convenience: rmorie’s morie_datasets_vic_table() defaults to offline = TRUE, and with nothing bundled it returned a 0-row frame on any machine that had not already downloaded the workbook.

rmoriedata 0.2.5

load_chicago_data(full = TRUE) gains limit (exact row cap) and fraction (share of the live dataset, total looked up first); bounded fetches never touch the full-dataset cache. Network examples are now \donttest{} with try() so they degrade gracefully offline.
SIU corpus also bundled as Parquet; load_siu_reports(format = "parquet") reads it via nanoparquet (columnar / SQL-friendly). CSV remains the default (no extra dependency).

rmoriedata 0.2.4

SIU corpus: bundle the multi-agent panel-REVIEWED director’s reports (subject-official count now 100% for all 2182 English reports; witness-officer-only cases resolved to 0). Adds a panel_reviewed flag column (65 cols).

rmoriedata 0.2.3

CRAN size compliance: removed the legacy inst/extdata/rmoriedata.sqlite (retired by the sqlite-to-parquet migration; no shipped code read it — the loader API is Parquet-only via nanoparquet, and the file remains backed up outside the package). Tarball drops from 5.5 MB to 4.25 MB.
License field is now plain AGPL (>= 3) (the LICENSE file was a verbatim copy of the standard license, which CRAN flags).
CITATION.cff, NOTICE, and LICENSE are excluded from the built package (.Rbuildignore), clearing the top-level-files NOTE.

rmoriedata 0.2.0

Connected to the shared rmorie C core

rmoriedata now declares LinkingTo: rmoriebricklayer (>= 0.2.0) and links the ecosystem’s shared compiled core instead of duplicating any C code.
New exports morie_core_sha256() and morie_core_mean() call the shared kernels directly (fast data-integrity hashing + summaries for the integrated fixtures, with no dependency on rmorie). Tests assert they are byte-identical to rmoriebricklayer’s own core_sha256() / core_mean().

rmoriedata 0.1.1

New exported helpers — differential privacy + re-identification risk

Six small, base-R-only helpers for analysts releasing aggregate statistics from the integrated fixtures (or any other dataset) without re-identification risk:

morie_dp_laplace_count() — (epsilon, 0)-DP count via the Laplace mechanism (sensitivity 1).
morie_dp_gaussian_mean() — approximate (epsilon, delta)-DP mean of a bounded numeric vector via the analytic Gaussian mechanism.
morie_dp_laplace_histogram() — per-bin Laplace noise on a histogram.
morie_k_anonymity_verify() — checks k-anonymity over a set of quasi-identifiers and returns the offending equivalence classes.
morie_l_diversity_verify() — per-class distinct-count check on a sensitive attribute.
morie_cell_suppress() — small-cell suppression with optional complementary suppression so primary-suppressed cells can’t be reconstructed from row/column marginals (StatCan-style).

All helpers use only base R + the existing stats import; no new runtime dependencies. Inputs are validated and edge cases (empty vectors, NAs, out-of-bounds inputs, invalid privacy parameters) throw informative errors.

Tests

tests/testthat/test-dp.R — Monte-Carlo convergence + variance-scaling sanity tests for the DP mechanisms; input-validation coverage.
tests/testthat/test-k-anonymity.R — hand-built fixtures with known equivalence-class structure; complementary-suppression correctness.

rmoriedata 0.1.0

Initial public release. Integrated fixtures only; no exported functions.