Curated datasets at data.rmorie.com

The MORIE project keeps databases materialised from Google BigQuery public datasets (Chicago crime, EPA air quality, US census, FEC, FDA, NOAA, NHTSA, Hacker News, Ethereum, World Bank, ...) and serves their tables from the edge, under the data access terms at www.rmorie.com/data-license/. They open with the key dlogin stores, the same key rmorie and morie use; keys are personal: sign in with GitHub or an emailed code, or ask at www.rmorie.com/access/. The service address comes from svcs. dmnf() returns the gateway's manifest (every table with rows, columns, size, SHA-256 and its BigQuery source), kept for a day; dtabs() the same as a data frame; dload("db/table") one table as a data frame, cached under tempdir() (or the directory in options(rmoriebricklayer.dcache = )) so later calls are local. From the shell: rmoriebricklayer data list and rmoriebricklayer data pull db/table [--out FILE.csv].

Usage

dmnf(refresh = FALSE)

dtabs(refresh = FALSE)

dload(key, refresh = FALSE, token = NULL)

Arguments

refresh

Logical scalar, default FALSE: fetch again even when a day-old copy (of the manifest, or for dload() of the table) is cached.

key

Character scalar, a db/table key from the manifest (the key column of dtabs()).

token

For dload(): the MORIE key to use for this call, in place of the one dlogin stored; NULL (the default) uses the stored key. It is never printed or written to disk.

Value

dmnf(): the manifest as a list (element datasets, one list per table with at least key, and as the service provides them rows, columns, sizes, the SHA-256 of the compressed file and the BigQuery source); dtabs(): a data frame with key, name (the description, else the source, else the key), integer rows and source (class dtabs, which prints one left-aligned line per table, cut to the console width); dload(): the table as a data frame.

Details

The manifest is cached as manifest.json in the cache folder and fetched again when it is more than a day old or refresh = TRUE. A reply that is not a manifest (not a JSON object with a list of datasets each carrying a key), or one that lists no tables while the cached copy lists some, is refused and the cached copy kept.

dload() downloads db/table.csv.gz once into the cache folder (as db__table.csv.gz) and checks its SHA-256 against the manifest before every use, cached or not: a file that does not match is deleted and refused (after one refresh of the manifest, since a table rebuilt upstream has a new digest). When the manifest cannot be fetched, the last copy on disk, however old, supplies the digest; with none, a fresh download is not kept and a cached file is kept but not used. The CSV is read in compiled code with the same result as read.csv(gzfile(f), stringsAsFactors = FALSE), two malformed shapes apart, which are refused with the record they are in where read.csv() changes the data without a word: a record with more fields than the header (read.csv() turns the first column into row names) and a quote mark inside an unquoted field (read.csv() takes the rest of the file as one quoted run). Tables that inflate past 512 MiB are read by read.csv() itself, a piece at a time. A table whose SHA-256 matches the manifest is the file the service holds, so it is read whatever its size; one the manifest lists without a digest may decompress to at most the option rmoriebricklayer.maxcsvbytes (1 GiB by default). A large table needs memory for all of it: a 1 GB decompressed CSV takes a few GB of RAM as a data frame.

The key is sent only to the address of the signed services document. With MORIE_DATA_URL set to another address the key is not sent and the download fails with a message saying so. Some tables (ICPSR studies, the NCVS 38604 extract, the A2AJ case law) open only after their terms are accepted on the terms page, data.rmorie.com/terms. Downloads draw a progress bar when options(morie.progress) allows it.

Errors: refresh not TRUE or FALSE; no key stored (the message says how to sign in); the service unavailable, or the key rejected; a key that is not of the form db/table, or not listed in the manifest; a token that is not one non-empty string; a digest mismatch; a restricted table whose terms were not accepted.

See also

dlogin, svcs.

Examples

# Offline, against a cache folder holding a manifest and one table made
# here (no key is stored and nothing is fetched).
oldenv <- Sys.getenv(c("XDG_CONFIG_HOME", "MORIE_HOSTED_KEY"), unset = NA)
Sys.setenv(XDG_CONFIG_HOME = tempfile())
Sys.unsetenv("MORIE_HOSTED_KEY")
cache <- file.path(tempdir(), "data-hub-example")
dir.create(cache)
old <- options(rmoriebricklayer.dcache = cache)
f <- file.path(cache, "demo__counts.csv.gz")
con <- gzfile(f, "w")
write.csv(
  data.frame(region = c("a", "b"), n = c(3L, 5L)),
  con,
  row.names = FALSE
)
close(con)
m <- list(
  datasets = list(list(
    key = "demo/counts",
    rows = 2,
    source = "an example",
    meta = list(description = "Counts by region"),
    sha256_gz = csha256(readBin(f, "raw", file.size(f)))
  ))
)
writeLines(tojs(m, auto_unbox = TRUE), file.path(cache, "manifest.json"))

dtabs()
#> ╭── Curated tables at data.rmorie.com ──╮
#> │   Tables  1                           │
#> │                                       │
#> │   key          rows  name             │
#> │   demo/counts     2  Counts by region │
#> │                                       │
#> │   1 tables; dload("KEY") loads one    │
#> ╰───────────────────────────────────────╯
dload("demo/counts")
#>   region n
#> 1      a 3
#> 2      b 5

# keys are checked against the manifest
try(dload("counts"))
#> Error : key must be db/table (see dtabs())
try(dload("demo/other"))
#> Error : no table demo/other at data.rmorie.com (`rmoriebricklayer data list` shows them)

# a cached file that no longer matches its digest is deleted and refused
writeBin(as.raw(1:10), f)
try(dload("demo/counts"))
#> Error : demo/counts does not match the manifest's SHA-256 (c848e1013f9f... vs c346413f308c...): the cached file was deleted; run again to download it, and if it still fails the file was tampered with in transit
file.exists(f)
#> [1] FALSE

options(old)
unlink(cache, recursive = TRUE)
for (v in names(oldenv)) {
  if (is.na(oldenv[[v]])) {
    Sys.unsetenv(v)
  } else {
    do.call(Sys.setenv, as.list(oldenv[v]))
  }
}
if (FALSE) { # interactive()
# the real service: needs a key and the network
dlogin(token = "sk-...")  # a key issued at www.rmorie.com/access/
head(dtabs())
df <- dload("chicago_crime/incidents")
}