Rebuild the Ontario SIU DRID manifest by probing the live site

Sweeps director's-report ids 1..max_drid and writes a small CSV recording which ids return a healthy report page, the parsed case number, and the response body size. The harvester (morie_fetch_siu) then uses this manifest to short-circuit the ~30-50 percent of ids that have no report, saving bandwidth and WAF-trigger risk on every run.

Usage

morie_siu_refresh_manifest(
  out_path = NULL,
  max_drid = NULL,
  min_drid = 1L,
  concurrency = 4L,
  rate_rps = 4,
  progress = TRUE
)

Arguments

out_path

Path to write the gzipped CSV. Default is the in-place manifest location (only useful for maintainers building from a source checkout).

max_drid

Highest drid to probe. Default NULL auto-discovers from the SIU index endpoint and adds a margin.

min_drid

Lowest drid to probe (default 1L).

concurrency

Maximum simultaneous transfers (default 4).

rate_rps

Maximum request starts per second (default 4).

progress

Logical; print a per-batch progress line.

Value

Invisibly, a data frame of the full sweep (every probed drid, including misses), parallel to what was written to out_path.

Details

The shipped manifest at inst/extdata/siu_drid_manifest.csv.gz is a snapshot. Users who want the latest can call this function; it is also how morie maintainers regenerate the snapshot.

Examples

# \donttest{
# Bounded probe of 11 report ids (seconds at the polite default
# rate); a full refresh omits min_drid/max_drid and takes ~25-40 min.
df <- morie_siu_refresh_manifest(
  out_path = tempfile(fileext = ".csv.gz"),
  min_drid = 5150, max_drid = 5160
)
#> SIU manifest: probing 11 drids [5150..5160] at 4 req/s ...
#> SIU manifest: wrote 11 rows (11 healthy 200s) to /tmp/RtmpOKyZlD/file2e26458248f8.csv.gz
table(df$http_code)
#> 
#> 200 
#>  11 
# }