MRM dataset fetchers and included samples
Source:vignettes/mrm-dataset-fetchers.Rmd
mrm-dataset-fetchers.RmdOverview
The MRM empirical callables (mrm_otis_*,
mrm_tps_*, mrm_siu_*) operate on three
external data sources. This vignette documents how rmorie makes each of
them accessible:
-
OTIS (Offender Tracking Information System;
published by the Ontario Ministry of the Solicitor General) – public
CKAN release on
data.ontario.ca, reached via the existingmorie_load_dataset()infrastructure. -
TPS (Toronto Police Service) – public ArcGIS Open
Data, reached via
morie_fetch_tps(). -
SIU (Ontario Special Investigations Unit) – public
Director’s Reports.
morie_fetch_siu()writes the panel-reviewed corpus bundled in rmoriedata, or fetches the reports live on request.
Four small reference tables, read by morie_sample(), let
the examples run offline.
Included reference samples
otis_b09 and otis_c11 ship in rmorie and
are the whole published tables. otis_b01 and
tps_assault ship in rmoriedata and are 1,000-row
extracts.
# Included samples
b01 <- morie_sample("otis_b01")
b09 <- morie_sample("otis_b09")
c11 <- morie_sample("otis_c11")
tps <- morie_sample("tps_assault")
# Schema sanity
str(b01, max.level = 1)
#> 'data.frame': 1000 obs. of 18 variables:
#> $ EndFiscalYear : int 2025 2023 2023 2023 2024 2024 2025 2025 2024 2023 ...
#> $ UniqueIndividual_ID : chr "2025-08960-SG" "2023-11152-SG" "2023-08955-SG" "2023-11360-SG" ...
#> $ Gender : chr "Male" "Male" "Female" "Male" ...
#> $ Region_AtTimeOfPlacement : chr "Eastern" "Northern" "Central" "Central" ...
#> $ Region_MostRecentPlacement : chr "Eastern" "Northern" "Central" "Central" ...
#> $ Age_Category : chr "25 to 49" "25 to 49" "25 to 49" "25 to 49" ...
#> $ NumberConsecutiveDays_Segregation : int 4 1 6 1 2 1 1 1 2 3 ...
#> $ SegReason_SecurityOfInstitution_SafetyOfOthers : chr "No" "No" "No" "No" ...
#> $ SegReason_InmateNeedsProtection : chr "No" "Yes" "No" "No" ...
#> $ SegReason_InmateNeedsProtection_Medical : chr "Yes" "No" "No" "No" ...
#> $ SegReason_SecurityOfInstitution_SafetyOfOthers_Medical: chr "No" "No" "Yes" "Yes" ...
#> $ SegReason_Disciplinary_Segregation : chr "No" "No" "No" "No" ...
#> $ SegReason_InmateRefuseSearch_Scan : chr "No" "No" "No" "No" ...
#> $ MentalHealth_Alert : chr "Yes" "No" "Yes" "No" ...
#> $ SuicideRisk_Alert : chr "Yes" "No" "Yes" "No" ...
#> $ SuicideWatch_Alert : chr "Yes" "No" "Yes" "No" ...
#> $ SegReason_Other : chr "No" "" "" "" ...
#> $ Number_Of_Placements : int 1 8 1 1 1 2 2 1 2 1 ...
nrow(b09)
#> [1] 78
nrow(c11)
#> [1] 33
dim(tps)
#> [1] 1000 31OTIS via CKAN (data.ontario.ca)
The OTIS public release is hosted at
data.ontario.ca/dataset/data-on-inmates-in-ontario (31
resources when this vignette was written, 29 of them CSV). The catalogue
registers these tables with their CKAN resource IDs:
catalog <- morie_dataset_catalog()
catalog[catalog$source == "otis", c("key", "name", "ckan_resource_id")]
#> key
#> 37 otisloc
#> 38 otisa01
#> 39 otisb01
#> 40 otisb09
#> 41 otisc11
#> name
#> 37 OTIS: Ontario's adult provincial correctional institutions (locations)
#> 38 OTIS a01: Restrictive Confinement - Detailed Dataset
#> 39 OTIS b01: Segregation - Detailed Dataset
#> 40 OTIS b09: Individuals in Segregation - Number of Placements
#> 41 OTIS c11: Individuals in Segregation/RC by Aggregate Length
#> ckan_resource_id
#> 37
#> 38 5a0c5804-a055-4031-9743-73f556e43bb4
#> 39 406e6d90-d568-4553-8ca7-bc9f90e133b9
#> 40 df24e943-d52b-43a8-a10e-a3cc906e26bb
#> 41 9c7b74a5-53ad-4ef0-a7a6-97772cd01c55To pull the full b01 from CKAN (network; not run when the vignette is built):
b01_full <- morie_load_dataset("otisb01")
nrow(b01_full)
# 82001 (October 2026)The loader downloads the CSV on the first call, keeps it in the
per-user cache under morie_cache_dir(), and reads that copy
on later calls (refresh = TRUE downloads it again).
TPS via ArcGIS Open Data
Toronto Police Service publishes per-category crime events through ArcGIS Online. MORIE knows the layer URLs for nine categories.
morie_tps_layer_urls()
#> Assault
#> "https://services.arcgis.com/S9th0jAJ7bqgIRjw/arcgis/rest/services/Assault_Open_Data/FeatureServer/0"
#> AutoTheft
#> "https://services.arcgis.com/S9th0jAJ7bqgIRjw/arcgis/rest/services/Auto_Theft_Open_Data/FeatureServer/0"
#> BicycleTheft
#> "https://services.arcgis.com/S9th0jAJ7bqgIRjw/arcgis/rest/services/Bicycle_Thefts_Open_Data/FeatureServer/0"
#> BreakAndEnter
#> "https://services.arcgis.com/S9th0jAJ7bqgIRjw/arcgis/rest/services/Break_and_Enter_Open_Data/FeatureServer/0"
#> Homicides
#> "https://services.arcgis.com/S9th0jAJ7bqgIRjw/arcgis/rest/services/Homicides_Open_Data_ASR_RC_TBL_002/FeatureServer/0"
#> Robbery
#> "https://services.arcgis.com/S9th0jAJ7bqgIRjw/arcgis/rest/services/Robbery_Open_Data/FeatureServer/0"
#> ShootingAndFirearmDiscarges
#> "https://services.arcgis.com/S9th0jAJ7bqgIRjw/arcgis/rest/services/Shooting_and_Firearm_Discharges_Open_Data/FeatureServer/0"
#> TheftFromMV
#> "https://services.arcgis.com/S9th0jAJ7bqgIRjw/arcgis/rest/services/Theft_From_Motor_Vehicle_Open_Data/FeatureServer/0"
#> TheftOver
#> "https://services.arcgis.com/S9th0jAJ7bqgIRjw/arcgis/rest/services/Theft_Over_Open_Data/FeatureServer/0"To fetch a single category (network; not run when the vignette is built):
csv_path <- morie_fetch_tps("Assault")
assault <- utils::read.csv(csv_path)
nrow(assault)
# 267796 (the layer count in October 2026)The fetcher pages through /query with up to 2,000
records per page (max_per_page) and writes
tps_Assault.csv into cache_dir, which defaults
to a directory under tempdir(). Later calls in the session
return the cached path unless overwrite = TRUE; pass a
persistent cache_dir to keep the file between sessions.
Filtering at the server side:
recent <- morie_fetch_tps("Homicides",
where = "OCC_YEAR >= 2024",
overwrite = TRUE)SIU Director’s Reports
The Ontario SIU publishes Director’s Reports at
siu.on.ca. morie_fetch_siu() returns the path
of an SIU.csv file. By default it writes the panel-reviewed corpus
bundled in rmoriedata, which needs no network access:
csv_path <- morie_fetch_siu(lang = "en")
#> SIU: wrote the panel-reviewed corpus from rmoriedata (2309 rows) to /tmp/Rtmp5SNd8H/morie/siu/SIU.csv. For a live re-fetch set options(morie.siu.allow_fetch = TRUE).
siu <- utils::read.csv(csv_path)
dim(siu)
#> [1] 2309 66With options(morie.siu.allow_fetch = TRUE) it fetches
the reports live instead: it walks the DRID manifest shipped with rmorie
(morie_siu_index()), requests each report and news release
at rate_rps requests per second with
concurrency parallel requests, and parses them with the
package’s C++ parser. That takes a long time and needs the network, so
it is not run here:
options(morie.siu.allow_fetch = TRUE)
csv_path <- morie_fetch_siu(lang = "en", cache_html = TRUE,
rate_rps = 4, overwrite = TRUE)The siu-pipeline vignette covers the manifest, auditing
and corrections.
Workflow: from included sample to verified result
Putting it together – a complete worked example without any network call:
b01 <- morie_sample("otis_b01")
# Mandela classification, one row per placement
mrm_classify_mandela(b01, denominator = "row")
#> year denominator n_mandela rate pct n_broader_rc rate_broader
#> 1 2023 362 0 0.000000000 0.00 0 0.000000000
#> 2 2024 337 3 0.008902077 0.89 3 0.008902077
#> 3 2025 301 4 0.013289037 1.33 4 0.013289037
#> 4 pooled 1000 7 0.007000000 0.70 7 0.007000000
# Segregation duration KM
mrm_otis_seg_duration_km(b01,
group_cols = "MentalHealth_Alert")
#> stratum n mean_days median_days q75_days pct_above_mandela
#> 1 No 499 2.76 2 3 0.6
#> 2 Yes 501 3.29 2 4 0.8
#> median_among_above_mandela days_at_S50 days_at_S25 days_at_S10 days_at_S05
#> 1 54.0 2 3 4 6
#> 2 41.5 2 4 5 8
#> days_at_S01
#> 1 11.02
#> 2 13.00
# Mortification co-occurrence: Cramer's V across alert pairs
mrm_otis_mortification_cooccurrence(b01)
#> alert_a alert_b n chi2 df p_value cramers_v
#> 1 MentalHealth_Alert SuicideRisk_Alert 1000 33.25 1 8.12e-09 0.1823
#> 2 MentalHealth_Alert SuicideWatch_Alert 1000 12.09 1 5.08e-04 0.1099
#> 3 SuicideRisk_Alert SuicideWatch_Alert 1000 470.37 1 2.66e-104 0.6858For a full-data version, swap morie_sample("otis_b01")
for morie_load_dataset("otisb01") and re-run the same
callables.
Canadian case law and legislation via A2AJ and CanLII
Access to Algorithmic Justice (A2AJ) publishes Canadian court and
tribunal decisions and statutes and regulations as open data, through a
REST API and as one Parquet file per court on Hugging Face. The
morie_ingest_a2aj_* functions wrap both channels. The
chunks in this section need the network and are not run when the
vignette is built.
cov <- morie_ingest_a2aj_coverage("cases")
cov[cov$dataset %in% c("SCC", "ONCA", "FC"),
c("dataset", "earliest_document_date", "number_of_documents")]Search is full-text or by name, capped at 50 hits per call, and a single decision comes back with its unofficial text:
hits <- morie_ingest_a2aj_search("right to housing",
dataset = c("SCC", "ONCA"),
start_date = "2020-01-01")
doc <- morie_ingest_a2aj_fetch("2023 SCC 17", include_citations = TRUE)
substr(doc$unofficial_text_en, 1, 300)
doc$cases_cited_en[[1]]For a whole court, load its Parquet file. The native reader decodes
the LIST-typed citation columns without arrow;
pass columns to skip the full texts, which dominate the
file:
sct <- morie_ingest_a2aj_load("SCT",
columns = c("citation_en", "document_date_en", "cases_cited_en"))
edges <- morie_ingest_a2aj_citation_edges(sct)
head(sort(table(edges$to), decreasing = TRUE))A2AJ does not yet carry the Human Rights Tribunal of Ontario, the BC
Human Rights Tribunal, the Ontario Superior Court, Alberta, Quebec or
the remaining provinces (its issues #1, #3 and #4), and individual
decisions can be missing (2007 BCSC 1700, issue #2). Those
are reachable through CanLII’s API, which needs a free key
(CANLII_API_KEY; the chunk stops with a message without
one) and returns metadata plus the citation network rather than full
text:
morie_ingest_a2aj_gaps()[, c("code", "canlii_database_id")]
hrto <- morie_ingest_canlii_cases("onhrt", result_count = 200,
decision_date_after = "2024-01-01")
ids <- morie_ingest_canlii_case_id("2007 BCSC 1700")
morie_ingest_canlii_case(ids$database_id, ids$case_id)Every A2AJ record carries an upstream_license field;
check it before any commercial use, and treat all texts as unofficial
copies.
Caching layout
| Data | Fetched by | Cached in |
|---|---|---|
| catalogue datasets (OTIS, PUMFs, …) | morie_load_dataset() |
per-user cache under morie_cache_dir()
|
| TPS layers | morie_fetch_tps() |
cache_dir argument (default under
tempdir()) |
| SIU reports | morie_fetch_siu() |
cache_dir argument (default under
tempdir()) |