Row-level sanity check on a parsed SIU table (regex-only, no LLM)
Source:R/siu.R
morie_siu_sanity_check.RdFor every row in a parser-emitted SIU table, flag cells that
don't match the expected format for their column – case_number
that doesn't look like an SIU case id, date_*_iso that isn't a
valid ISO 8601 date, number_of_* that isn't a positive integer,
charges_recommended that isn't "Yes" / "No", etc. Returns a
data frame ranked by issue count so the most-broken rows surface
at the top for manual inspection against the cached HTML.
Value
A data frame with one row per source row, columns:
case_number, drid, issues_count (integer
number of suspicious cells), issues (semicolon-separated
string of field:reason pairs). Ordered descending by
issues_count.
Details
Designed to be a fast first-pass quality filter – runs in
milliseconds, no network, no LLM, no API key. Doesn't try to
verify correctness against the underlying report (that's what
morie_siu_audit_columns() is for); just checks that each
value MATCHES THE EXPECTED FORMAT for its field. A clean sanity
check is necessary but not sufficient for correctness.
Examples
# \donttest{
# Corpus-first fetch is fast when rmoriedata is installed.
csv <- morie_fetch_siu(cache_dir = file.path(tempdir(), "siu_sanity"))
#> SIU: wrote the panel-reviewed corpus from rmoriedata (5157 rows) to /tmp/RtmpOKyZlD/siu_sanity/SIU.csv. For a live re-fetch set options(morie.siu.allow_fetch = TRUE).
df <- utils::read.csv(csv, colClasses = "character")
sanity <- morie_siu_sanity_check(utils::head(df, 50))
head(sanity, 10) # worst 10 rows -- inspect against HTML
#> case_number drid issues_count
#> 1 17-OVI-201 46 4
#> 2 17-TVI-108 48 4
#> 3 17-OVI-201 47 3
#> 4 1 1
#> 5 10 1
#> 6 11 1
#> 7 12 1
#> 8 13 1
#> 9 14 1
#> 10 15 1
#> issues
#> 1 number_of_officers_involved:bad-format;charges_recommended:not-Yes/No;sex_gender_affected:bad-value;supplemental_materials:page-chrome-leak
#> 2 number_of_officers_involved:bad-format;directors_decision_reasonable:not-Yes/No;sex_gender_affected:bad-value;supplemental_materials:page-chrome-leak
#> 3 police_service:empty-when-expected;narrative_summary:suspiciously-short;supplemental_materials:page-chrome-leak
#> 4 narrative_summary:page-chrome-leak
#> 5 narrative_summary:page-chrome-leak
#> 6 narrative_summary:page-chrome-leak
#> 7 narrative_summary:page-chrome-leak
#> 8 narrative_summary:page-chrome-leak
#> 9 narrative_summary:page-chrome-leak
#> 10 narrative_summary:page-chrome-leak
table(sanity$issues_count)
#>
#> 1 3 4
#> 47 1 2
# }