MORIE

References

Imai, K., & Ratkovic, M. (2013). Estimating treatment effect heterogeneity in randomized program evaluation. Annals of Applied Statistics, 7(1), 443–470.

Chernozhukov, V., Demirer, M., Duflo, E., & Fernandez-Val, I. (2020). Generic machine learning inference on heterogeneous treatment effects in randomized experiments. NBER Working Paper 24678.

morie.causal.estimate_cate(data, *, treatment, outcome, covariates, meta_learner='t_learner')[source]¶

Estimate per-unit Conditional Average Treatment Effects (CATE).

The CATE for unit i is $\tau(X_i) = \mathbb{E}[Y(1) - Y(0) \mid X = X_i]$.

T-learner (default): fits separate outcome models on treated and control units, then computes $\hat{\tau}(X_i) = \hat{\mu}_1(X_i) - \hat{\mu}_0(X_i)$.

S-learner: fits a single outcome model on all units with treatment as a feature, then computes $\hat{\tau}(X_i) = \hat{\mu}(X_i, T=1) - \hat{\mu}(X_i, T=0)$.

Parameters:

data (pandas.DataFrame) – Input DataFrame.
treatment (str) – Binary treatment column (0/1).
outcome (str) – Outcome column.
covariates (list[str]) – Covariate column names.
meta_learner (str) – "t_learner" or "s_learner" (default "t_learner").

Returns:

Series of per-unit CATE estimates, indexed like the input.

Return type:

pandas.Series

References

Kunzel, S. R., Sekhon, J. S., Bickel, P. J., & Yu, B. (2019). Metalearners for estimating heterogeneous treatment effects using machine learning. PNAS, 116(10), 4156–4165. https://doi.org/10.1073/pnas.1804597116

morie.causal.estimate_late(data, *, treatment, outcome, instrument, covariates=None)[source]¶

Estimate the Local Average Treatment Effect (LATE) via instrumental variables.

For a binary instrument $Z$, the Wald estimator (simple IV) is:

\[\widehat{\text{LATE}} = \frac{\text{Cov}(Y, Z)}{\text{Cov}(T, Z)} = \frac{\bar{Y}_{Z=1} - \bar{Y}_{Z=0}}{\bar{T}_{Z=1} - \bar{T}_{Z=0}}\]

With covariates, the function attempts 2SLS via linearmodels.iv.IV2SLS or statsmodels IV regression, falling back to the Wald estimator if neither IV library is installed.

Parameters:

data (pandas.DataFrame) – Input DataFrame.
treatment (str) – Endogenous treatment column.
outcome (str) – Outcome column.
instrument (str) – Instrument column.
covariates (list[str] or None) – Exogenous covariate column names (optional).

Returns:

Dictionary with late, se, ci (tuple), f_stat, method.

Return type:

dict[str, Any]

References

Imbens, G. W., & Angrist, J. D. (1994). Identification and estimation of local average treatment effects. Econometrica, 62(2), 467–475. https://doi.org/10.2307/2951620

Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. JASA, 91(434), 444–455.

morie.causal.estimate_irm(data, *, treatment, outcome, covariates, n_folds=5, random_state=42)[source]¶

Estimate the ATE via the Interactive Regression Model (IRM), native DML.

The IRM extends the partially linear model by allowing treatment effect heterogeneity. It models:

\[ \begin{align}\begin{aligned}Y = g_0(T, X) + U, \quad \mathbb{E}[U \mid X, T] = 0\\T = m_0(X) + V, \quad \mathbb{E}[V \mid X] = 0\end{aligned}\end{align} \]

where $g_0$ is the outcome regression and $m_0$ is the propensity score. The Neyman-orthogonal score for the ATE is:

\[\psi = g_0(1, X) - g_0(0, X) + \frac{T(Y - g_0(1,X))}{m_0(X)} - \frac{(1-T)(Y - g_0(0,X))}{1 - m_0(X)} - \theta\]

Parameters:

data (pandas.DataFrame) – Input DataFrame.
treatment (str) – Binary treatment column.
outcome (str) – Outcome column.
covariates (list[str]) – Covariate column names.
n_folds (int) – Number of cross-fitting folds (default 5).
random_state (int) – Random seed (default 42).

Returns:

Dictionary with ate, se, ci_lower, ci_upper, n, method.

Return type:

dict[str, Any]

References

Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68. https://doi.org/10.1111/ectj.12097

morie.causal.estimate_double_ml(data, outcome, treatment, covariates, *, random_state=42, n_folds=5, n_rep=1)[source]¶

Estimate the Average Treatment Effect using Double Machine Learning (DML).

Native Partially Linear Regression (morie.fn.plr): cross-fitted ridge nuisances for the outcome regression and the treatment model, partialling-out orthogonal score.

Reproducibility¶

The cross-fitting fold assignment is drawn from random_state with the package’s own RNG, so a given seed fixes the schedule; n_rep is accepted for signature compatibility and ignored (single repetition).

To change the seed for a sensitivity run, pass random_state=<int>.

param data:: The dataset containing outcome, treatment, and covariates.
type data:: pandas.DataFrame
param outcome:: Name of the continuous outcome variable.
type outcome:: str
param treatment:: Name of the treatment variable.
type treatment:: str
param covariates:: List of covariate column names to control for.
type covariates:: list[str]
param random_state:: Integer seed for all RNGs. Default 42.
type random_state:: int, optional
param n_folds:: Number of cross-fitting folds. Default 5.
type n_folds:: int, optional
param n_rep:: Number of repeated cross-fitting repetitions. Default 1.
type n_rep:: int, optional
return:: dict with ate, se, ci_lower, ci_upper, pval, n_obs from the native cross-fitted PLR.
rtype:: dict

References

Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68. https://doi.org/10.1111/ectj.12097

Parameters:

data (DataFrame)
outcome (str)
treatment (str)
covariates (list)
random_state (int)
n_folds (int)
n_rep (int)

Treatment effect estimations (ATE, ATT, ATU, LATE, G-computation).

This module provides:

estimate_ate() – IPW-weighted OLS ATE (existing, preserved).
estimate_plr() – Partially Linear Regression (native cross-fitted DML).
estimate_pliv() – Partially Linear IV (LATE), native DML or 2SLS.
estimate_ate_gcomputation() – G-computation (outcome regression) ATE.
sensitivity_rosenbaum() – Rosenbaum bounds for hidden confounding.
e_value() – E-value for unmeasured confounding (VanderWeele & Ding, 2017).

References

Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey,: W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1-C68.
Robins, J. M. (1986). A new approach to causal inference in mortality studies: with a sustained exposure period. Mathematical Modelling, 7, 1393-1512.
VanderWeele, T. J., & Ding, P. (2017). Sensitivity analysis in observational: research: Introducing the E-value. Annals of Internal Medicine, 167(4), 268-274.

Rosenbaum, P. R. (2002). Observational Studies (2nd ed.). Springer.

morie.effects.estimate_ate(data, outcome, treatment, weights_col)[source]¶

Estimate Average Treatment Effect (ATE) using a weighted linear model.

Parameters:

data (pandas.DataFrame) – The pandas DataFrame containing the analytical sample.
outcome (str) – The name of the outcome variable column.
treatment (str) – The name of the binary treatment indicator column.
weights_col (str) – The name of the column containing the analytical weights (e.g. IPTW).

Returns:

A tuple containing the estimated ATE coefficient and its standard error.

Return type:

morie.effects.estimate_plr(data, *, treatment, outcome, covariates, n_folds=5, random_state=42)[source]¶

Partially Linear Regression ATE via the native double-ML estimator (cross-fitted ridge nuisances, partialling-out score).

Thin front-end for morie.fn.plr.estimate_plr(); see that function for the model, the score, and references.

Returns:

dict with ate, se, ci_lower, ci_upper, pval, n_obs.

Parameters:

data (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
n_folds (int)
random_state (int)

Return type:

morie.effects.estimate_pliv(data, *, treatment, outcome, instrument, covariates, n_folds=5, random_state=42)[source]¶

Partially linear IV (LATE) via the native double-ML estimator (cross-fitted ridge nuisances, IV-type orthogonal score).

Thin front-end for morie.fn.pliv.estimate_pliv(); see that function for the model, the score, and references.

Returns:

dict with late, se, ci_lower, ci_upper, pval, n_obs, method.

Parameters:

data (DataFrame)
treatment (str)
outcome (str)
instrument (str)
covariates (list[str])
n_folds (int)
random_state (int)

Return type:

morie.effects.estimate_ate_gcomputation(data, *, treatment, outcome, covariates, outcome_model='linear')[source]¶

G-computation (outcome regression / standardisation) ATE estimator.

The G-computation estimator proceeds in three steps:

Fit an outcome model $E[Y | T, X]$ on the observed data.
Predict potential outcomes $\hat{Y}(1)$ and $\hat{Y}(0)$ for every unit by setting T = 1 and T = 0 respectively.
Compute ATE as the average difference:

\[\widehat{\text{ATE}} = \frac{1}{n} \sum_i \left(\hat{Y}_i(1) - \hat{Y}_i(0)\right)\]

The outcome model is the unpenalised GLM (OLS or logistic) and the standard error is the M-estimation sandwich of stdReg::stdGlm (Sjolander 2016) – the GLM score equations stacked with the two standardisation equations, so the outcome model’s own uncertainty is included – the same estimate and SE as the R arm.

Parameters:

data (DataFrame) – DataFrame containing all required columns.
treatment (str) – Column name of the binary treatment indicator (0/1).
outcome (str) – Column name of the outcome variable.
covariates (list[str]) – List of covariate column names.
outcome_model (str) – "linear" (OLS) for continuous outcomes or "logistic" for binary outcomes. Default "linear".

Returns:

dict with keys ate, se, ci_lower, ci_upper, n_obs, outcome_model.

Raises:

ValueError – If required columns are missing or outcome_model is invalid.

Return type:

References

Robins, J. M. (1986). A new approach to causal inference in mortality: studies. Mathematical Modelling, 7, 1393-1512.
Hernan, M. A., & Robins, J. M. (2020). Causal Inference: What If.: Chapman & Hall/CRC. (Chapter 13.)

morie.effects.sensitivity_rosenbaum(data, *, treatment, outcome, covariates, gamma_range=(1.0, 3.0), n_gamma=20)[source]¶

Rosenbaum bounds sensitivity analysis for hidden confounding.

Tests whether the sign-rank test conclusion is robust to an unmeasured confounder that could increase the odds of treatment assignment by a factor of Gamma (the sensitivity parameter).

For a range of Gamma values, the method computes: - p_lower: p-value under the most favourable assignment (best case) - p_upper: p-value under the most adverse assignment (worst case)

The analysis is based on the Wilcoxon signed-rank statistic applied to matched pairs. Here we approximate the matched analysis by using all discordant (T=1, T=0) pairs sorted by outcome.

Parameters:

data (DataFrame) – DataFrame containing all required columns.
treatment (str) – Column name of the binary treatment indicator (0/1).
outcome (str) – Column name of the outcome variable.
covariates (list[str]) – Covariate column names (used for matching approximation).
gamma_range (tuple[float, float]) – Tuple (min_gamma, max_gamma). Default (1.0, 3.0).
n_gamma (int) – Number of Gamma values to evaluate. Default 20.

Returns:

DataFrame with columns Gamma, p_lower, p_upper.

Raises:

ValueError – If required columns are missing or gamma_range is invalid.

Return type:

DataFrame

Notes

This implementation uses the normal approximation to the signed-rank distribution under sensitivity bounds (Rosenbaum, 2002, Chapter 4). For small samples or exact analysis, use the R sensitivitymw or rbounds package.

References

Rosenbaum, P. R. (2002). Observational Studies (2nd ed.). Springer. (Chapter 4.) Rosenbaum, P. R. (2007). Sensitivity analysis for m-estimates, tests, and

confidence intervals in matched observational studies. Biometrics, 63(2), 456-464.

morie.effects.e_value(ate, se, *, null=0.0)[source]¶

E-value for unmeasured confounding (VanderWeele & Ding, 2017).

The E-value is the minimum strength of association (on the risk-ratio scale) that an unmeasured confounder would need to have with both the treatment and the outcome to fully explain away the observed effect, conditional on the measured covariates.

For a risk ratio (RR) effect estimate the E-value is:

\[E = \text{RR} + \sqrt{\text{RR} \cdot (\text{RR} - 1)}\]

where RR > 1. For RR < 1, compute E on 1/RR.

Since the ATE here is a difference (not a ratio), we first convert using a delta-method approximation to get a risk-ratio-like effect, using the relationship RR ≈ exp(|ATE - null| / se) (treating the z-score as a log-RR approximation). This is an approximation appropriate for continuous-scale effects reported with a standard error.

Parameters:

ate (float) – Point estimate of the treatment effect.
se (float) – Standard error of the ATE estimate (must be > 0).
null (float) – Null value to test against. Default 0.0.

Returns:

E-value (float >= 1.0). Returns 1.0 if the estimate is at the null.

Raises:

ValueError – If se <= 0.

Return type:

Notes

For binary outcomes with a risk ratio or odds ratio estimate, convert the OR to the risk-ratio scale first, then apply the E-value formula directly. This function is designed for the continuous-ATE setting.

References

VanderWeele, T. J., & Ding, P. (2017). Sensitivity analysis in observational: research: Introducing the E-value. Annals of Internal Medicine, 167(4), 268-274.
VanderWeele, T. J., Mathur, M. B., & Ding, P. (2019). Correcting: misinterpretations of the E-value. Annals of Internal Medicine, 170(2), 131-132.

Matching methods for causal inference in observational studies.

Implements propensity score matching (nearest-neighbour, caliper, with/without replacement), exact matching, coarsened exact matching (CEM), Mahalanobis distance matching, optimal matching, genetic matching, entropy balancing, cardinality matching, subclassification, full matching, balance diagnostics, treatment effect estimation, Abadie-Imbens standard errors, Rosenbaum bounds, doubly-robust estimation, and multi-treatment/longitudinal matching.

References

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41

Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1), 1–21. https://doi.org/10.1214/09-STS313

Iacus, S. M., King, G., & Porro, G. (2012). Causal inference without balance checking: Coarsened exact matching. Political Analysis, 20(1), 1–24. https://doi.org/10.1093/pan/mpr013

Abadie, A., & Imbens, G. W. (2006). Large sample properties of matching estimators for average treatment effects. Econometrica, 74(1), 235–267. https://doi.org/10.1111/j.1468-0262.2006.00655.x

Hainmueller, J. (2012). Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political Analysis, 20(1), 25–46. https://doi.org/10.1093/pan/mpr025

class morie.matching.MatchResult(matched_data, n_treated, n_matched_control, match_pairs, method='nearest_neighbor', details=<factory>)[source]¶

Bases: object

Container for matching output.

Variables:

matched_data (pd.DataFrame) – Matched dataset with match indicators.
n_treated (int) – Number of treated units.
n_matched_control (int) – Number of matched control units.
match_pairs (pd.DataFrame) – Mapping from treated to matched control indices.
method (str) – Matching method used.
details (dict) – Additional diagnostics (e.g., caliper, replacement).

Parameters:

matched_data (DataFrame)
n_treated (int)
n_matched_control (int)
match_pairs (DataFrame)
method (str)
details (dict[str, Any])

matched_data: DataFrame¶

n_treated: int¶

n_matched_control: int¶

match_pairs: DataFrame¶

method: str = 'nearest_neighbor'¶

details: dict[str, Any]¶

class morie.matching.BalanceResult(balance_table, overall_balance, max_smd, balanced)[source]¶

Bases: object

Container for balance diagnostics.

Variables:

balance_table (pd.DataFrame) – Table with standardised mean differences, variance ratios, etc.
overall_balance (float) – Mean absolute SMD across covariates.
max_smd (float) – Maximum absolute SMD.
balanced (bool) – Whether all SMDs are below the threshold (default 0.1).

Parameters:

balance_table (DataFrame)
overall_balance (float)
max_smd (float)
balanced (bool)

balance_table: DataFrame¶

overall_balance: float¶

max_smd: float¶

balanced: bool¶

class morie.matching.TreatmentEffectResult(estimand, estimate, std_error, ci_lower, ci_upper, p_value, n_obs, details=<factory>)[source]¶

Bases: object

Container for treatment effect estimates from matched samples.

Variables:

estimand (str) – ATT, ATE, or ATC.
estimate (float) – Point estimate.
std_error (float) – Standard error.
ci_lower (float) – Lower CI bound.
ci_upper (float) – Upper CI bound.
p_value (float) – Two-sided p-value.
n_obs (int) – Effective sample size.
details (dict) – Extra information.

Parameters:

estimand (str)
estimate (float)
std_error (float)
ci_lower (float)
ci_upper (float)
p_value (float)
n_obs (int)
details (dict[str, Any])

estimand: str¶

estimate: float¶

std_error: float¶

ci_lower: float¶

ci_upper: float¶

p_value: float¶

n_obs: int¶

details: dict[str, Any]¶

summary()[source]¶

One-row summary.

Return type:: DataFrame

morie.matching.estimate_propensity_score(data, treatment, covariates, *, model='logistic', max_iter=1000)[source]¶

Estimate propensity scores via logistic regression.

Parameters:

data (pd.DataFrame)
treatment (str) – Binary treatment column (0/1).
covariates (list of str) – Covariates for the propensity model.
model (str) – "logistic" (default) or "gbm" for gradient boosting.
max_iter (int) – Maximum iterations for logistic regression.

Returns:

Propensity scores indexed like data.

Return type:

pd.Series

References

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.

morie.matching.trim_propensity_scores(ps, lower=0.01, upper=0.99)[source]¶

Clip propensity scores to [lower, upper].

Parameters:

ps (pd.Series) – Raw propensity scores.
lower (float)
upper (float)

Return type:

pd.Series

morie.matching.common_support(data, treatment, ps_col='propensity_score', *, method='minmax')[source]¶

Restrict sample to the region of common support.

Parameters:

data (pd.DataFrame)
treatment (str)
ps_col (str) – Propensity score column name.
method (str) – "minmax" (overlap of ranges) or "trim" (drop extreme 5 % from each tail).

Returns:

Subset with common support.

Return type:

pd.DataFrame

morie.matching.match_nearest_neighbor(data, treatment, covariates, *, n_neighbors=1, caliper=None, replace=False, ps=None, alpha=0.05)[source]¶

Nearest-neighbour matching on the logit propensity score.

MatchIt’s matcher (the R arm’s native port, pair-identical to MatchIt::matchit(method = "nearest", link = "linear.logit")): without replacement every treated unit gets its first control before any gets a second, treated units taken in decreasing score; with replacement each treated unit takes its n_neighbors nearest controls.

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
n_neighbors (int) – Number of matches per treated unit.
caliper (float, optional) – Maximum logit-propensity distance for a valid match, in standard deviations of the logit score.
replace (bool) – Whether controls can be matched to multiple treated units.
ps (pd.Series, optional) – Pre-computed propensity scores (replace the fitted model).
alpha (float) – Significance level for the balance checks in details.

Return type:

morie.matching.match_exact(data, treatment, exact_vars)[source]¶

Exact matching on discrete covariates.

Matches treated and control units that share identical values on all exact_vars. Unmatched units are dropped.

Parameters:

data (pd.DataFrame)
treatment (str)
exact_vars (list of str)

Return type:

morie.matching.match_cem(data, treatment, covariates, *, n_bins=5)[source]¶

Coarsened Exact Matching (Iacus, King & Porro, 2012).

Coarsens continuous covariates into bins and performs exact matching on the coarsened values. Returns the original (uncoarsened) data with CEM strata weights.

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
n_bins (int or dict) – Number of bins for each covariate. A single int applies to all.

Returns:

matched_data includes a _cem_weight column.

Return type:

References

Iacus, S. M., King, G., & Porro, G. (2012). Causal inference without balance checking: Coarsened exact matching. Political Analysis, 20(1), 1–24.

morie.matching.match_mahalanobis(data, treatment, covariates, *, n_neighbors=1, caliper=None, replace=False, exact=None)[source]¶

Mahalanobis distance matching.

Matches on the Mahalanobis distance of covariates rather than propensity score. Optionally combined with exact matching on discrete variables.

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str) – Continuous covariates for distance calculation.
n_neighbors (int)
caliper (float, optional)
replace (bool)
exact (list of str, optional) – Variables to match exactly before distance matching.

Return type:

morie.matching.match_optimal_pair(data, treatment, covariates, *, distance='propensity', ps=None)[source]¶

Optimal pair matching minimising total distance.

Uses a greedy approximation to the assignment problem (Hungarian algorithm is O(n^3); we use sorted nearest-neighbor for scalability).

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
distance (str) – "propensity" or "mahalanobis".
ps (pd.Series, optional)

Return type:

morie.matching.match_full(data, treatment, covariates, *, ps=None, n_subclasses=10)[source]¶

Optimal full matching (Rosenbaum 1991; Hansen 2004).

Every unit is placed in a matched set of one treated unit and one or more controls, or one control and one or more treated units, with the smallest possible total within-set propensity-score distance – computed exactly as a minimum-weight edge cover (the R arm’s engine). Weights are MatchIt’s ATT weights for method = "full".

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
ps (pd.Series, optional) – Propensity scores; estimated by logistic regression when omitted.
n_subclasses (int) – Ignored (the number of sets is part of the optimum); kept for compatibility.

Returns:

matched_data with distance, weights and subclass columns; details["total_distance"].

Return type:

References

Rosenbaum, P. R. (1991). A characterization of optimal designs for observational studies. JRSS B, 53(3), 597–610. Hansen, B. B. (2004). Full matching in an observational study of coaching for the SAT. JASA, 99(467), 609–618.

morie.matching.subclassify(data, treatment, covariates, *, ps=None, n_strata=5)[source]¶

Subclassification on the propensity score, MatchIt’s rules.

Cut points are quantiles of the treated units’ scores; a subclass left without a treated or a control unit takes the nearest unit of that group from the closest subclass that can spare one (MatchIt’s min.n = 1). Weights: 1 for treated, the subclass’s treated/control ratio for controls, each group rescaled to its size (the R arm’s engine).

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
ps (pd.Series, optional)
n_strata (int)

Returns:

data_with_strata (pd.DataFrame) – The data with distance, weights, subclass, _ps and _stratum columns.
stratum_effects (pd.DataFrame) – Per-stratum sample sizes and score ranges.

Return type:

tuple[DataFrame, DataFrame]

morie.matching.entropy_balance(data, treatment, covariates, *, max_moment=1, max_iter=500, tol=1e-06)[source]¶

Entropy balancing weights for the control group.

Finds weights $w_i$ for control units that minimise the Kullback-Leibler divergence from uniform weights subject to moment balance constraints:

\[\sum_{i: D_i=0} w_i\, x_i^k = \bar x^k_{D=1} \quad \text{for } k = 1, \ldots, \text{max\_moment}\]

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
max_moment (int) – Balance on mean (1), mean + variance (2), or up to 3rd moment.
max_iter (int)
tol (float)

Returns:

Entropy balancing weights indexed like data. Treated units receive weight 1; control units receive the balancing weights.

Return type:

pd.Series

References

Hainmueller, J. (2012). Entropy balancing for causal effects. Political Analysis, 20(1), 25–46.

morie.matching.match_genetic(data, treatment, covariates, *, n_neighbors=1, pop_size=50, n_generations=20, seed=42)[source]¶

Genetic matching (Diamond & Sekhon, 2013).

Uses a genetic algorithm to find the optimal weight matrix for Mahalanobis distance matching that maximises covariate balance.

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
n_neighbors (int)
pop_size (int) – Genetic algorithm population size.
n_generations (int)
seed (int)

Return type:

References

Diamond, A., & Sekhon, J. S. (2013). Genetic matching for estimating causal effects. Review of Economics and Statistics, 95(3), 932–945.

morie.matching.match_variable_ratio(data, treatment, covariates, *, min_ratio=1, max_ratio=5, caliper=0.2, ps=None)[source]¶

Variable-ratio (“extremal”) matching (Ming & Rosenbaum 2000).

MatchIt’s rules for method = "nearest", min.controls, max.controls (the R arm’s engine): treated units with the most typical scores get max_ratio controls and the rest min_ratio, about the midpoint per treated unit in all; greedy matching without replacement on the propensity score within caliper standard deviations of it.

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
min_ratio (int)
max_ratio (int)
caliper (float or None)
ps (pd.Series, optional)

Returns:

matched_data with distance, weights and subclass columns.

Return type:

morie.matching.match_cardinality(data, treatment, covariates, *, balance_threshold=0.1, ps=None)[source]¶

Cardinality matching: maximise sample size subject to balance.

Finds the largest subset of matched pairs such that the standardised mean difference on each covariate is below balance_threshold.

Uses an iterative caliper-tightening heuristic.

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
balance_threshold (float)
ps (pd.Series, optional)

Return type:

References

Zubizarreta, J. R. (2012). Using mixed integer programming for matching in an observational study of kidney failure after surgery. Journal of the American Statistical Association, 107(500), 1360–1371.

morie.matching.balance_diagnostics(data, treatment, covariates, *, weights=None, threshold=0.1)[source]¶

Compute balance diagnostics for matched/weighted samples.

Reports standardised mean differences (SMD), variance ratios, and Kolmogorov-Smirnov statistics for each covariate.

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
weights (str, optional) – Column of matching/weighting weights.
threshold (float) – SMD threshold for balanced flag (default 0.1).

Return type:

BalanceResult

morie.matching.love_plot_data(unmatched_data, matched_data, treatment, covariates, *, weights_col=None)[source]¶

Generate data for a Love plot comparing pre- and post-matching balance.

Parameters:

unmatched_data (pd.DataFrame)
matched_data (pd.DataFrame)
treatment (str)
covariates (list of str)
weights_col (str, optional)

Returns:

Columns: covariate, smd_before, smd_after.

Return type:

pd.DataFrame

morie.matching.balance_table(data, treatment, covariates, *, weights=None)[source]¶

Construct a publication-ready balance table.

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
weights (str, optional)

Return type:

pd.DataFrame

morie.matching.estimate_att_matched(data, outcome, treatment, match_pairs, *, weights=None, alpha=0.05)[source]¶

Estimate ATT from matched data.

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
match_pairs (pd.DataFrame) – With columns treated_idx and control_idx.
weights (str, optional)
alpha (float)

Return type:

morie.matching.estimate_ate_matched(data, outcome, treatment, covariates, *, weights=None, alpha=0.05)[source]¶

Estimate ATE from matched/weighted data via weighted mean difference.

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
covariates (list of str)
weights (str, optional)
alpha (float)

Return type:

morie.matching.estimate_atc_matched(data, outcome, treatment, match_pairs, *, alpha=0.05)[source]¶

Estimate ATC from matched data (matching controls to treated).

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
match_pairs (pd.DataFrame) – With columns control_idx and treated_idx (reverse matching direction).
alpha (float)

Return type:

morie.matching.abadie_imbens_se(data, outcome, treatment, match_pairs, *, n_matches=1, covariates=None)[source]¶

Abadie-Imbens (2006) standard error of the matching ATT (sample ATT), as Matching::Match(estimand = "ATT", sample = TRUE):

\[\hat V = \frac{1}{N_1^2} \sum_i \bigl(W_i + (1 - W_i) K_M(i) / M\bigr)^2 \hat\sigma^2(X_i, W_i)\]

where $K_M(i)$ counts the times control i is used as a match and $\hat\sigma^2(X_i, W_i) = \tfrac12 (Y_i - Y_{l(i)})^2$ with $l(i)$ the nearest unit of the SAME treatment arm on the covariates scaled by their inverse variances (Var.calc = 1). Without covariates the conditional variance is approximated by half the squared outcome difference of each matched pair.

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
match_pairs (pd.DataFrame) – treated_idx / control_idx index labels of the matches.
n_matches (int) – M, the matches per treated unit.
covariates (list of str, optional) – The matching covariates, for AI’s within-arm variance estimate.

Returns:

Abadie-Imbens SE.

Return type:

References

Abadie, A., & Imbens, G. W. (2006). Large sample properties of matching estimators for average treatment effects. Econometrica, 74(1), 235–267.

morie.matching.rosenbaum_bounds(data, outcome, treatment, match_pairs, *, gamma_range=None)[source]¶

Rosenbaum bounds for sensitivity to hidden bias.

For each value of $\Gamma$ (the maximum odds ratio of differential treatment assignment due to an unobserved confounder), computes the upper and lower bounds on the p-value for the treatment effect.

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
match_pairs (pd.DataFrame)
gamma_range (sequence of float, optional) – Values of $\Gamma$ to evaluate (default 1.0 to 3.0).

Returns:

Columns: gamma, p_lower, p_upper, significant_lower, significant_upper.

Return type:

pd.DataFrame

References

Rosenbaum, P. R. (2002). Observational Studies (2nd ed.). Springer.

morie.matching.doubly_robust_matching(data, outcome, treatment, covariates, *, ps=None, n_bootstrap=200, seed=42, alpha=0.05)[source]¶

Doubly-robust ATT estimation combining matching and regression.

Matches on propensity score, then uses bias-corrected regression adjustment within matched pairs. Consistent if either the propensity score model or the outcome model is correct.

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
covariates (list of str)
ps (pd.Series, optional)
n_bootstrap (int)
seed (int)
alpha (float)

Return type:

morie.matching.match_multi_treatment(data, treatment, covariates, *, reference_group=None, method='nearest_neighbor')[source]¶

Matching with multiple (> 2) treatment groups.

For each non-reference treatment level, matches treated units to the reference group using propensity scores from multinomial logistic regression.

Parameters:

data (pd.DataFrame)
treatment (str) – Column with multiple treatment levels.
covariates (list of str)
reference_group (any, optional) – The reference (control) group level. Defaults to the most frequent level.
method (str) – "nearest_neighbor" or "mahalanobis".

Returns:

Keys are treatment levels; values are MatchResult.

Return type:

morie.matching.match_longitudinal(data, treatment, covariates, unit, time, treatment_time, *, n_pre_periods=1, method='nearest_neighbor')[source]¶

Longitudinal matching for panel data.

Matches treated and control units based on pre-treatment covariate values, allowing for time-varying covariates.

Parameters:

data (pd.DataFrame) – Panel data.
treatment (str)
covariates (list of str)
unit (str)
time (str)
treatment_time (str) – Column indicating when treatment begins for each unit.
n_pre_periods (int) – Number of pre-treatment periods to use for matching.
method (str)

Return type:

morie.matching.matching_quality(unmatched_data, matched_data, treatment, covariates, *, weights=None)[source]¶

Comprehensive matching quality assessment.

Compares balance before and after matching and computes summary metrics including percent bias reduction, number of balanced covariates, and overlap statistics.

Parameters:

unmatched_data (pd.DataFrame)
matched_data (pd.DataFrame)
treatment (str)
covariates (list of str)
weights (str, optional)

Returns:

Keys: balance_before, balance_after, bias_reduction, pct_balanced_before, pct_balanced_after.

Return type:

morie.matching.overlap_diagnostics(data, treatment, covariates, *, ps=None)[source]¶

Propensity score overlap diagnostics.

Parameters:

data (pd.DataFrame)
treatment (str)
covariates (list of str)
ps (pd.Series, optional)

Returns:

ps_summary (by group), overlap_region, n_off_support, effective_sample_size.

Return type:

Instrumental Variables and Two-Stage Least Squares estimators.

Implements 2SLS, LIML, GMM, CUE-GMM, weak-instrument diagnostics (Cragg-Donald, Kleibergen-Paap, Stock-Yogo), Anderson-Rubin inference, Sargan/Hansen J tests, Hausman tests, JIVE, split-sample IV, IV probit, panel IV, control function, and comprehensive residual diagnostics.

References

Angrist, J. D., & Pischke, J.-S. (2009). Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press.

Stock, J. H., & Yogo, M. (2005). Testing for weak instruments in linear IV regression. In Identification and Inference for Econometric Models: Essays in Honor of Thomas Rothenberg (pp. 80–108). Cambridge University Press.

Kleibergen, F. (2002). Pivotal statistics for testing structural parameters in instrumental variables regression. Econometrica, 70(5), 1781–1803. https://doi.org/10.1111/1468-0262.00353

Sargan, J. D. (1958). The estimation of economic relationships using instrumental variables. Econometrica, 26(3), 393–415.

class morie.iv.IVResult(coefficients, std_errors, t_stats, p_values, ci_lower, ci_upper, variable_names, n_obs, method='2sls', details=<factory>)[source]¶

Bases: object

Container for an IV/2SLS regression result.

Variables:

coefficients (np.ndarray) – Estimated coefficient vector.
std_errors (np.ndarray) – Standard errors.
t_stats (np.ndarray) – t-statistics.
p_values (np.ndarray) – Two-sided p-values.
ci_lower (np.ndarray) – Lower 95 % CI bounds.
ci_upper (np.ndarray) – Upper 95 % CI bounds.
variable_names (list of str) – Names of variables corresponding to each coefficient.
n_obs (int) – Number of observations.
method (str) – Estimation method name.
details (dict) – Additional diagnostics.

Parameters:

coefficients (marr)
std_errors (marr)
t_stats (marr)
p_values (marr)
ci_lower (marr)
ci_upper (marr)
variable_names (list[str])
n_obs (int)
method (str)
details (dict[str, Any])

coefficients: marr¶

std_errors: marr¶

t_stats: marr¶

p_values: marr¶

ci_lower: marr¶

ci_upper: marr¶

variable_names: list[str]¶

n_obs: int¶

method: str = '2sls'¶

details: dict[str, Any]¶

summary()[source]¶

Return a summary table.

Return type:: DataFrame

property endogenous_coef: float¶: Return the coefficient on the first endogenous variable.

property endogenous_se: float¶: Return the SE on the first endogenous variable.

class morie.iv.DiagnosticResult(statistic, p_value, test_name, critical_values=None, details=<factory>)[source]¶

Bases: object

Container for a single diagnostic test.

Parameters:

statistic (float)
p_value (float)
test_name (str)
critical_values (dict[str, float] | None)
details (dict[str, Any])

statistic: float¶

p_value: float¶

test_name: str¶

critical_values: dict[str, float] | None = None¶

details: dict[str, Any]¶

morie.iv.tsls(data, outcome, endogenous, instruments, exogenous=None, *, cluster=None, robust=True, alpha=0.05)[source]¶

Two-Stage Least Squares (2SLS) estimator.

First stage: regress each endogenous variable on the instruments and exogenous variables. Second stage: regress the outcome on the fitted endogenous variables and exogenous variables.

\[\hat\beta_{2SLS} = (X'P_Z X)^{-1} X'P_Z y\]

where $P_Z = Z(Z'Z)^{-1}Z'$ and $Z$ includes both excluded instruments and included exogenous variables.

Parameters:

data (pd.DataFrame)
outcome (str) – Dependent variable.
endogenous (list of str) – Endogenous regressors.
instruments (list of str) – Excluded instruments.
exogenous (list of str, optional) – Included exogenous regressors (added automatically to Z).
cluster (str, optional) – Cluster variable for cluster-robust SE.
robust (bool) – Use heteroskedasticity-robust SE (default True).
alpha (float) – Significance level.

Return type:

References

Angrist, J. D., & Pischke, J.-S. (2009). Mostly Harmless Econometrics. Princeton University Press.

Examples

>>> import random
>>> from morie.fn import _frame_core as pd
>>> r = random.Random(3)
>>> z = [r.gauss(0, 1) for _ in range(120)]
>>> u = [r.gauss(0, 1) for _ in range(120)]                 # the confounder
>>> d = [0.8 * zi + ui + r.gauss(0, 0.5) for zi, ui in zip(z, u)]
>>> y = [1.5 * di + ui + r.gauss(0, 0.5) for di, ui in zip(d, u)]
>>> fit = tsls(pd.DataFrame({"y": y, "d": d, "z": z}), "y", ["d"], ["z"])
>>> (fit.variable_names, [round(float(c), 3) for c in fit.coefficients], fit.n_obs)
(['const', 'd'], [-0.081, 1.48], 120)

morie.iv.liml(data, outcome, endogenous, instruments, exogenous=None, *, robust=True, alpha=0.05)[source]¶

Limited Information Maximum Likelihood (LIML) estimator.

More robust to weak instruments than 2SLS. The LIML estimator is the k-class estimator with $\kappa$ equal to the smallest eigenvalue of a certain matrix ratio.

\[\hat\beta_{\text{LIML}} = \bigl(X'(I - \kappa M_Z) X\bigr)^{-1} X'(I - \kappa M_Z) y\]

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)
robust (bool)
alpha (float)

Return type:

References

Anderson, T. W., & Rubin, H. (1949). Estimation of the parameters of a single equation in a complete system of stochastic equations. Annals of Mathematical Statistics, 20(1), 46–63.

morie.iv.gmm_iv(data, outcome, endogenous, instruments, exogenous=None, *, weight_matrix='optimal', robust=True, alpha=0.05)[source]¶

Generalised Method of Moments (GMM) IV estimator.

Two-step efficient GMM: first estimates with the identity weight matrix, then re-estimates with the optimal weight matrix constructed from first-step residuals.

\[\hat\beta_{\text{GMM}} = \arg\min_\beta \bigl[Z'(y - X\beta)\bigr]' \hat W \bigl[Z'(y - X\beta)\bigr]\]

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)
weight_matrix (str) – "optimal" (efficient GMM) or "identity" (one-step).
robust (bool)
alpha (float)

Return type:

morie.iv.cue_gmm(data, outcome, endogenous, instruments, exogenous=None, *, max_iter=100, tol=1e-08, alpha=0.05)[source]¶

Continuously Updated GMM estimator.

Unlike two-step GMM, CUE simultaneously optimises over $\beta$ and the weight matrix, yielding an estimator that is invariant to the normalisation of moment conditions.

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)
max_iter (int)
tol (float)
alpha (float)

Return type:

morie.iv.wald_estimator(data, outcome, treatment, instrument, *, alpha=0.05)[source]¶

Wald (simple IV) estimator for a single binary instrument.

\[\hat\beta_{\text{Wald}} = \frac{E[Y | Z=1] - E[Y | Z=0]}{E[D | Z=1] - E[D | Z=0]}\]

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str) – Endogenous treatment variable.
instrument (str) – Binary instrument.
alpha (float)

Return type:

morie.iv.first_stage_diagnostics(data, endogenous, instruments, exogenous=None)[source]¶

First-stage regression diagnostics for each endogenous variable.

Reports the F-statistic, partial R-squared, and Shea’s partial R-squared for each endogenous variable.

Parameters:

data (pd.DataFrame)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)

Returns:

Columns: endogenous_var, f_stat, partial_r2, shea_partial_r2, n_obs.

Return type:

pd.DataFrame

References

Shea, J. (1997). Instrument relevance in multivariate linear models: A simple measure. Review of Economics and Statistics, 79(2), 348–352.

morie.iv.cragg_donald_test(data, endogenous, instruments, exogenous=None)[source]¶

Cragg-Donald (1993) minimum eigenvalue statistic for weak instruments.

Parameters:

data (pd.DataFrame)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)

Returns:

The statistic is the Cragg-Donald F-statistic (minimum eigenvalue of the concentration parameter matrix).

Return type:

morie.iv.stock_yogo_critical_values(n_endogenous=1, n_instruments=1)[source]¶

Stock-Yogo (2005) critical values for weak instrument testing.

Returns approximate critical values for the Cragg-Donald statistic at various maximal bias/size thresholds.

Parameters:

n_endogenous (int)
n_instruments (int)

Returns:

Critical values keyed by bias threshold (e.g., "10%_bias").

Return type:

References

Stock, J. H., & Yogo, M. (2005). Testing for weak instruments in linear IV regression.

morie.iv.kleibergen_paap_test(data, endogenous, instruments, exogenous=None, vcov='robust', cluster=None)[source]¶

Kleibergen-Paap (2006) rk statistic and its Wald F for weak instruments.

Theta = chol(Qzz) Pi chol(Qxx)^-1 from the partialled first stage; the rk statistic n lambda’ V_lambda^-1 lambda projects vec(Theta) on the smallest-singular-value directions (A_perp, B_perp of the SVD), with V_lambda the robust, iid or clustered covariance of vec(Theta). The statistic is the rk Wald F, rk/n (n - L)/K (clustered: rk/(n-1) (n - L)(G-1)/G/K), compared with the rule of thumb 10 rather than the Stock-Yogo tables, which assume iid errors. Matches the R arm’s morie_iv_kleibergen_paap() to 1e-14.

References

Kleibergen, F., & Paap, R. (2006). Generalized reduced rank tests using the singular value decomposition. Journal of Econometrics, 133(1), 97–126.

Parameters:

data (DataFrame)
endogenous (list[str])
instruments (list[str])
exogenous (list[str] | None)
vcov (str)
cluster (str | None)

Return type:

DiagnosticResult

morie.iv.montiel_olea_pflueger_test(data, outcome, endogenous, instruments, exogenous=None, vcov='robust', cluster=None, lag=None, alpha=0.05, tau=(0.05, 0.1, 0.2, 0.3))[source]¶

Montiel Olea-Pflueger (2013) effective F with computed critical values.

F_eff = x’ P_Z x / tr(W2) on the partialled, orthonormalised instruments, robust to heteroskedasticity, clustering and serial correlation (vcov “robust”, “iid”, “cluster” or “hac”). Critical values come from the Patnaik approximation for the simplified test (x = 1/tau) and the TSLS and LIML tests (x = B/tau, B the worst-case Nagar bias maximised over beta), computed from the data, not a table. critical_values maps “tau_10” etc. to the TSLS critical value; details holds every column. Matches the R arm’s morie_iv_montiel_olea_pflueger().

References

Montiel Olea, J. L., & Pflueger, C. (2013). A robust test for weak instruments. Journal of Business & Economic Statistics, 31(3), 358–369.

Parameters:

data (DataFrame)
outcome (str)
endogenous (str)
instruments (list[str])
exogenous (list[str] | None)
vcov (str)
cluster (str | None)
lag (int | None)
alpha (float)
tau (tuple)

Return type:

DiagnosticResult

morie.iv.anderson_rubin_test(data, outcome, endogenous, instruments, exogenous=None, *, beta0=None, alpha=0.05)[source]¶

Anderson-Rubin (1949) test: robust to weak instruments.

Tests $H_0: \beta = \beta_0$ using the statistic:

\[AR = \frac{(y - X\beta_0)'P_Z(y - X\beta_0) / q} {(y - X\beta_0)'M_Z(y - X\beta_0) / (n - k)}\]

which has an $F(q, n-k)$ distribution under the null regardless of instrument strength.

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)
beta0 (float, optional) – Null hypothesis value (default 0).
alpha (float)

Returns:

details contains the confidence set if computed by inversion.

Return type:

morie.iv.anderson_rubin_ci(data, outcome, endogenous, instruments, exogenous=None, *, grid_min=-10.0, grid_max=10.0, grid_n=200, alpha=0.05)[source]¶

Anderson-Rubin confidence interval by test inversion.

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (str)
instruments (list of str)
exogenous (list of str, optional)
grid_min (float)
grid_max (float)
grid_n (int)
alpha (float)

Returns:

(lower, upper) bounds of the AR confidence set.

Return type:

tuple of float

morie.iv.conditional_lr_test(data, outcome, endogenous, instruments, exogenous=None, *, beta0=0.0)[source]¶

Conditional likelihood ratio test (Moreira, 2003).

More powerful than Anderson-Rubin when instruments are weak but not completely irrelevant.

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)
beta0 (float)

Return type:

References

Moreira, M. J. (2003). A conditional likelihood ratio test for structural models. Econometrica, 71(4), 1027–1048.

morie.iv.sargan_test(data, outcome, endogenous, instruments, exogenous=None)[source]¶

Sargan (1958) / Hansen (1982) J-test for overidentifying restrictions.

Tests whether the excluded instruments are uncorrelated with the structural error. Requires more instruments than endogenous variables (overidentification).

\[J = n \cdot \bar g' \hat S^{-1} \bar g \sim \chi^2(L - K)\]

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)

Return type:

morie.iv.hansen_j_test(data, outcome, endogenous, instruments, exogenous=None)[source]¶

Hansen J-test (heteroskedasticity-robust overidentification test).

Uses the efficient GMM objective function as the test statistic.

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)

Return type:

morie.iv.hausman_test(data, outcome, endogenous, instruments, exogenous=None)[source]¶

Hausman (1978) specification test: OLS vs 2SLS.

Tests whether the endogenous variables are in fact exogenous. Under the null, OLS is consistent and efficient; under the alternative, only 2SLS is consistent.

\[H = (\hat\beta_{2SLS} - \hat\beta_{OLS})' (V_{2SLS} - V_{OLS})^{-1} (\hat\beta_{2SLS} - \hat\beta_{OLS})\]

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)

Return type:

References

Hausman, J. A. (1978). Specification tests in econometrics. Econometrica, 46(6), 1251–1271.

morie.iv.durbin_wu_hausman(data, outcome, endogenous, instruments, exogenous=None)[source]¶

Durbin-Wu-Hausman regression-based endogeneity test.

Augments the structural equation with first-stage residuals and tests their joint significance.

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)

Return type:

morie.iv.jive(data, outcome, endogenous, instruments, exogenous=None, *, alpha=0.05)[source]¶

Jackknife IV estimator (JIVE).

Uses leave-one-out predicted values from the first stage to reduce finite-sample bias from many or weak instruments.

\[\hat D_{i,(-i)} = Z_i' \hat\Pi_{(-i)}\]

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)
alpha (float)

Return type:

References

Angrist, J. D., Imbens, G. W., & Krueger, A. B. (1999). Jackknife instrumental variables estimation. Journal of Applied Econometrics, 14(1), 57–67.

morie.iv.split_sample_iv(data, outcome, endogenous, instruments, exogenous=None, *, split_fraction=0.5, seed=42, alpha=0.05)[source]¶

Split-sample IV estimator (Angrist & Krueger, 1995).

Randomly splits the sample in two halves. Estimates the first stage on one half and the second stage on the other, reducing overfitting bias from many instruments.

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)
split_fraction (float)
seed (int)
alpha (float)

Return type:

morie.iv.control_function(data, outcome, endogenous, instruments, exogenous=None, *, robust=True, alpha=0.05)[source]¶

Control function (CF) approach to endogeneity.

Two-step: (1) regress endogenous on instruments and exogenous to get residuals; (2) include residuals in the structural equation as additional regressors. Numerically equivalent to 2SLS for linear models but extends naturally to nonlinear models.

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)
robust (bool)
alpha (float)

Returns:

details includes the test of endogeneity (significance of first-stage residuals).

Return type:

morie.iv.iv_probit(data, outcome, endogenous, instruments, exogenous=None, *, alpha=0.05)[source]¶

IV Probit via control function (Rivers & Vuong, 1988).

For binary outcomes. Uses the control-function approach: first-stage OLS residuals are included in a probit second stage.

Parameters:

data (pd.DataFrame)
outcome (str) – Binary outcome (0/1).
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)
alpha (float)

Return type:

References

Rivers, D., & Vuong, Q. H. (1988). Limited information estimators and exogeneity tests for simultaneous probit models. Journal of Econometrics, 39(3), 347–366.

morie.iv.panel_iv(data, outcome, endogenous, instruments, unit, *, exogenous=None, time_fe=None, alpha=0.05)[source]¶

Panel IV with unit (and optionally time) fixed effects.

Demeans all variables by unit (and time if specified), then applies 2SLS to the demeaned data.

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
unit (str) – Unit identifier for fixed effects.
exogenous (list of str, optional)
time_fe (str, optional) – Time column for two-way fixed effects.
alpha (float)

Return type:

morie.iv.iv_diagnostics(data, outcome, endogenous, instruments, exogenous=None)[source]¶

Comprehensive IV regression diagnostic suite.

Runs first-stage diagnostics, weak instrument tests, overidentification tests, and endogeneity tests in one call.

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)

Returns:

Keys: first_stage, cragg_donald, hausman, sargan (if overidentified), durbin_wu_hausman.

Return type:

morie.iv.iv_residual_analysis(data, outcome, endogenous, instruments, exogenous=None)[source]¶

Residual analysis for IV regression.

Returns a DataFrame with fitted values, residuals, leverage, and standardised residuals.

Parameters:

data (pd.DataFrame)
outcome (str)
endogenous (list of str)
instruments (list of str)
exogenous (list of str, optional)

Return type:

pd.DataFrame

Regression Discontinuity Design estimators for causal inference.

Implements sharp and fuzzy RDD, optimal bandwidth selection (IK, CCT, rule-of-thumb), local polynomial regression, kernel functions, manipulation tests, covariate balance, placebo tests, donut-hole RDD, geographic/multi-dimensional RDD, RD plots, kink RD, sensitivity analysis, and power calculations.

References

Imbens, G. W., & Kalyanaraman, K. (2012). Optimal bandwidth choice for the regression discontinuity estimator. Review of Economic Studies, 79(3), 933–959. https://doi.org/10.1093/restud/rdr043

Calonico, S., Cattaneo, M. D., & Titiunik, R. (2014). Robust nonparametric confidence intervals for regression-discontinuity designs. Econometrica, 82(6), 2295–2326. https://doi.org/10.3982/ECTA11757

McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142(2), 698–714. https://doi.org/10.1016/j.jeconom.2007.05.005

Cattaneo, M. D., Jansson, M., & Ma, X. (2020). Simple local polynomial density estimators. Journal of the American Statistical Association, 115(531), 1449–1455. https://doi.org/10.1080/01621459.2019.1635480

morie.rdd.kernel_triangular(u)[source]¶

Triangular kernel: $K(u) = (1 - |u|)$ for $|u| \le 1$.

Parameters:: u (marr)
Return type:: marr

morie.rdd.kernel_epanechnikov(u)[source]¶

Epanechnikov kernel: $K(u) = \frac{3}{4}(1 - u^2)$ for $|u| \le 1$.

Parameters:: u (marr)
Return type:: marr

morie.rdd.kernel_uniform(u)[source]¶

Uniform (rectangular) kernel: $K(u) = 0.5$ for $|u| \le 1$.

Parameters:: u (marr)
Return type:: marr

morie.rdd.kernel_gaussian(u)[source]¶

Gaussian kernel: $K(u) = \phi(u)$ (standard normal density).

Parameters:: u (marr)
Return type:: marr

class morie.rdd.RDDResult(estimate, std_error, t_stat, p_value, ci_lower, ci_upper, bandwidth, n_left, n_right, method='sharp_rdd', details=<factory>)[source]¶

Bases: object

Container for an RDD treatment effect estimate.

Variables:

estimate (float) – Point estimate of the treatment effect at the cutoff.
std_error (float) – Standard error.
t_stat (float) – t-statistic.
p_value (float) – Two-sided p-value.
ci_lower (float) – Lower bound of confidence interval.
ci_upper (float) – Upper bound of confidence interval.
bandwidth (float) – Bandwidth used for estimation.
n_left (int) – Observations to the left of the cutoff within the bandwidth.
n_right (int) – Observations to the right of the cutoff within the bandwidth.
method (str) – Estimation method name.
details (dict) – Additional diagnostic information.

Parameters:

estimate (float)
std_error (float)
t_stat (float)
p_value (float)
ci_lower (float)
ci_upper (float)
bandwidth (float)
n_left (int)
n_right (int)
method (str)
details (dict[str, Any])

estimate: float¶

std_error: float¶

t_stat: float¶

p_value: float¶

ci_lower: float¶

ci_upper: float¶

bandwidth: float¶

n_left: int¶

n_right: int¶

method: str = 'sharp_rdd'¶

details: dict[str, Any]¶

summary()[source]¶

Return a one-row summary DataFrame.

Return type:: DataFrame

class morie.rdd.BandwidthResult(h_opt, method, details=<factory>)[source]¶

Bases: object

Container for bandwidth selection output.

Variables:

h_opt (float) – Optimal bandwidth.
method (str) – Selection method used.
details (dict) – Intermediate quantities.

Parameters:

h_opt (float)
method (str)
details (dict[str, Any])

h_opt: float¶

method: str¶

details: dict[str, Any]¶

class morie.rdd.DensityTestResult(statistic, p_value, method, details=<factory>)[source]¶

Bases: object

Container for manipulation/density test output.

Parameters:

statistic (float)
p_value (float)
method (str)
details (dict[str, Any])

statistic: float¶

p_value: float¶

method: str¶

details: dict[str, Any]¶

morie.rdd.local_polynomial_regression(x, y, eval_points, h, p=1, kernel='triangular')[source]¶

Evaluate a local polynomial regression at multiple points.

Parameters:

x (np.ndarray) – Running variable.
y (np.ndarray) – Outcome.
eval_points (np.ndarray) – Points at which to evaluate the regression.
h (float) – Bandwidth.
p (int) – Polynomial order (0=Nadaraya-Watson, 1=local linear, etc.).
kernel (str) – Kernel name.

Returns:

Columns: x, fitted, se.

Return type:

pd.DataFrame

morie.rdd.bandwidth_ik(x, y, cutoff=0.0, kernel='triangular')[source]¶

Imbens-Kalyanaraman MSE-optimal bandwidth.

The three-step plug-in of IK (2012): a pilot window at the cutoff for the density and conditional variance, a global cubic for the third derivative, one-sided quadratics for the second derivatives, and the regularised MSE-optimal rule with the kernel constant.

Parameters:

x (np.ndarray)
y (np.ndarray)
cutoff (float)
kernel (str)

Return type:

BandwidthResult

References

Imbens, G. and Kalyanaraman, K. (2012). Optimal bandwidth choice for: the regression discontinuity estimator. Review of Economic Studies, 79(3), 933-959.

morie.rdd.bandwidth_rot(x, y, cutoff=0.0)[source]¶

Rule-of-thumb bandwidth selector.

Uses Silverman’s rule adapted for the RDD context.

Parameters:

x (np.ndarray)
y (np.ndarray)
cutoff (float)

Return type:

BandwidthResult

morie.rdd.bandwidth_cct(x, y, cutoff=0.0, kernel='triangular', p=1, deriv=0, treatment=None, vce='nn')[source]¶

MSE-optimal RD bandwidths of Calonico, Cattaneo & Farrell (2020), as rdrobust::rdbwselect(bwselect = "mserd"): h for the order-p estimate of the deriv-th derivative jump (fuzzy with treatment) and b for its bias correction. The CER-optimal h (bwselect = "cerrd") is h * N^(-p/((3+p)(3+2p))).

Parameters:

x (np.ndarray)
y (np.ndarray)
cutoff (float)
kernel (str)
p (int) – Polynomial order.
deriv (int) – Derivative of the regression function (1 for a kink).
treatment (np.ndarray, optional) – Treatment received, for fuzzy designs.
vce (str) – "nn" (default) or "hc0"-"hc3".

Returns:

h_opt is the MSE-optimal h; details has h_mse, b_mse and h_cer.

Return type:

BandwidthResult

References

Calonico, S., Cattaneo, M. D., & Titiunik, R. (2014). Robust nonparametric confidence intervals for regression-discontinuity designs. Econometrica, 82(6), 2295–2326.

Calonico, S., Cattaneo, M. D., & Farrell, M. H. (2020). Optimal bandwidth choice for robust bias-corrected inference in regression discontinuity designs. Econometrics Journal, 23(2), 192–210.

morie.rdd.sharp_rdd(data, outcome, running, cutoff=0.0, *, bandwidth=None, p=1, kernel='triangular', cluster=None, covariates=None, alpha=0.05, vce='nn')[source]¶

Sharp regression discontinuity design estimator.

Estimates the local average treatment effect at the cutoff using local polynomial regression of order p separately on each side of the cutoff:

\[\hat\tau = \hat\mu_+(c) - \hat\mu_-(c)\]

If no bandwidth is provided, the CCT MSE-optimal bandwidth is used.

Parameters:

data (pd.DataFrame)
outcome (str) – Outcome column.
running (str) – Running variable column.
cutoff (float) – RD cutoff value.
bandwidth (float, optional) – Estimation bandwidth. If None, computed via bandwidth_cct().
p (int) – Local polynomial order (default 1 = local linear).
kernel (str) – Kernel function name.
cluster (str, optional) – Column for cluster-robust variance.
covariates (list of str, optional) – Additional covariates (included additively).
alpha (float) – Significance level.
vce (str)

Return type:

References

Hahn, J., Todd, P., & Van der Klaauw, W. (2001). Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica, 69(1), 201–209.

morie.rdd.fuzzy_rdd(data, outcome, running, treatment, cutoff=0.0, *, bandwidth=None, p=1, kernel='triangular', alpha=0.05, vce='nn')[source]¶

Fuzzy regression discontinuity design estimator.

Uses the cutoff as an instrument for treatment takeup. The fuzzy RD estimand is:

\[\hat\tau_{\text{FRD}} = \frac{\hat\mu_{Y+}(c) - \hat\mu_{Y-}(c)} {\hat\mu_{D+}(c) - \hat\mu_{D-}(c)}\]

Parameters:

data (pd.DataFrame)
outcome (str) – Outcome column.
running (str) – Running variable.
treatment (str) – Treatment takeup column (binary).
cutoff (float) – RD cutoff.
bandwidth (float, optional)
p (int) – Polynomial order.
kernel (str)
alpha (float)
vce (str)

Returns:

details includes the first-stage jump.

Return type:

morie.rdd.rdd_bias_corrected(data, outcome, running, cutoff=0.0, *, bandwidth=None, rho=None, p=1, kernel='triangular', alpha=0.05, vce='nn')[source]¶

Bias-corrected RDD with robust confidence intervals (CCT, 2014).

Uses a higher-order local polynomial to estimate and remove the leading bias term, then constructs robust CIs.

Parameters:

data (pd.DataFrame)
outcome (str)
running (str)
cutoff (float)
bandwidth (float, optional) – Main bandwidth. If None, uses CCT MSE-optimal.
rho (float, optional) – Ratio h/b. By default b is the MSE-optimal bias bandwidth when no bandwidth is given, and b = h when one is (as rdrobust).
p (int) – Main polynomial order.
kernel (str)
alpha (float)
vce ({"nn", "hc"}) – Residual variance: nearest-neighbour (rdrobust’s default) or plug-in residuals of the local fits.

Returns:

The estimate is bias-corrected; ci_lower/ci_upper are the robust confidence intervals.

Return type:

morie.rdd.mccrary_test(x, cutoff=0.0, *, n_bins=50, bandwidth=None)[source]¶

McCrary (2008) density test for manipulation of the running variable.

Constructs a histogram of the running variable and tests for a discontinuity in the density at the cutoff using local linear regression on the bin counts.

Parameters:

x (np.ndarray) – Running variable.
cutoff (float) – RD cutoff.
n_bins (int) – Number of histogram bins on each side.
bandwidth (float, optional)

Returns:

statistic is the log-difference in estimated densities.

Return type:

DensityTestResult

References

McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142(2), 698–714.

morie.rdd.cattaneo_density_test(x, cutoff=0.0, *, p=2, kernel='triangular', bandwidth=None)[source]¶

Cattaneo, Jansson, & Ma (2020) density test for manipulation.

The local polynomial density estimator of the empirical CDF on each side of the cutoff (unrestricted fit, order q = p + 1 for the robust test), with the jackknife variance, as rddensity::rddensity(x, c, p, h): the statistic is its t_jk.

Parameters:

x (np.ndarray)
cutoff (float)
p (int) – Polynomial order for density estimation.
kernel (str)
bandwidth (float, optional) – Common bandwidth; by default rddensity’s data-driven comb bandwidths from rdbwdensity.

Return type:

DensityTestResult

References

Cattaneo, M. D., Jansson, M., & Ma, X. (2020). Simple local polynomial density estimators. JASA, 115(531), 1449–1455.

morie.rdd.covariate_balance_rdd(data, running, covariates, cutoff=0.0, *, bandwidth=None, kernel='triangular', alpha=0.05)[source]¶

Test covariate balance at the RD cutoff.

For each covariate, estimates a sharp RDD and reports whether there is a statistically significant jump at the cutoff.

Parameters:

data (pd.DataFrame)
running (str)
covariates (list of str)
cutoff (float)
bandwidth (float, optional)
kernel (str)
alpha (float)

Returns:

Columns: covariate, estimate, std_error, p_value, balanced.

Return type:

pd.DataFrame

morie.rdd.placebo_cutoff_test(data, outcome, running, true_cutoff, placebo_cutoffs, *, bandwidth=None, p=1, kernel='triangular', alpha=0.05)[source]¶

Test for RDD effects at placebo (false) cutoff values.

Estimates a sharp RDD at each placebo cutoff using only data on one side of the true cutoff. A well-designed RDD should show no significant effects at placebo cutoffs.

Parameters:

data (pd.DataFrame)
outcome (str)
running (str)
true_cutoff (float)
placebo_cutoffs (list of float)
bandwidth (float, optional)
p (int)
kernel (str)
alpha (float)

Return type:

pd.DataFrame

morie.rdd.donut_rdd(data, outcome, running, cutoff=0.0, donut=0.0, *, bandwidth=None, p=1, kernel='triangular', alpha=0.05)[source]¶

Donut-hole RDD: exclude observations within donut of the cutoff.

Addresses concerns about manipulation or heaping at the cutoff by removing observations in $[c - \text{donut}, c + \text{donut}]$.

Parameters:

data (pd.DataFrame)
outcome (str)
running (str)
cutoff (float)
donut (float) – Exclusion radius around the cutoff.
bandwidth (float, optional)
p (int)
kernel (str)
alpha (float)

Return type:

morie.rdd.rdd_discrete(data, outcome, running, cutoff=0.0, *, bandwidth=None, p=0, alpha=0.05)[source]¶

RDD estimator for discrete running variables.

Uses local constant (Nadaraya-Watson) or local linear regression with a uniform kernel. Standard errors are clustered at the level of the running variable mass points (Lee & Card, 2008).

Parameters:

data (pd.DataFrame)
outcome (str)
running (str)
cutoff (float)
bandwidth (float, optional)
p (int) – Polynomial order (default 0 = local constant).
alpha (float)

Return type:

References

Lee, D. S., & Card, D. (2008). Regression discontinuity inference with specification error. Journal of Econometrics, 142(2), 655–674.

morie.rdd.rd_plot_data(data, outcome, running, cutoff=0.0, *, n_bins=20, p_global=4, p_local=1, bandwidth=None, kernel='triangular')[source]¶

Generate data for standard RD plots.

Returns three datasets: - binned: binned scatter plot (evenly-spaced or quantile bins) - global_poly: global polynomial fit - local_poly: local polynomial fit within bandwidth

Parameters:

data (pd.DataFrame)
outcome (str)
running (str)
cutoff (float)
n_bins (int) – Number of bins on each side.
p_global (int) – Order of the global polynomial.
p_local (int) – Order of the local polynomial.
bandwidth (float, optional)
kernel (str)

Returns:

Keys: "binned", "global_poly", "local_poly".

Return type:

morie.rdd.bandwidth_sensitivity(data, outcome, running, cutoff=0.0, *, bandwidth_range=None, p=1, kernel='triangular', alpha=0.05)[source]¶

Sensitivity of the RDD estimate to bandwidth choice.

Parameters:

data (pd.DataFrame)
outcome (str)
running (str)
cutoff (float)
bandwidth_range (list of float, optional) – Bandwidths to evaluate. If None, uses multiples of the optimal bandwidth.
p (int)
kernel (str)
alpha (float)

Returns:

Columns: bandwidth, estimate, std_error, ci_lower, ci_upper, p_value, n_eff.

Return type:

pd.DataFrame

morie.rdd.kink_rdd(data, outcome, running, cutoff=0.0, *, bandwidth=None, kernel='triangular', alpha=0.05, vce='nn')[source]¶

Regression Kink Design (RKD) estimator.

Tests for a change in the slope (rather than the level) of the conditional expectation function at the cutoff.

\[\hat\tau_{\text{RKD}} = \frac{\hat\mu'_+(c) - \hat\mu'_-(c)} {\text{kink in policy variable}}\]

Parameters:

data (pd.DataFrame)
outcome (str)
running (str)
cutoff (float)
bandwidth (float, optional)
kernel (str)
alpha (float)
vce (str)

Returns:

estimate is the change in slope at the cutoff.

Return type:

References

Card, D., Lee, D. S., Pei, Z., & Weber, A. (2015). Inference on causal effects in a generalized regression kink design. Econometrica, 83(6), 2453–2483.

morie.rdd.rdd_local_randomisation(data, outcome, running, cutoff=0.0, window=1.0, *, n_permutations=1000, seed=42, alpha=0.05)[source]¶

RDD via local randomisation (Cattaneo, Frandsen & Titiunik, 2015).

Within a narrow window around the cutoff, assumes treatment is as-if randomly assigned and uses a permutation test.

Parameters:

data (pd.DataFrame)
outcome (str)
running (str)
cutoff (float)
window (float) – Symmetric window around cutoff for local randomisation.
n_permutations (int)
seed (int)
alpha (float)

Returns:

p_value is the permutation-based p-value.

Return type:

References

Cattaneo, M. D., Frandsen, B. R., & Titiunik, R. (2015). Randomization inference in the regression discontinuity design. Journal of Causal Inference, 3(1), 1–24.

morie.rdd.geographic_rdd(data, outcome, distance_to_boundary, side, *, bandwidth=None, p=1, kernel='triangular', alpha=0.05)[source]¶

Geographic (multi-dimensional) RDD using distance to a boundary.

Reduces a spatial RDD to a univariate RDD by using the signed distance from each unit to the treatment boundary as the running variable.

Parameters:

data (pd.DataFrame)
outcome (str)
distance_to_boundary (str) – Column with signed distance (negative on one side, positive on the other).
side (str) – Binary column indicating which side of the boundary each unit falls on.
bandwidth (float, optional)
p (int)
kernel (str)
alpha (float)

Return type:

References

Keele, L. J., & Titiunik, R. (2015). Geographic boundaries as regression discontinuities. Political Analysis, 23(1), 127–155.

morie.rdd.rdd_power(n, tau, sigma, cutoff_density=1.0, bandwidth=None, kernel='triangular', alpha=0.05)[source]¶

Power calculation for a sharp RDD.

Computes approximate power based on the asymptotic distribution of the local linear estimator:

\[\text{Power} = \Phi\left(\frac{|\tau|}{\hat\sigma_\tau} - z_{1-\alpha/2}\right)\]

Parameters:

n (int) – Total sample size.
tau (float) – Hypothesised treatment effect.
sigma (float) – Outcome standard deviation.
cutoff_density (float) – Density of the running variable at the cutoff.
bandwidth (float, optional) – Bandwidth (if None, uses a default fraction of the range).
kernel (str) – Kernel name (affects efficiency constants).
alpha (float) – Significance level.

Returns:

power, n_effective, mde (minimum detectable effect).

Return type:

morie.rdd.rdd_sample_size(tau, sigma, cutoff_density=1.0, bandwidth=1.0, power=0.8, kernel='triangular', alpha=0.05)[source]¶

Compute the required sample size for a sharp RDD.

Parameters:

tau (float) – Target treatment effect.
sigma (float) – Outcome SD.
cutoff_density (float)
bandwidth (float)
power (float) – Target power (default 0.80).
kernel (str)
alpha (float)

Returns:

Required total sample size.

Return type:

Difference-in-Differences estimators for causal inference.

Implements classic 2x2 DiD, generalized DiD with multiple time periods (Callaway–Sant’Anna), staggered adoption, event studies, Bacon decomposition, doubly-robust DiD, triple differences, synthetic DiD, and comprehensive pre-trend and placebo testing infrastructure.

References

Callaway, B., & Sant’Anna, P. H. C. (2021). Difference-in-Differences with multiple time periods. Journal of Econometrics, 225(2), 200–230. https://doi.org/10.1016/j.jeconom.2020.12.001

Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254–277. https://doi.org/10.1016/j.jeconom.2021.03.014

Sant’Anna, P. H. C., & Zhao, J. (2020). Doubly robust difference-in-differences estimators. Journal of Econometrics, 219(1), 101–122. https://doi.org/10.1016/j.jeconom.2020.06.003

Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2021). Synthetic difference-in-differences. American Economic Review, 111(12), 4088–4118. https://doi.org/10.1257/aer.20190159

class morie.did.DiDResult(estimate, std_error, t_stat, p_value, ci_lower, ci_upper, n_treated, n_control, method='did_2x2', details=<factory>)[source]¶

Bases: object

Container for a Difference-in-Differences estimate.

Variables:

estimate (float) – Point estimate of the treatment effect (ATT).
std_error (float) – Standard error of the estimate.
t_stat (float) – t-statistic (estimate / std_error).
p_value (float) – Two-sided p-value.
ci_lower (float) – Lower bound of the 95 % confidence interval.
ci_upper (float) – Upper bound of the 95 % confidence interval.
n_treated (int) – Number of treated observations.
n_control (int) – Number of control observations.
method (str) – Name of the estimation method used.
details (dict) – Additional diagnostic information.

Parameters:

estimate (float)
std_error (float)
t_stat (float)
p_value (float)
ci_lower (float)
ci_upper (float)
n_treated (int)
n_control (int)
method (str)
details (dict[str, Any])

estimate: float¶

std_error: float¶

t_stat: float¶

p_value: float¶

ci_lower: float¶

ci_upper: float¶

n_treated: int¶

n_control: int¶

method: str = 'did_2x2'¶

details: dict[str, Any]¶

summary()[source]¶

Return a one-row summary DataFrame.

Return type:: DataFrame

class morie.did.EventStudyResult(coefficients, reference_period, pre_trend_f_stat, pre_trend_p_value, details=<factory>)[source]¶

Bases: object

Container for event-study coefficient estimates.

Variables:

coefficients (pd.DataFrame) – DataFrame with columns relative_time, estimate, std_error, ci_lower, ci_upper, p_value.
reference_period (int) – The omitted (normalised-to-zero) relative time period.
pre_trend_f_stat (float) – Joint F-statistic for pre-treatment coefficients.
pre_trend_p_value (float) – p-value for the joint pre-trend test.

Parameters:

coefficients (DataFrame)
reference_period (int)
pre_trend_f_stat (float)
pre_trend_p_value (float)
details (dict[str, Any])

coefficients: DataFrame¶

reference_period: int¶

pre_trend_f_stat: float¶

pre_trend_p_value: float¶

details: dict[str, Any]¶

class morie.did.BaconDecomposition(components, overall_estimate)[source]¶

Bases: object

Container for Goodman-Bacon decomposition.

Variables:

components (pd.DataFrame) – DataFrame with columns group1, group2, estimate, weight, type (timing, always-treated, never-treated).
overall_estimate (float) – Weighted-average TWFE DiD estimate.

Parameters:

components (DataFrame)
overall_estimate (float)

components: DataFrame¶

overall_estimate: float¶

morie.did.did_2x2(data, outcome, treatment, post, *, covariates=None, cluster=None, alpha=0.05)[source]¶

Estimate a classic 2x2 Difference-in-Differences treatment effect.

The canonical DiD estimator for settings with two groups (treated vs control) and two periods (pre vs post). With no covariates the estimator is:

\[\hat\tau_{\text{DiD}} = \bigl(\bar Y_{1,\text{post}} - \bar Y_{1,\text{pre}}\bigr) - \bigl(\bar Y_{0,\text{post}} - \bar Y_{0,\text{pre}}\bigr)\]

When covariates are provided the regression specification is:

\[Y_{it} = \alpha + \beta\,D_i + \gamma\,\text{Post}_t + \tau\,(D_i \times \text{Post}_t) + X_{it}'\delta + \varepsilon_{it}\]

and $\hat\tau$ is the coefficient on the interaction term.

Parameters:

data (pd.DataFrame) – Panel or repeated cross-section data.
outcome (str) – Name of the outcome column.
treatment (str) – Binary (0/1) column indicating the treatment group.
post (str) – Binary (0/1) column indicating the post-treatment period.
covariates (list of str, optional) – Additional covariates for the regression specification.
cluster (str, optional) – Column for cluster-robust standard errors.
alpha (float) – Significance level for confidence intervals (default 0.05).

Returns:

Estimation results including point estimate, SE, CI, and diagnostics.

Return type:

References

Angrist, J. D., & Pischke, J.-S. (2009). Mostly Harmless Econometrics. Princeton University Press.

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({
...     "y": [1.0, 1.2, 1.1, 1.3, 2.0, 2.4, 2.1, 2.6, 1.5, 1.6, 1.4, 1.7, 2.6, 2.9, 2.7, 3.1],
...     "treated": [0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1],
...     "post": [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1],
... })
>>> r = did_2x2(df, "y", "treated", "post")
>>> (round(r.estimate, 4), r.n_treated, r.n_control)   # (2.825 - 2.275) - (1.55 - 1.15)
(0.15, 8, 8)

morie.did.did_repeated_cross_section(data, outcome, treatment, post, *, covariates=None, weights=None, cluster=None, alpha=0.05)[source]¶

DiD estimator designed for repeated cross-section data.

Identical to did_2x2() but optionally incorporates survey weights. When weights is provided, weighted least squares is used for estimation.

Parameters:

data (pd.DataFrame) – Repeated cross-section data.
outcome (str) – Column names for outcome, treatment indicator, and post indicator.
treatment (str) – Column names for outcome, treatment indicator, and post indicator.
post (str) – Column names for outcome, treatment indicator, and post indicator.
covariates (list of str, optional) – Additional covariates.
weights (str, optional) – Column containing survey / sampling weights.
cluster (str, optional) – Column for cluster-robust SE.
alpha (float) – Significance level.

Return type:

morie.did.did_panel_fe(data, outcome, treatment, unit, time, *, covariates=None, cluster=None, alpha=0.05)[source]¶

DiD with unit and time fixed effects (two-way fixed effects, TWFE).

Demeans the outcome by unit and time means (within transformation) and regresses on the treatment indicator.

Parameters:

data (pd.DataFrame) – Balanced or unbalanced panel data.
outcome (str) – Outcome column.
treatment (str) – Binary column (0/1) indicating treatment status in each period.
unit (str) – Unit identifier column.
time (str) – Time period column.
covariates (list of str, optional) – Time-varying covariates.
cluster (str, optional) – Cluster variable for standard errors (defaults to unit).
alpha (float) – Significance level.

Return type:

morie.did.event_study(data, outcome, unit, time, treatment_time, *, covariates=None, reference_period=-1, leads=4, lags=4, cluster=None, alpha=0.05)[source]¶

Estimate an event-study specification around treatment onset.

Constructs relative-time dummies $\{1[t - g = k]\}$ for $k \in [-\text{leads}, \text{lags}]$ and regresses the outcome on these indicators with unit and time fixed effects. The reference_period dummy is omitted (normalised to zero).

Parameters:

data (pd.DataFrame) – Panel data with a treatment-onset column.
outcome (str) – Outcome column.
unit (str) – Unit identifier column.
time (str) – Calendar-time column (integer-valued).
treatment_time (str) – Column giving the period in which each unit first received treatment (np.inf or NaN for never-treated units).
covariates (list of str, optional) – Time-varying covariates.
reference_period (int) – Relative-time period omitted as baseline (default -1).
leads (int) – Number of pre-treatment periods to include.
lags (int) – Number of post-treatment periods to include.
cluster (str, optional) – Cluster variable (defaults to unit).
alpha (float) – Significance level.

Return type:

EventStudyResult

morie.did.test_parallel_trends(data, outcome, treatment, time, *, unit=None, cluster=None, pre_periods=None)[source]¶

Test the parallel trends assumption using pre-treatment data.

Runs two tests:

Individual coefficient test: regresses the outcome on group-by-time interactions in the pre-period and reports each coefficient.
Joint F-test: tests that all interaction coefficients are jointly zero.

Parameters:

data (pd.DataFrame) – Panel or repeated cross-section data.
outcome (str) – Outcome column name.
treatment (str) – Treatment group indicator.
time (str) – Time column (integer-valued).
unit (str, optional) – Unit identifier for panel demeaning.
cluster (str, optional) – Cluster variable for robust SE.
pre_periods (list, optional) – Explicit list of pre-treatment time values to test.

Returns:

coefficients (pd.DataFrame), joint_f_stat, joint_p_value, parallel_trends_plausible (bool, True when p > 0.05).

Return type:

morie.did.parallel_trends_data(data, outcome, treatment, time, *, weights=None)[source]¶

Compute group-level outcome means over time for parallel-trends plots.

Parameters:

data (pd.DataFrame) – Panel or repeated cross-section.
outcome (str) – Column names.
treatment (str) – Column names.
time (str) – Column names.
weights (str, optional) – Survey weight column.

Returns:

Columns: time, group (0 or 1), mean_outcome, se, n.

Return type:

pd.DataFrame

morie.did.group_time_att(data, outcome, unit, time, treatment_time, *, covariates=None, method='doubly_robust', control_group='never_treated', n_bootstrap=200, seed=42, alpha=0.05, se_convention='reference')[source]¶

Estimate group-time average treatment effects (Callaway & Sant’Anna 2021).

For each cohort $g$ (units first treated at time $g$) and each post-treatment calendar period $t$, estimates $\text{ATT}(g, t)$.

Parameters:

data (pd.DataFrame) – Panel data.
outcome (str) – Outcome column.
unit (str) – Unit identifier.
time (str) – Calendar-time column (integer).
treatment_time (str) – Column with treatment-onset period (use np.inf for never-treated).
covariates (list of str, optional) – Covariates for doubly-robust estimation.
method (str) – "doubly_robust" (default), "ipw", or "outcome_regression".
control_group (str) – "never_treated" or "not_yet_treated".
n_bootstrap (int) – Mammen multiplier-bootstrap replications (0 = analytic influence-function standard errors).
seed (int) – Random seed for the multiplier bootstrap.
alpha (float) – Significance level.
se_convention (str) – Analytic SE: "reference" (did’s sqrt(mean(IF^2)/n)) or "bessel" (sd(IF)/sqrt(n)).

Returns:

Columns: cohort, time, att, std_error, ci_lower, ci_upper, p_value, n_treated, post; attrs["did_fit"] holds the influence functions used by aggregate_gt_att().

Return type:

pd.DataFrame

Notes

Base period is g - 1 after treatment and t - 1 before it; each cell is DRDID’s panel estimator (dr / reg / ipw) with the covariates of the base period. Point estimates and analytic SEs equal did::att_gt(..., bstrap = FALSE).

References

Callaway, B., & Sant’Anna, P. H. C. (2021). Difference-in-Differences with multiple time periods. Journal of Econometrics, 225(2), 200–230.

morie.did.aggregate_gt_att(gt_results, *, aggregation='overall', time_col='time', cohort_col='cohort', att_col='att', se_col='std_error')[source]¶

Aggregate group-time ATTs into summary treatment effect parameters.

On the unmodified output of group_time_att() the aggregation uses its influence functions and equals did::aggte (simple, group, dynamic, calendar), estimated-weight term included; any other table is combined as independent cell estimates.

Parameters:

gt_results (pd.DataFrame) – Output of group_time_att().
aggregation (str) – "overall" (single ATT), "cohort" (by cohort), "calendar_time" (by period), or "event_time" (by relative period since treatment).
time_col (str) – Column name overrides.
cohort_col (str) – Column name overrides.
att_col (str) – Column name overrides.
se_col (str) – Column name overrides.

Returns:

Aggregated estimates with estimate, std_error, ci_lower, ci_upper.

Return type:

pd.DataFrame

morie.did.staggered_did(data, outcome, unit, time, treatment_time, *, covariates=None, n_bootstrap=200, seed=42, alpha=0.05)[source]¶

Staggered DiD via group-time ATTs with aggregation.

A convenience wrapper around group_time_att() and aggregate_gt_att() that returns overall, cohort-level, and event-time aggregated results.

Parameters:

data (pd.DataFrame) – Panel data.
outcome (str) – Column names.
unit (str) – Column names.
time (str) – Column names.
treatment_time (str) – Column names.
covariates (list of str, optional) – Covariates for doubly-robust estimation.
n_bootstrap (int) – Bootstrap replications.
seed (int) – Random seed.
alpha (float) – Significance level.

Returns:

Keys: group_time (pd.DataFrame), overall (pd.DataFrame), by_cohort (pd.DataFrame), by_event_time (pd.DataFrame).

Return type:

morie.did.did_doubly_robust(data, outcome, treatment, post, covariates, *, ps_model='logistic', or_model='linear', cluster=None, n_bootstrap=200, seed=42, alpha=0.05)[source]¶

Doubly-robust DiD estimator (Sant’Anna & Zhao, 2020).

Combines an outcome regression model with an inverse-probability weighting model. The estimator is consistent if either model is correctly specified.

\[\hat\tau_{\text{DR}} = \frac{1}{n_1}\sum_{i:D_i=1}\bigl[\Delta Y_i - \hat\mu_0(\mathbf X_i)\bigr] - \frac{1}{\sum_{j:D_j=0}\hat w_j} \sum_{j:D_j=0}\hat w_j\bigl[\Delta Y_j - \hat\mu_0(\mathbf X_j)\bigr]\]

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
post (str)
covariates (list of str)
ps_model (str) – "logistic" or "gbm" for propensity score model.
or_model (str) – "linear" or "gbm" for outcome regression model.
cluster (str, optional)
n_bootstrap (int)
seed (int)
alpha (float)

Return type:

References

Sant’Anna, P. H. C., & Zhao, J. (2020). Doubly robust difference-in-differences estimators. Journal of Econometrics, 219(1), 101–122.

morie.did.did_triple_difference(data, outcome, treatment, post, third_diff, *, covariates=None, cluster=None, alpha=0.05)[source]¶

Triple-difference (DDD) estimator.

Extends the standard DiD by adding a third differencing dimension (e.g., within-state variation across affected and unaffected sub-populations).

\[Y = \alpha + \beta_1 D + \beta_2 \text{Post} + \beta_3 S + \beta_4 (D \times \text{Post}) + \beta_5 (D \times S) + \beta_6 (\text{Post} \times S) + \tau (D \times \text{Post} \times S) + \varepsilon\]

The DDD estimate is the coefficient on the three-way interaction.

Parameters:

data (pd.DataFrame)
outcome (str) – Column names. third_diff is the binary variable for the additional differencing group (e.g., age group, sub-population).
treatment (str) – Column names. third_diff is the binary variable for the additional differencing group (e.g., age group, sub-population).
post (str) – Column names. third_diff is the binary variable for the additional differencing group (e.g., age group, sub-population).
third_diff (str) – Column names. third_diff is the binary variable for the additional differencing group (e.g., age group, sub-population).
covariates (list of str, optional)
cluster (str, optional)
alpha (float)

Return type:

morie.did.bacon_decomposition(data, outcome, treatment, unit, time)[source]¶

Goodman-Bacon (2021) decomposition of the TWFE DiD estimator.

Decomposes the two-way fixed-effects DiD estimate into a weighted average of all possible 2x2 DiD comparisons: earlier-vs-later treated, treated-vs-never-treated, and later-vs-earlier treated.

Parameters:

data (pd.DataFrame) – Balanced panel data.
outcome (str) – Outcome column.
treatment (str) – Binary treatment indicator that turns on at treatment onset.
unit (str) – Unit identifier.
time (str) – Time period.

Return type:

BaconDecomposition

References

Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254–277.

morie.did.synthetic_did(data, outcome, unit, time, treatment_time, *, treated_units=None, zeta=None, n_bootstrap=200, seed=42, alpha=0.05, se_method='bootstrap')[source]¶

Synthetic difference-in-differences (Arkhangelsky, Athey, Hirshberg, Imbens & Wager 2021), computed as synthdid::synthdid_estimate: unit and time weights by Frank-Wolfe with a sparsify pass, zeta_omega = (N1 T1)^(1/4) sigma and zeta_lambda = 1e-6 sigma, sigma the sd of the controls’ pre-period first differences. Block design: every treated unit starts at the first treatment time.

Parameters:

data (pd.DataFrame) – Long panel.
outcome (str)
unit (str)
time (str)
treatment_time (str) – Column of first treatment time (NaN or 0 for never treated).
treated_units (list, optional)
zeta (float, optional) – Overrides zeta_omega.
n_bootstrap (int) – Replications for the bootstrap and placebo standard errors.
seed (int)
alpha (float)
se_method ({"bootstrap", "jackknife", "placebo"}) – As synthdid::vcov: the jackknife holds the weights fixed; the bootstrap and placebo re-solve them from the renormalised full-sample weights.

Return type:

morie.did.wild_cluster_bootstrap(data, outcome, treatment, post, cluster, *, covariates=None, n_bootstrap=999, weight_type='rademacher', seed=42, alpha=0.05)[source]¶

DiD with wild cluster bootstrap p-values (Cameron, Gelbach & Miller, 2008).

Recommended when the number of clusters is small (< 50). Multiplies cluster-level residuals by random weights drawn from the Rademacher or Webb distributions.

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
post (str)
cluster (str)
covariates (list of str, optional)
n_bootstrap (int) – Number of bootstrap replications.
weight_type (str) – "rademacher" (+1/-1 with equal probability) or "webb" (six-point distribution).
seed (int)
alpha (float)

Returns:

The p_value is the bootstrap p-value.

Return type:

References

Cameron, A. C., Gelbach, J. B., & Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. Review of Economics and Statistics, 90(3), 414–427.

morie.did.did_continuous_treatment(data, outcome, dose, post, *, covariates=None, cluster=None, alpha=0.05)[source]¶

DiD with a continuous treatment variable (dose–response DiD).

Models the outcome as:

\[Y_{it} = \alpha + \beta\,\text{Dose}_i + \gamma\,\text{Post}_t + \tau\,(\text{Dose}_i \times \text{Post}_t) + X_{it}'\delta + \varepsilon_{it}\]

$\hat\tau$ is the marginal effect of a one-unit increase in treatment intensity in the post period.

Parameters:

data (pd.DataFrame)
outcome (str)
dose (str) – Continuous treatment intensity column.
post (str) – Binary post-period indicator.
covariates (list of str, optional)
cluster (str, optional)
alpha (float)

Return type:

morie.did.did_fuzzy(data, outcome, assignment, takeup, post, *, covariates=None, cluster=None, alpha=0.05)[source]¶

Fuzzy DiD estimator for settings with imperfect compliance.

Uses the interaction of assignment $\times$ post as an instrument for takeup $\times$ post in a 2SLS framework to recover a local average treatment effect (LATE).

Parameters:

data (pd.DataFrame)
outcome (str)
assignment (str) – Intent-to-treat assignment indicator (binary).
takeup (str) – Actual treatment takeup indicator (binary).
post (str) – Post-period indicator.
covariates (list of str, optional)
cluster (str, optional)
alpha (float)

Returns:

The estimate is the LATE from fuzzy DiD.

Return type:

morie.did.placebo_test_time(data, outcome, treatment, time, true_treatment_time, placebo_times, *, covariates=None, cluster=None, alpha=0.05)[source]¶

Run placebo DiD tests at fake treatment times.

For each time in placebo_times, re-define the post indicator as time >= placebo_time using only pre-treatment data (before true_treatment_time) and estimate a DiD. A well-identified design should yield estimates close to zero.

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
time (str)
true_treatment_time (any) – The actual treatment onset time (to restrict to pre-period data).
placebo_times (list) – Candidate fake treatment times to test.
covariates (list of str, optional)
cluster (str, optional)
alpha (float)

Returns:

One row per placebo time with placebo_time, estimate, std_error, p_value, significant.

Return type:

pd.DataFrame

morie.did.placebo_test_outcome(data, placebo_outcomes, treatment, post, *, covariates=None, cluster=None, alpha=0.05)[source]¶

Run placebo DiD on outcomes that should not be affected by treatment.

Parameters:

data (pd.DataFrame)
placebo_outcomes (list of str) – Outcome columns expected to show no treatment effect.
treatment (str)
post (str)
covariates (list of str, optional)
cluster (str, optional)
alpha (float)

Returns:

One row per placebo outcome.

Return type:

pd.DataFrame

morie.did.placebo_test_group(data, outcome, treatment, post, group_col, unaffected_groups, *, covariates=None, cluster=None, alpha=0.05)[source]¶

Run placebo DiD on groups that should not be affected.

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
post (str)
group_col (str) – Column defining sub-groups.
unaffected_groups (list) – Group values where no treatment effect is expected.
covariates (list of str, optional)
cluster (str, optional)
alpha (float)

Return type:

pd.DataFrame

morie.did.did_heterogeneous(data, outcome, treatment, post, moderator, *, covariates=None, cluster=None, n_quantiles=4, alpha=0.05)[source]¶

Heterogeneity-robust DiD: estimate treatment effects by subgroups.

Splits the sample by quantiles (or categories) of moderator and estimates separate DiD effects for each stratum.

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
post (str)
moderator (str)
covariates (list of str, optional)
cluster (str, optional)
n_quantiles (int) – Number of quantile bins if moderator is continuous.
alpha (float)

Returns:

Columns: group, estimate, std_error, ci_lower, ci_upper, p_value, n.

Return type:

pd.DataFrame

morie.did.did_chaisemartin_dhaultfoeuille(data, outcome, treatment, unit, time, *, n_bootstrap=200, seed=42, alpha=0.05)[source]¶

Heterogeneity-robust DiD (de Chaisemartin & D’Haultfoeuille, 2020).

Computes the instantaneous treatment effect for switchers–units whose treatment status changes–using appropriate comparisons.

\[\hat\delta = \sum_{(i,t): D_{it}>D_{i,t-1}} w_{it} \bigl[\Delta Y_{it} - \Delta \bar Y_{ct}\bigr]\]

Parameters:

data (pd.DataFrame) – Panel data.
outcome (str)
treatment (str)
unit (str)
time (str)
n_bootstrap (int)
seed (int)
alpha (float)

Return type:

References

de Chaisemartin, C., & D’Haultfoeuille, X. (2020). Two-way fixed effects estimators with heterogeneous treatment effects. American Economic Review, 110(9), 2964–2996.

morie.did.did_sensitivity_analysis(data, outcome, treatment, post, *, covariates=None, delta_range=None, cluster=None, alpha=0.05)[source]¶

Sensitivity of DiD estimate to violations of parallel trends.

Bounds the bias from a parallel-trends violation by $\delta$ standard errors of the 2x2 estimate (a fixed-bias band in the spirit of Rambachan & Roth 2023; a 2x2 design has no pre-periods to calibrate their relative-magnitudes set, for which see honest_sensitivity()):

\[|\text{bias}| \le \delta \cdot \hat\sigma\]

For each $\delta$, computes a bias-adjusted confidence set.

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
post (str)
covariates (list of str, optional)
delta_range (sequence of float, optional) – Values of $\delta$ to evaluate (default: 0 to 2 in 0.25 steps).
cluster (str, optional)
alpha (float)

Returns:

Columns: delta, ci_lower, ci_upper, covers_zero.

Return type:

pd.DataFrame

References

Rambachan, A., & Roth, J. (2023). A more credible approach to parallel trends. Review of Economic Studies, 90(5), 2555–2591.

morie.did.honest_sensitivity(event_study_result, *, m_bar_range=(0.0, 0.5, 1.0, 1.5, 2.0), target_time=0, alpha=0.05)[source]¶

Rambachan-Roth (2023) relative-magnitudes sensitivity for an event study.

Under $\Delta^{RM}(\bar M)$ every post-period change in the trend violation is at most $\bar M$ times the largest change between consecutive pre-periods (the reference period counts as zero), so the violation at the target lies within $k \bar M m$ of the last pre-period, $k$ steps later. id_lower / id_upper are that identified set at the estimated coefficients (equal to HonestDiD’s .compute_IDset_DeltaRM); the interval widens it by $z$ standard errors of the target, a plug-in band that treats the pre-period changes as known (not HonestDiD’s conditional / hybrid ARP interval). Mirrors morie_did_honest_sensitivity in the R arm.

Parameters:

event_study_result (EventStudyResult) – Output of event_study().
m_bar_range (sequence of float) – Values of $\bar M$.
target_time (int) – Post-treatment relative time to bound (default 0, onset).
alpha (float) – Significance level.

Returns:

Columns m_bar, estimate, id_lower, id_upper, ci_lower, ci_upper, covers_zero; attrs holds breakdown_m_bar and max_pre_deviation.

Return type:

pd.DataFrame

References

Rambachan, A., & Roth, J. (2023). A more credible approach to parallel trends. Review of Economic Studies, 90(5), 2555–2591.

morie.did.did_diagnostics(data, outcome, treatment, post, *, covariates=None, cluster=None)[source]¶

Comprehensive diagnostics for a 2x2 DiD setting.

Checks: - Sample sizes by group and period - Baseline covariate balance - Outcome distributions by group and period - Pre-period outcome correlation between groups

Parameters:

data (pd.DataFrame)
outcome (str)
treatment (str)
post (str)
covariates (list of str, optional)
cluster (str, optional)

Returns:

Keys: sample_sizes, outcome_stats, covariate_balance.

Return type:

Sensitivity analysis for causal inference assumptions.

Provides tools to assess the robustness of causal effect estimates to unmeasured confounding, model specification, and other threats to internal validity. Implements Rosenbaum bounds, E-value, Ding-VanderWeele bias formulas, tipping-point analysis, and specification curve analysis.

References

Rosenbaum (2002). Observational Studies, 2nd ed.
VanderWeele & Ding (2017). Sensitivity analysis in observational research.
Cinelli & Hazlett (2020). Making sense of sensitivity.

class morie.sensitivity.EValueResult(point_estimate, e_value_point, e_value_ci, rr, ci_lower, ci_upper, interpretation)[source]¶

Bases: object

Result from E-value computation.

Parameters:

point_estimate (float)
e_value_point (float)
e_value_ci (float)
rr (float)
ci_lower (float)
ci_upper (float)
interpretation (str)

point_estimate: float¶

e_value_point: float¶

e_value_ci: float¶

rr: float¶

ci_lower: float¶

ci_upper: float¶

interpretation: str¶

class morie.sensitivity.RosenbaumBounds(gamma_values, p_upper, p_lower, critical_gamma, method, interpretation)[source]¶

Bases: object

Result from Rosenbaum sensitivity analysis.

Parameters:

gamma_values (marr)
p_upper (marr)
p_lower (marr)
critical_gamma (float)
method (str)
interpretation (str)

gamma_values: marr¶

p_upper: marr¶

p_lower: marr¶

critical_gamma: float¶

method: str¶

interpretation: str¶

class morie.sensitivity.TippingPointResult(delta_values, adjusted_estimates, adjusted_p_values, tipping_point, original_estimate, interpretation)[source]¶

Bases: object

Result from tipping-point analysis.

Parameters:

delta_values (marr)
adjusted_estimates (marr)
adjusted_p_values (marr)
tipping_point (float)
original_estimate (float)
interpretation (str)

delta_values: marr¶

adjusted_estimates: marr¶

adjusted_p_values: marr¶

tipping_point: float¶

original_estimate: float¶

interpretation: str¶

class morie.sensitivity.OmittedVariableBias(estimate, se, rv_q, rv_qa, partial_r2_treatment, benchmark_bounds, interpretation)[source]¶

Bases: object

Result from omitted variable bias analysis (Cinelli & Hazlett).

Parameters:

estimate (float)
se (float)
rv_q (float)
rv_qa (float)
partial_r2_treatment (float)
benchmark_bounds (dict[str, tuple[float, float]])
interpretation (str)

estimate: float¶

se: float¶

rv_q: float¶

rv_qa: float¶

partial_r2_treatment: float¶

benchmark_bounds: dict[str, tuple[float, float]]¶

interpretation: str¶

class morie.sensitivity.SpecificationCurveResult(estimates, ses, p_values, specifications, median_estimate, iqr_lower, iqr_upper, pct_significant, pct_same_sign)[source]¶

Bases: object

Result from specification curve analysis.

Parameters:

estimates (marr)
ses (marr)
p_values (marr)
specifications (list[dict])
median_estimate (float)
iqr_lower (float)
iqr_upper (float)
pct_significant (float)
pct_same_sign (float)

estimates: marr¶

ses: marr¶

p_values: marr¶

specifications: list[dict]¶

median_estimate: float¶

iqr_lower: float¶

iqr_upper: float¶

pct_significant: float¶

pct_same_sign: float¶

morie.sensitivity.e_value_rr(rr, ci_lower=None, ci_upper=None)[source]¶

Compute E-value for a risk ratio.

The E-value is the minimum strength of association (on the risk ratio scale) that an unmeasured confounder would need to have with both treatment and outcome to fully explain away the observed association.

Parameters:

rr (float) – Observed risk ratio.
ci_lower (float, optional) – Lower bound of 95% CI for the RR.
ci_upper (float, optional) – Upper bound of 95% CI for the RR.

Return type:

morie.sensitivity.e_value_or(odds_ratio, ci_lower=None, ci_upper=None, prevalence=None)[source]¶

Compute E-value for an odds ratio.

For rare outcomes (prevalence < 15%, or prevalence not given) the OR approximates the RR. For common outcomes the OR is converted with $RR \approx \sqrt{OR}$ (VanderWeele & Ding 2017), as EValue::evalues.OR(rare = FALSE).

Parameters:

odds_ratio (float) – Observed odds ratio.
ci_lower (float, optional) – Lower CI bound.
ci_upper (float, optional) – Upper CI bound.
prevalence (float, optional) – Outcome prevalence; 0.15 or more selects the common-outcome conversion.

Return type:

morie.sensitivity.e_value_hr(hr, ci_lower=None, ci_upper=None, rare=False)[source]¶

Compute E-value for a hazard ratio.

For a common outcome the HR is converted with VanderWeele’s (2017) approximation $RR = (1 - 0.5^{\sqrt{HR}}) / (1 - 0.5^{\sqrt{1/HR}})$; for a rare outcome (rare=True) the HR approximates the RR, as EValue::evalues.HR.

Parameters:

hr (float) – Observed hazard ratio.
ci_lower (float, optional) – Lower CI bound.
ci_upper (float, optional) – Upper CI bound.
rare (bool) – Outcome rare (< 15%) at the end of follow-up.

Return type:

morie.sensitivity.e_value_d(d, se=None, n=None)[source]¶

Compute E-value for a standardized mean difference (Cohen’s d).

Converts d to a risk ratio with the VanderWeele & Ding (2017) approximation $RR \approx \exp(0.91 d)$, CI $\exp(0.91 d \pm 1.78\,s)$, as EValue::evalues.MD.

Parameters:

d (float) – Standardized mean difference.
se (float, optional) – Standard error of d.
n (int, optional) – Sample size (used to compute SE if not provided).

Return type:

morie.sensitivity.rosenbaum_bounds(treated_outcomes, control_outcomes, gamma_range=None, method='wilcoxon')[source]¶

Rosenbaum sensitivity analysis for matched pair designs.

Tests how strong hidden bias (Gamma) would need to be to alter the study’s conclusions.

Parameters:

treated_outcomes (array-like) – Outcomes for treated units in matched pairs.
control_outcomes (array-like) – Outcomes for control units in matched pairs.
gamma_range (array-like, optional) – Range of Gamma values to test. Default: 1.0 to 5.0 by 0.25.
method (str) – Test method: ‘wilcoxon’ (default), ‘sign’, ‘mcnemar’.

Return type:

RosenbaumBounds

morie.sensitivity.tipping_point_analysis(estimate, se, n_treated, n_control, delta_range=None, outcome_type='continuous')[source]¶

Tipping-point analysis for missing data sensitivity.

Evaluates how much the treatment effect changes if missing outcomes are systematically different from observed outcomes, shifting the estimate by $\delta$. The tipping point is the smallest shift that makes the two-sided 5% test non-significant, $\delta^* = \hat\tau - \mathrm{sign}(\hat\tau)\, z_{0.975}\, se$ (0 when the estimate is already non-significant); the grid reports the adjusted estimates and p-values over delta_range.

Parameters:

estimate (float) – Observed treatment effect estimate.
se (float) – Standard error of the estimate.
n_treated (int) – Number of treated units.
n_control (int) – Number of control units.
delta_range (array-like, optional) – Range of bias parameters to evaluate.
outcome_type (str) – ‘continuous’ or ‘binary’.

Return type:

TippingPointResult

morie.sensitivity.omitted_variable_bias(estimate, se, dof, r2_yd_x, partial_r2_treatment, q=1.0, alpha=0.05, benchmark_covariates=None, kd=1.0, ky=None)[source]¶

Omitted-variable-bias sensitivity (Cinelli & Hazlett 2020).

Robustness values as sensemakr::robustness_value: with $f_q = q |t| / \sqrt{dof}$, $RV_q = \tfrac12(\sqrt{f_q^4 + 4 f_q^2} - f_q^2)$, and $RV_{q,\alpha}$ the same in $f_q - f^*$ with $f^* = |t^*_{\alpha, dof-1}| / \sqrt{dof - 1}$ (the extreme robustness value when $f_q > 1/f^*$). Benchmark bounds follow sensemakr::ovb_bounds: a confounder kd / ky times as strong as the benchmark covariate.

Parameters:

estimate (float) – Treatment coefficient estimate.
se (float) – Standard error of the estimate.
dof (int) – Residual degrees of freedom.
r2_yd_x (float) – Partial R-squared of treatment with outcome (controlling for X).
partial_r2_treatment (float) – Same as r2_yd_x (for clarity).
q (float) – Fraction of the estimate to be explained away. Default 1.0 (full).
alpha (float) – Significance level.
benchmark_covariates (dict, optional) – Covariate name to (r2_dxj_x, r2_yxj_dx): its partial R-squared with the treatment (given the other covariates) and with the outcome (given treatment and the other covariates). A single number is used for both.
kd (float) – Strength multipliers of the confounder relative to a benchmark (ky defaults to kd).
ky (float) – Strength multipliers of the confounder relative to a benchmark (ky defaults to kd).

Returns:

benchmark_bounds maps each covariate to a dict with r2dz_x, r2yz_dx, adjusted_estimate, adjusted_se, adjusted_lower_ci and adjusted_upper_ci.

Return type:

OmittedVariableBias

morie.sensitivity.specification_curve(data, outcome, treatment, covariate_sets, sample_filters=None, model_types=None, alpha=0.05)[source]¶

Run a specification curve analysis.

Estimates the treatment effect across many reasonable model specifications to assess robustness.

Parameters:

data (pd.DataFrame) – Analysis dataset.
outcome (str) – Outcome variable name.
treatment (str) – Treatment variable name.
covariate_sets (list of list[str]) – Different sets of covariates to try.
sample_filters (list of (name, callable), optional) – Different sample restrictions to try.
model_types (list[str], optional) – Model types: ‘ols’, ‘logistic’, ‘robust’. Default: [‘ols’].
alpha (float) – Significance level.

Return type:

SpecificationCurveResult

morie.sensitivity.manski_bounds(outcome_treated, outcome_control, p_treated, outcome_range=None)[source]¶

Compute Manski worst-case bounds for the ATE.

Under no assumptions about selection, the ATE is only partially identified (Manski 1990): with the outcome in $[a, b]$, $E[Y_1] \in [p\bar y_1 + a(1-p),\; p\bar y_1 + b(1-p)]$ and $E[Y_0] \in [(1-p)\bar y_0 + a p,\; (1-p)\bar y_0 + b p]$, so the bounds always have width $b - a$.

Parameters:

outcome_treated (array-like) – Outcomes for treated units.
outcome_control (array-like) – Outcomes for control units.
p_treated (float) – Proportion treated.
outcome_range (tuple[float, float], optional) – Logical bounds on the outcome. Default: (0, 1) for binary.

Returns:

With keys: lower_bound, upper_bound, point_estimate, width.

Return type:

morie.sensitivity.bias_adjusted_estimate(estimate, se, rr_ud, rr_eu, prevalence_confounder=None)[source]¶

Compute bias-adjusted treatment effect on the log-RR scale.

Without prevalence_confounder the adjustment is the Ding & VanderWeele (2016) bounding factor $B = RR_{UD} RR_{EU} / (RR_{UD} + RR_{EU} - 1)$, the largest bias any confounder with those strengths can produce, applied toward the null. With the prevalence $p_0$ of a binary confounder among the unexposed, rr_eu is its prevalence ratio ($p_1 = RR_{EU} p_0$) and the bias is Schlesselman’s (1978) exact factor $(1 + (RR_{UD}-1)p_1) / (1 + (RR_{UD}-1)p_0)$.

Parameters:

estimate (float) – Observed treatment effect (log-RR scale).
se (float) – Standard error.
rr_ud (float) – Risk ratio relating confounder to outcome.
rr_eu (float) – Risk ratio relating treatment to confounder.
prevalence_confounder (float, optional) – Prevalence of the confounder among the unexposed.

Returns:

With keys: adjusted_estimate, bias, adjusted_ci_lower, adjusted_ci_upper, original_estimate (bias on the log scale).

Return type:

morie.sensitivity.probabilistic_bias_analysis(estimate, se, n_simulations=10000, bias_parms=None, seed=42)[source]¶

Probabilistic (Monte Carlo) sensitivity analysis.

Draws bias parameters from specified prior distributions and the estimate from its sampling distribution, and removes Schlesselman’s binary-confounder bias $\log[(1 + (RR_{UD}-1)p_1) / (1 + (RR_{UD}-1)p_0)]$ with $p_0$ the drawn prevalence among the unexposed and $p_1 = \min(RR_{EU} p_0, 1)$ (Lash, Fox & Fink 2009, ch. 8).

Parameters:

estimate (float) – Observed estimate (log-RR scale).
se (float) – Standard error.
n_simulations (int) – Number of Monte Carlo draws.
bias_parms (dict, optional) – Parameters as (mean, sd) tuples for: ‘rr_ud’, ‘rr_eu’, ‘prevalence’.
seed (int) – Random seed.

Returns:

Summary of bias-adjusted estimate distribution.

Return type:

morie.sensitivity.konfound(estimate, se, n, n_covariates=0, alpha=0.05)[source]¶

Robustness of an inference to replacement and confounding (Frank et al.).

Native konfound::pkonfound for a regression coefficient (two tails, null 0): the threshold is $t^* se$ with $t^*$ on $n - n_{cov} - 2$ df; the percent bias to invalidate is $100 (1 - t^* se / \hat\beta)$ (to sustain when not significant) and RIR the corresponding number of cases; the impact threshold of a confounding variable is $(r - r^*) / (1 \pm |r^*|)$ with $r = t / \sqrt{t^2 + df}$. Mirrors morie_sensitivity_konfound in the R arm.

Parameters:

estimate (float) – Coefficient and its standard error.
se (float) – Coefficient and its standard error.
n (int) – Number of observations.
n_covariates (int) – Number of covariates besides the focal predictor.
alpha (float) – Significance level.

Returns:

percent_bias_to_invalidate, rir (cases to replace), impact_threshold_confounder, beta_threshold.

Return type:

morie.sensitivity.tipping_point_smd(effect_observed, smd)[source]¶

Confounder-outcome effect that tips a ratio estimate to the null.

Native tipr::tip: an unmeasured confounder whose standardized mean difference between exposure groups is smd tips an observed ratio $b$ to 1 when its effect on the outcome is $b^{1/smd}$ (Lin, Psaty & Kronmal 1998). Mirrors morie_sensitivity_tipping_point in the R arm.

Parameters:

effect_observed (float) – Observed ratio (e.g. risk ratio), the bound nearest the null.
smd (float) – Exposure-confounder standardized mean difference.

Returns:

effect_adjusted (1), confounder_outcome_effect.

Return type:

morie.sensitivity.sensitivity_summary(estimate, se, rr=None, odds_ratio=None, hazard_ratio=None, prevalence=None)[source]¶

Generate a comprehensive sensitivity analysis summary.

Computes E-values, tipping points, and robustness metrics for a single treatment effect estimate.

Parameters:

estimate (float) – Treatment effect estimate.
se (float) – Standard error.
rr (float, optional) – Risk ratio (if applicable).
odds_ratio (float, optional) – Odds ratio (if applicable).
hazard_ratio (float, optional) – Hazard ratio (if applicable).
prevalence (float, optional) – Outcome prevalence.

Returns:

Summary table of sensitivity metrics.

Return type:

pd.DataFrame

Survey + descriptive statistics¶

Survey-weighted estimation wrappers. Parallels the survey and srvyr packages in R.

This module provides:

SurveyDesign – encapsulates survey data with weights and strata.
Horvitz-Thompson and Hájek estimators for population totals and means.
Ratio estimator for auxiliary-variable-assisted estimation.
Post-stratification and calibration (raking) weight adjustments.
Domain/subpopulation estimation with design-based standard errors.
Complex-survey GLM with weights, clustering, and stratification.

Statistical assumptions and caveats¶

All functions assume the supplied weights are probability/analytic weights (i.e., w_i = 1 / pi_i where pi_i is the first-order inclusion probability). They are NOT frequency or expansion weights.
Standard errors under complex sampling are approximated by Taylor linearisation or the jackknife/bootstrap where noted.
For production survey analysis, validate against the R survey package (Lumley, 2010) or Stata’s svy prefix.

References

Lumley, T. (2010). Complex Surveys: A Guide to Analysis Using R. Wiley. Cochran, W. G. (1977). Sampling Techniques (3rd ed.). Wiley. Horvitz, D. G., & Thompson, D. J. (1952). A generalisation of sampling without

replacement from a finite universe. JASA, 47(260), 663-685.

Hájek, J. (1971). Comment on “An essay on the logical foundations of survey: sampling.” In Godambe & Sprott (Eds.), Foundations of Statistical Inference. Holt, Rinehart and Winston.

class morie.survey.SurveyDesign(data, weights_col, strata_col=None, cluster_col=None, fpc_col=None, nest=False)[source]¶

Bases: object

A one-stage survey design: sampling weights, optional strata, primary sampling units (clusters) and a finite-population correction – the design of rmorie’s morie_survey_design(); the variance of every estimator on it is the Taylor linearisation of survey::svyrecvar.

Parameters:

data (DataFrame)
weights_col (str)
strata_col (str | None)
cluster_col (str | None)
fpc_col (str | None)
nest (bool)

__init__(data, weights_col, strata_col=None, cluster_col=None, fpc_col=None, nest=False)[source]¶

Parameters:

data (DataFrame) – The survey data.
weights_col (str) – Column of sampling weights (inverse inclusion probabilities).
strata_col (str | None) – Optional strata column.
cluster_col (str | None) – Optional PSU / cluster column.
fpc_col (str | None) – Optional finite-population correction: the population size of the stratum (number of PSUs when clustered), or the sampling fraction when every value is <= 1.
nest (bool) – If True, cluster IDs are only unique within a stratum; if False a cluster ID that appears in two strata is an error, as in survey::svydesign.

weighted_mean(variable)[source]¶

The survey-weighted (Hajek) mean of a variable.

Parameters:: variable (str)
Return type:: float

mean(variable)[source]¶

The Hajek mean and its design-based (Taylor) SE, as survey::svymean; a missing value gives NaN, as there.

Parameters:: variable (str)
Return type:: dict

svyglm(formula, family=None)[source]¶

A design-weighted GLM with the linearisation (sandwich) variance of survey::svyglm over this design’s strata, PSUs and fpc.

Parameters:

formula (str) – A formula string.
family – Family name or object, defaults to binomial.

References

Binder, D. A. (1983). On the variances of asymptotically normal estimators from complex surveys. International Statistical Review, 51, 279-292. Lumley, T. (2010). Complex Surveys: A Guide to Analysis Using R. Wiley.

morie.survey.horvitz_thompson_total(y, inclusion_probs)[source]¶

Horvitz-Thompson estimator of population total.

The HT estimator is:

\[\hat{\tau}_{HT} = \sum_{i \in s} \frac{y_i}{\pi_i}\]

where $\pi_i$ is the first-order inclusion probability for unit i.

The variance estimator is the Horvitz-Thompson one under Poisson sampling, where the pairwise inclusion probabilities are exactly $\pi_{ij} = \pi_i \pi_j$ (as survey::svytotal with pps = poisson_sampling(pi)):

\[\hat{V}(\hat{\tau}_{HT}) \approx \sum_i \frac{y_i^2}{\pi_i^2} \cdot \frac{1 - \pi_i}{1}\]

Under other without-replacement designs it ignores the pairwise inclusion probabilities the design actually has.

Parameters:

y (marr) – Observed response values for sampled units (1-D array-like).
inclusion_probs (marr) – First-order inclusion probabilities pi_i for each sampled unit (values in (0, 1]).

Returns:

dict with keys total, se, ci_lower, ci_upper.

Raises:

ValueError – If y and inclusion_probs have different lengths, or any inclusion probability is <= 0 or > 1.

Return type:

References

Horvitz, D. G., & Thompson, D. J. (1952). A generalisation of sampling: without replacement from a finite universe. JASA, 47(260), 663-685.

Cochran, W. G. (1977). Sampling Techniques (3rd ed.). Wiley. (Chapter 9A.)

morie.survey.hajek_mean(y, weights)[source]¶

Hájek estimator for population mean (ratio estimator).

The Hájek estimator normalises the HT estimator by the estimated population size, eliminating sensitivity to total weight magnitude:

\[\bar{y}_H = \frac{\sum_i w_i y_i}{\sum_i w_i}\]

where $w_i = 1 / \pi_i$ are the survey weights.

Standard error is computed via Taylor (delta-method) linearisation:

\[\hat{V}(\bar{y}_H) \approx \frac{1}{N^2} \sum_i w_i^2 (y_i - \bar{y}_H)^2 \cdot \frac{1 - \pi_i}{\pi_i}\]

For the with-replacement approximation (pi_i = w_i / N_hat) this simplifies to the weighted sample variance divided by the effective sample size.

Parameters:

y (marr) – Response values (1-D array-like).
weights (marr) – Survey weights w_i = 1/pi_i (must be > 0).

Returns:

dict with keys mean, se, ci_lower, ci_upper.

Raises:

ValueError – If y and weights have different lengths or weights <= 0.

Return type:

References

Hájek, J. (1971). Comment in Godambe & Sprott (Eds.), Foundations of: Statistical Inference. Holt, Rinehart and Winston.

Cochran, W. G. (1977). Sampling Techniques (3rd ed.). Wiley. (Section 6.13.)

Examples

>>> from morie.fn import _array_core as np
>>> r = hajek_mean(np.array([3.0, 5.0, 4.0, 6.0]), np.array([1.0, 2.0, 1.0, 2.0]))
>>> (round(r["mean"], 4), round(r["se"], 4))
(4.8333, 0.5966)

morie.survey.ratio_estimator(y, x, weights, X_population_total)[source]¶

Survey ratio estimator for a population total.

The ratio estimator exploits auxiliary information X whose population total is known:

\[\hat{Y}_R = \hat{R} \cdot X_{\text{pop}}\]

where $\hat{R} = \hat{Y}_{HT} / \hat{X}_{HT}$ is the estimated ratio of totals.

Standard error via Taylor linearisation:

\[\hat{V}(\hat{Y}_R) \approx \frac{1}{\hat{X}_{HT}^2} \sum_i w_i^2 (y_i - \hat{R} x_i)^2\]

Parameters:

y (marr) – Response variable values for sampled units (1-D array-like).
x (marr) – Auxiliary variable values for sampled units (same length as y).
weights (marr) – Survey weights w_i (must be > 0).
X_population_total (float) – Known population total of the auxiliary variable.

Returns:

dict with keys ratio, total_estimate, se, ci_lower, ci_upper.

Raises:

ValueError – If inputs have different lengths, weights <= 0, or X_population_total <= 0.

Return type:

References

Cochran, W. G. (1977). Sampling Techniques (3rd ed.). Wiley. (Section 6.4.) Lumley, T. (2010). Complex Surveys: A Guide to Analysis Using R. Wiley.

morie.survey.poststratification_weights(df, strata_col, population_counts)[source]¶

Compute post-stratification weights so the sample distribution matches the known population distribution.

For each stratum h:

\[w_i^{\text{PS}} = \frac{N_h / N}{n_h / n}\]

where $N_h$ is the known population count in stratum h, $N$ is the total population, $n_h$ is the sample count in stratum h, and $n$ is the total sample size.

Parameters:

df (DataFrame) – Input DataFrame.
strata_col (str) – Column name identifying strata.
population_counts (dict[str, int]) – Dict mapping each stratum label (as string) to its population count.

Returns:

pd.Series of post-stratification weights, indexed like df.

Raises:

ValueError – If strata_col not in df, any stratum in df is absent from population_counts, or sample stratum count is zero.

Return type:

Series

References

Lumley, T. (2010). Complex Surveys: A Guide to Analysis Using R. Wiley. (Chapter 7.) Little, R. J. A. (1993). Post-stratification: A modeler’s perspective.

JASA, 88(423), 1001-1012.

morie.survey.calibration_weights(df, aux_vars, population_totals, *, max_iter=50, tol=1e-06)[source]¶

Raking calibration to known population totals of the auxiliary variables (Deville & Sarndal 1992, raking distance).

For each auxiliary variable x_j, the calibration adjusts weights multiplicatively so that:

\[\sum_i w_i x_{ij} = T_j\]

where $T_j$ is the known population total for variable j.

IPF convergence is assessed by the maximum relative deviation across all margins at each iteration.

Parameters:

df (DataFrame) – Input DataFrame with auxiliary variable columns.
aux_vars (list[str]) – List of column names to calibrate on.
population_totals (dict[str, float]) – Dict mapping column name to known population total.
max_iter (int) – Maximum number of IPF iterations. Default 50.
tol (float) – Convergence tolerance (maximum relative deviation). Default 1e-6.

Returns:

pd.Series of calibrated weights.

Raises:

ValueError – If any aux_var is missing from df or population_totals, or initial weights are zero.

Return type:

Series

References

Deville, J.-C., & Sarndal, C.-E. (1992). Calibration estimators in survey: sampling. JASA, 87(418), 376-382.
Deming, W. E., & Stephan, F. F. (1940). On a least squares adjustment of: a sampled frequency table when the expected marginal totals are known. Annals of Mathematical Statistics, 11(4), 427-444.

morie.survey.subpopulation_estimate(df, domain_col, domain_value, outcome_col, weight_col)[source]¶

Design-based estimate for a domain (subpopulation) mean.

Treats domain membership as a 0/1 indicator and applies the Hájek estimator within the full-sample design frame (i.e., does NOT subset then reweight, which would ignore the variance contribution from domain membership uncertainty).

Domain mean:

\[\bar{y}_d = \frac{\sum_{i: d_i = 1} w_i y_i}{\sum_{i: d_i = 1} w_i}\]

SE via linearisation (Woodruff, 1971):

\[\hat{V}(\bar{y}_d) \approx \frac{1}{N_d^2} \sum_i w_i^2 z_i^2\]

where $z_i = d_i (y_i - \bar{y}_d)$ and $N_d = \sum_{i: d_i=1} w_i$.

Parameters:

df (DataFrame) – Full sample DataFrame (do NOT pre-filter to domain).
domain_col (str) – Column name identifying the domain.
domain_value – Value of domain_col that identifies the target subpopulation.
outcome_col (str) – Column name of the outcome variable.
weight_col (str) – Column name of survey weights.

Returns:

dict with keys mean, se, ci_lower, ci_upper, n_domain.

Raises:

ValueError – If required columns are missing.

Return type:

References

Woodruff, R. S. (1971). A simple method for approximating the variance of a: complicated estimate. JASA, 66(334), 411-414.

Lumley, T. (2010). Complex Surveys. Wiley. (Section 4.2.)

morie.survey.complex_survey_glm(df, formula, weight_col, *, family='gaussian', cluster_col=None, strata_col=None, nest=False)[source]¶

Fit a GLM with complex survey design (weights, optional clustering, strata).

The coefficients are the weighted IRLS fit and the standard errors the design-based linearisation variance of survey::svyglm: score totals per PSU (cluster_col, or each unit), their between-PSU variance within strata (strata_col) scaled by n_h/(n_h - 1), and t tests on (PSUs - strata) + 1 - p degrees of freedom.

Supported families (string): "gaussian", "binomial", "poisson", "gamma", "negativebinomial".

Parameters:

df (DataFrame) – Input DataFrame.
formula (str) – Patsy-compatible formula string (e.g. "y ~ x1 + x2").
weight_col (str) – Column name of analytic/probability survey weights.
family (str) – GLM family string. Default "gaussian".
cluster_col (str | None) – Column identifying clusters for robust SE. Default None.
strata_col (str | None) – Column identifying strata (informational only). Default None.
nest (bool)

Returns:

Fitted statsmodels GLM result object.

Raises:

ValueError – If required columns are missing or family is unrecognised.

Return type:

object

References

Lumley, T. (2010). Complex Surveys: A Guide to Analysis Using R. Wiley. White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator

and a direct test for heteroskedasticity. Econometrica, 48(4), 817-838.

Survey sampling, resampling, and design-weight utilities.

This module provides functions for common sampling designs used in epidemiological and public health research: simple random sampling, stratified sampling, cluster sampling, probability-proportional-to-size (PPS) sampling, bootstrap resampling, jackknife variance estimation, and design-weight computation.

All stochastic functions use numpy.random.default_rng() for reproducibility rather than legacy global-state RNGs.

References

Cochran, W. G. (1977). Sampling Techniques (3rd ed.). Wiley.

Kish, L. (1965). Survey Sampling. Wiley.

Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC.

Wolter, K. M. (2007). Introduction to Variance Estimation (2nd ed.). Springer.

Lumley, T. (2010). Complex Surveys: A Guide to Analysis Using R. Wiley.

morie.sampling.simple_random_sample(df, n, *, replace=False, seed=42)[source]¶

Draw a simple random sample (SRS) from a DataFrame.

Parameters:

df (pandas.DataFrame) – Population frame.
n (int) – Number of units to sample.
replace (bool, optional) – If True, sample with replacement (default False).
seed (int, optional) – Random seed for reproducibility (default 42).

Returns:

Sampled rows with their original index preserved.

Return type:

Raises:

ValueError – If n exceeds the number of rows when sampling without replacement.

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({"x": range(100)})
>>> sample = simple_random_sample(df, 10, seed=0)
>>> len(sample)
10

References

Cochran, W. G. (1977). Sampling Techniques (3rd ed.), Chapter 2.

morie.sampling.stratified_sample(df, strata_col, n_per_stratum, *, proportional=False, seed=42)[source]¶

Draw a stratified random sample.

Supports both fixed allocation (same n per stratum) and proportional allocation. When proportional=True, n_per_stratum is interpreted as the total desired sample size and is allocated proportionally across strata.

Parameters:

df (pandas.DataFrame) – Population frame.
strata_col (str) – Column defining strata.
n_per_stratum (int or dict[str, int]) – If proportional=False: fixed integer per stratum (or a dict mapping stratum values to per-stratum sample sizes). If proportional=True: total desired sample size (must be int).
proportional (bool, optional) – If True, allocate sample sizes proportionally to stratum sizes (default False).
seed (int, optional) – Random seed (default 42).

Returns:

Sampled rows with original index preserved.

Return type:

Raises:

ValueError – If a requested stratum size exceeds the stratum population.

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({"stratum": ["A"]*50 + ["B"]*50, "x": range(100)})
>>> sample = stratified_sample(df, "stratum", 10, seed=0)
>>> sample.groupby("stratum").size().to_dict()
{'A': 10, 'B': 10}

References

Cochran, W. G. (1977). Sampling Techniques (3rd ed.), Chapter 5.

morie.sampling.cluster_sample(df, cluster_col, n_clusters, *, seed=42)[source]¶

Two-stage cluster sampling: select clusters, then take all units within.

This implements single-stage cluster sampling (select clusters, enumerate all units within selected clusters).

Parameters:

df (pandas.DataFrame) – Population frame with a cluster identifier column.
cluster_col (str) – Column identifying clusters (e.g., PSU, school, clinic).
n_clusters (int) – Number of clusters to select.
seed (int, optional) – Random seed (default 42).

Returns:

All rows belonging to the selected clusters.

Return type:

Raises:

ValueError – If n_clusters exceeds the number of distinct clusters.

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({
...     "cluster": [1]*10 + [2]*10 + [3]*10,
...     "y": range(30),
... })
>>> sample = cluster_sample(df, "cluster", 2, seed=0)
>>> sample["cluster"].nunique()
2

References

Cochran, W. G. (1977). Sampling Techniques (3rd ed.), Chapter 9.

morie.sampling.pps_sample(df, size_col, n, *, seed=42)[source]¶

Probability-proportional-to-size (PPS) sampling.

Draws n units without replacement with inclusion probability $\pi_i = n x_i / \sum x$ (units whose value reaches 1 are taken with certainty and the rest rescaled), by randomised systematic sampling (Hartley & Rao 1962). The returned rows carry the design weight .weight = $1/\pi_i$. Negative or zero size values are excluded with a warning.

Parameters:

df (pandas.DataFrame) – Population frame.
size_col (str) – Column containing the size measure (must be positive).
n (int) – Number of units to draw.
seed (int, optional) – Random seed (default 42).

Returns:

Sampled rows.

Return type:

Raises:

ValueError – If no positive size values remain after filtering.

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({"pop": [1000, 500, 200, 100], "name": ["A","B","C","D"]})
>>> sample = pps_sample(df, "pop", 2, seed=0)
>>> len(sample)
2

References

Brewer, K. R. W., & Hanif, M. (1983). Sampling with Unequal Probabilities. Springer.

morie.sampling.bootstrap_sample(df, n_bootstrap=1000, *, statistic, seed=42)[source]¶

Bootstrap resampling for any scalar statistic.

Draws n_bootstrap resamples with replacement from df, applies statistic to each, and returns the bootstrap distribution with percentile confidence interval.

Parameters:

df (pandas.DataFrame) – Input data.
n_bootstrap (int, optional) – Number of bootstrap replicates (default 1000).
statistic (callable) – Function f(df) -> float computing the statistic of interest.
seed (int, optional) – Random seed (default 42).

Returns:

Keys: mean (float), se (float), ci_lower (float), ci_upper (float), distribution (numpy.ndarray of length n_bootstrap).

Return type:

Examples

>>> from morie.fn import _array_core as np
>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({"x": np.random.default_rng(0).normal(5, 1, 200)})
>>> result = bootstrap_sample(df, 500, statistic=lambda d: d["x"].mean(), seed=0)
>>> 4.5 < result["mean"] < 5.5
True

References

Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC.

morie.sampling.jackknife_estimate(df, *, statistic)[source]¶

Delete-1 jackknife variance estimate for a scalar statistic.

Computes the leave-one-out estimates $\hat{\theta}_{(-i)}$ and derives the jackknife variance estimate:

\[\widehat{\text{Var}}_J = \frac{n-1}{n} \sum_{i=1}^{n} \left(\hat{\theta}_{(-i)} - \bar{\theta}_{(\cdot)}\right)^2\]

Parameters:

df (pandas.DataFrame) – Input data.
statistic (callable) – Function f(df) -> float computing the statistic of interest.

Returns:

Keys: estimate (float — full-sample statistic), se (float — jackknife standard error), bias (float — jackknife bias estimate).

Return type:

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({"x": [1.0, 2.0, 3.0, 4.0, 5.0]})
>>> jk = jackknife_estimate(df, statistic=lambda d: d["x"].mean())
>>> abs(jk["estimate"] - 3.0) < 1e-10
True

References

Quenouille, M. H. (1956). Notes on bias in estimation. Biometrika, 43(3/4), 353–360.

Tukey, J. W. (1958). Bias and confidence in not-quite large samples (abstract). Annals of Mathematical Statistics, 29, 614.

morie.sampling.compute_design_weights(df, strata_col, population_sizes)[source]¶

Compute inverse-probability design weights for a stratified sample.

For each stratum h, the design weight is:

\[w_{hi} = \frac{N_h}{n_h}\]

where $N_h$ is the population size and $n_h$ is the sample size in stratum h.

Parameters:

df (pandas.DataFrame) – Sample data.
strata_col (str) – Column defining strata.
population_sizes (dict[str, int]) – Mapping of stratum value to population size.

Returns:

Design weights aligned with the DataFrame index.

Return type:

pandas.Series

Raises:

KeyError – If a stratum in df is not found in population_sizes.

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({"stratum": ["A"]*10 + ["B"]*20})
>>> w = compute_design_weights(df, "stratum", {"A": 1000, "B": 2000})
>>> float(w[df["stratum"] == "A"].iloc[0])
100.0

References

Kish, L. (1965). Survey Sampling, Chapter 2. Wiley.

morie.sampling.effective_sample_size(weights)[source]¶

Compute the Kish effective sample size.

\[n_{\text{eff}} = \frac{\left(\sum_i w_i\right)^2}{\sum_i w_i^2}\]

Parameters:: weights (numpy.ndarray or pandas.Series) – Sampling or analytic weights.
Returns:: Effective sample size.
Return type:: float

Examples

>>> effective_sample_size([1.0] * 100)
100.0
>>> ess = effective_sample_size([1.0, 1.0, 10.0])
>>> ess < 3.0
True

References

Kish, L. (1965). Survey Sampling, p. 162. Wiley.

morie.sampling.design_effect(weights)[source]¶

Approximate design effect (DEFF) from weights.

The Kish approximation to the design effect is:

\[\text{DEFF} \approx \frac{n}{n_{\text{eff}}} = \frac{n \sum_i w_i^2}{\left(\sum_i w_i\right)^2}\]

A DEFF of 1.0 indicates no inflation (equal weights); values > 1 indicate variance inflation from unequal weighting.

Parameters:: weights (numpy.ndarray or pandas.Series) – Sampling or analytic weights.
Returns:: Approximate design effect.
Return type:: float

Examples

>>> design_effect([1.0] * 100)
1.0

References

Kish, L. (1965). Survey Sampling, p. 162. Wiley.

Design and measurement tools for small or under-enumerated subpopulations.

Full parity with rmorie’s R/subpop_design.R and R/subpop_psychometrics.R. The arithmetic is implemented here rather than delegated: morie is not a wrapper.

The functions are general survey methodology. The motivating application is estimation for Indigenous populations in Canada, where three conditions usually hold together: the group is a small share of any general frame, it is reached through designs that are not simple random samples, and the frame itself identifies it imperfectly.

Three results drive the planning arithmetic.

A subgroup that is a share q of the frame needs roughly 1/q times the general sample to reach a given subgroup size. At q = 0.05, a subgroup target of 385 needs about 7,700 general interviews.
Clustering by community multiplies the requirement by 1 + (m - 1) * rho and unequal weights multiply it again; the two sources compose.
If the frame records only a fraction of true members, that fraction divides the achievable subgroup size a second time, and every rate computed on the recorded counts is biased downward (Rogan and Gladen, 1978).

morie.subpop_design.morie_deff_cluster(m, icc)[source]¶: Cluster design effect 1 + (m - 1) * icc.

morie.subpop_design.morie_neff_cluster(n, m, icc)[source]¶: Effective sample size n / deff under clustering.

morie.subpop_design.morie_sample_size_proportion(p, moe, conf=0.95, N=inf, deff=1.0, response_rate=1.0)[source]¶

Completed sample size for a proportion, with FPC, deff and non-response.

Precision first, then the population limit, then the design, then fieldwork loss.

morie.subpop_design.morie_sample_size_domain(p, moe, domain_prevalence, conf=0.95, N_domain=inf, deff=1.0, response_rate=1.0, coverage=1.0)[source]¶: How large the whole sample must be for the subgroup to be large enough.

morie.subpop_design.morie_oversample_factor(domain_prevalence, target_share)[source]¶: Multiplier on the subgroup’s selection probability, and its weight cost.

morie.subpop_design.morie_screen_design(prevalence, target_n, cost_screen=1.0, cost_interview=1.0, screen_response=1.0, interview_response=1.0)[source]¶: Expected screens, cost, and the share of budget spent screening.

morie.subpop_design.morie_alloc_optimal(N_h, S_h, n, cost_h=None)[source]¶

Cost-constrained optimal allocation, n_h propto N_h S_h / sqrt(c_h).

With equal costs this is Neyman allocation.

morie.subpop_design.morie_rake(data, margins, weights=None, max_iter=50, tol=1e-08)[source]¶

Rake weights to known population margins by iterative proportional fitting.

data is a mapping of column name to a sequence of level labels, one entry per row. margins maps a subset of those columns to a mapping of level to population total.

morie.subpop_design.morie_misclass_correct(p_obs, sensitivity, specificity, n=None, conf=0.95)[source]¶

(p_obs + spec - 1) / (sens + spec - 1), after Rogan and Gladen (1978).

When n is given, the interval propagates sampling error in p_obs only: sensitivity and specificity are treated as known constants. If they come from a validation study of finite size, their own uncertainty widens the true interval and this one is optimistic.

morie.subpop_design.morie_misclass_count(observed_count, n, sensitivity, specificity)[source]¶: Adjust an observed member count and report what the records miss.

morie.subpop_design.morie_dif_sample_size(p_reference, odds_ratio=None, p_focal=None, ratio=1.0, power=0.8, alpha=0.05)[source]¶: Two-proportion sample size; power is governed by the smaller group.

morie.subpop_design.morie_dif_delta_mh(or_mh)[source]¶

ETS delta scale -2.35 * log(or) with the A/B/C classification.

Returns a list of dicts with or_mh, delta_mh, magnitude and favours.

morie.subpop_design.morie_invariance_compare(fits, cfi_cut=0.01, rmsea_cut=0.015)[source]¶

Compare nested invariance models on chi-square, CFI and RMSEA.

fits is a sequence of mappings with model, chisq, df, cfi and rmsea, ordered from least to most constrained.

morie.subpop_design.morie_irt_theta_se(information)[source]¶: SE(theta) = 1 / sqrt(I(theta)).

morie.subpop_design.morie_irt_marginal_reliability(se, var_theta=1.0)[source]¶: 1 - mean(se^2) / var_theta, the marginal reliability.

Survival analysis toolkit for epidemiological time-to-event research.

This module provides a comprehensive set of tools for censored time-to-event data commonly encountered in cohort studies, clinical trials, and public health surveillance. Implementations are built on numpy, scipy, and pandas; optional integration with lifelines is used where available but never required.

Major components¶

Non-parametric estimators: Kaplan–Meier, Nelson–Aalen
Log-rank family tests: standard, Peto–Peto, Gehan–Wilcoxon, Tarone–Ware
Cox proportional hazards: partial likelihood estimation with Breslow and Efron tie-handling, Schoenfeld / Cox–Snell / martingale / deviance residuals
Parametric models: Exponential, Weibull, log-normal, log-logistic, Gompertz
Accelerated failure time (AFT) models
Competing risks: Cumulative incidence functions (CIF), Fine–Gray model
Discrimination: Concordance index (Harrell’s C)
RMST: Restricted mean survival time comparison

References

Collett, D. (2015). Modelling Survival Data in Medical Research (3rd ed.).: Chapman & Hall/CRC.
Kalbfleisch, J. D., & Prentice, R. L. (2002). *The Statistical Analysis of: Failure Time Data* (2nd ed.). Wiley.
Fine, J. P., & Gray, R. J. (1999). A proportional hazards model for the: subdistribution of a competing risk. JASA, 94(446), 496–509.
Klein, J. P., & Moeschberger, M. L. (2003). *Survival Analysis: Techniques: for Censored and Truncated Data* (2nd ed.). Springer.

class morie.survival.SurvivalCurve(times, survival, ci_lower, ci_upper, at_risk, events, censored, method='Kaplan-Meier', median_survival=None)[source]¶

Bases: object

Stores a non-parametric survival or cumulative-hazard estimate.

Parameters:

times (ndarray) – Ordered unique event times.
survival (ndarray) – Estimated survival probabilities (or cumulative hazard).
ci_lower (ndarray) – Lower confidence band.
ci_upper (ndarray) – Upper confidence band.
at_risk (ndarray) – Number at risk just before each event time.
events (ndarray) – Number of events at each event time.
censored (ndarray) – Number censored at each event time.
method (str) – Estimator name.
median_survival (float or None) – Median survival time (first time S(t) <= 0.5).

times: marr¶

survival: marr¶

ci_lower: marr¶

ci_upper: marr¶

at_risk: marr¶

events: marr¶

censored: marr¶

method: str = 'Kaplan-Meier'¶

median_survival: float | None = None¶

class morie.survival.LogRankResult(method, test_statistic, p_value, df, n_groups, n_total, extra=<factory>)[source]¶

Bases: object

Result container for log-rank family tests.

Parameters:

method (str)
test_statistic (float)
p_value (float)
df (int)
n_groups (int)
n_total (int)
extra (dict)

method: str¶

test_statistic: float¶

p_value: float¶

df: int¶

n_groups: int¶

n_total: int¶

extra: dict¶

class morie.survival.CoxResult(coefficients, standard_errors, hazard_ratios, z_scores, p_values, ci_lower, ci_upper, covariate_names, concordance, log_likelihood, n_events, n_observations, method='Cox PH', baseline_hazard=None, extra=<factory>)[source]¶

Bases: object

Result container for Cox PH model.

Parameters:

coefficients (marr)
standard_errors (marr)
hazard_ratios (marr)
z_scores (marr)
p_values (marr)
ci_lower (marr)
ci_upper (marr)
covariate_names (list[str])
concordance (float)
log_likelihood (float)
n_events (int)
n_observations (int)
method (str)
baseline_hazard (DataFrame | None)
extra (dict)

coefficients: marr¶

standard_errors: marr¶

hazard_ratios: marr¶

z_scores: marr¶

p_values: marr¶

ci_lower: marr¶

ci_upper: marr¶

covariate_names: list[str]¶

concordance: float¶

log_likelihood: float¶

n_events: int¶

n_observations: int¶

method: str = 'Cox PH'¶

baseline_hazard: DataFrame | None = None¶

extra: dict¶

class morie.survival.ParametricSurvivalResult(distribution, coefficients, log_likelihood, aic, bic, n_observations, n_events, extra=<factory>)[source]¶

Bases: object

Result container for parametric survival / AFT models.

Parameters:

distribution (str)
coefficients (dict)
log_likelihood (float)
aic (float)
bic (float)
n_observations (int)
n_events (int)
extra (dict)

distribution: str¶

coefficients: dict¶

log_likelihood: float¶

aic: float¶

bic: float¶

n_observations: int¶

n_events: int¶

extra: dict¶

class morie.survival.CompetingRiskResult(times, cif, ci_lower, ci_upper, event_of_interest, n_total, method='Aalen-Johansen', variance=None)[source]¶

Bases: object

Result container for competing-risk analyses.

Parameters:

times (marr)
cif (marr)
ci_lower (marr)
ci_upper (marr)
event_of_interest (int)
n_total (int)
method (str)
variance (marr | None)

times: marr¶

cif: marr¶

ci_lower: marr¶

ci_upper: marr¶

event_of_interest: int¶

n_total: int¶

method: str = 'Aalen-Johansen'¶

variance: marr | None = None¶

morie.survival.kaplan_meier(time, event, confidence=0.95, ci_method='greenwood')[source]¶

Kaplan–Meier product-limit survival estimator.

Parameters:

time (array-like) – Observed times (non-negative).
event (array-like) – Event indicator (1 = event, 0 = censored).
confidence (float, default 0.95) – Confidence level for survival bands.
ci_method (str, default "greenwood") – "greenwood" for Greenwood’s formula or "log-log" for the complementary log-log transformation.

Return type:

SurvivalCurve

References

Kaplan, E. L., & Meier, P. (1958). Nonparametric estimation from incomplete observations. JASA, 53(282), 457–481.

morie.survival.nelson_aalen(time, event, confidence=0.95)[source]¶

Nelson–Aalen cumulative hazard estimator.

Parameters:

time (array-like) – Observed times.
event (array-like) – Event indicator (1 = event, 0 = censored).
confidence (float, default 0.95) – Confidence level.

Returns:

The survival field here contains the cumulative hazard H(t), not S(t).

Return type:

SurvivalCurve

morie.survival.logrank_test(time, event, group)[source]¶

Standard log-rank test (Mantel–Haenszel).

Parameters:

time (array-like) – Observed times, event indicators, and group labels.
event (array-like) – Observed times, event indicators, and group labels.
group (array-like) – Observed times, event indicators, and group labels.

Return type:

morie.survival.peto_peto_test(time, event, group)[source]¶

Peto–Peto modification of the Gehan–Wilcoxon test.

Parameters:

time (array-like)
event (array-like)
group (array-like)

Return type:

morie.survival.gehan_wilcoxon_test(time, event, group)[source]¶

Gehan–Wilcoxon test (Breslow test).

Parameters:

time (array-like)
event (array-like)
group (array-like)

Return type:

morie.survival.tarone_ware_test(time, event, group)[source]¶

Tarone–Ware weighted log-rank test.

Parameters:

time (array-like)
event (array-like)
group (array-like)

Return type:

morie.survival.cox_ph(data, duration_col, event_col, covariate_cols, ties='efron', confidence=0.95, penalizer=0.0)[source]¶

Fit a Cox proportional hazards model.

Uses Newton–Raphson optimisation of the partial likelihood.

Parameters:

data (DataFrame) – Analysis dataset.
duration_col (str) – Column with observed times.
event_col (str) – Column with event indicators (1 = event).
covariate_cols (list[str]) – Names of covariate columns.
ties (str, default "efron") – Tie handling: "efron" or "breslow".
confidence (float, default 0.95) – Confidence level for hazard ratio CIs.
penalizer (float, default 0.0) – L2 regularisation strength.

Return type:

CoxResult

morie.survival.schoenfeld_residuals(data, duration_col, event_col, covariate_cols, cox_result)[source]¶

Compute Schoenfeld residuals for testing the proportional hazards assumption.

Parameters:

data (DataFrame) – Analysis dataset.
duration_col (str) – Time and event columns.
event_col (str) – Time and event columns.
covariate_cols (list[str]) – Covariates used in the Cox model.
cox_result (CoxResult) – A fitted CoxResult from cox_ph().

Returns:

One row per event with columns for each covariate’s Schoenfeld residual and the event time.

Return type:

DataFrame

morie.survival.cox_snell_residuals(data, duration_col, event_col, covariate_cols, cox_result)[source]¶

Compute Cox–Snell residuals.

The Cox–Snell residual for observation i is $r_i = \hat{H}_0(t_i) \exp(x_i^T \hat{\beta})$.

If the model is correct these should follow a unit-rate exponential distribution.

Parameters:

data (DataFrame)
duration_col (str)
event_col (str)
covariate_cols (list[str])
cox_result (CoxResult)

Return type:

ndarray

morie.survival.martingale_residuals(data, duration_col, event_col, covariate_cols, cox_result)[source]¶

Compute martingale residuals.

$M_i = \delta_i - r_i^{CS}$ where $r_i^{CS}$ is the Cox–Snell residual.

Parameters:

data (DataFrame)
duration_col (str)
event_col (str)
covariate_cols (list[str])
cox_result (CoxResult)

Return type:

ndarray

morie.survival.deviance_residuals(data, duration_col, event_col, covariate_cols, cox_result)[source]¶

Compute deviance residuals from a Cox PH model.

$d_i = \text{sign}(M_i) \sqrt{-2[M_i + \delta_i \log(\delta_i - M_i)]}$

Parameters:

data (DataFrame)
duration_col (str)
event_col (str)
covariate_cols (list[str])
cox_result (CoxResult)

Return type:

ndarray

morie.survival.test_ph_assumption(data, duration_col, event_col, covariate_cols, cox_result)[source]¶

Test the proportional hazards assumption via Schoenfeld residual correlation with time.

For each covariate, computes the Pearson correlation between scaled Schoenfeld residuals and event time and reports a p-value.

Parameters:

data (DataFrame)
duration_col (str)
event_col (str)
covariate_cols (list[str])
cox_result (CoxResult)

Returns:

Columns: covariate, rho, chi2, p_value.

Return type:

DataFrame

morie.survival.exponential_model(time, event)[source]¶

Fit an exponential survival model (constant hazard).

The exponential distribution has hazard $h(t) = \lambda$ and survival $S(t) = e^{-\lambda t}$.

Parameters:

time (array-like)
event (array-like)

Return type:

morie.survival.weibull_model(time, event)[source]¶

Fit a Weibull survival model via MLE.

Parameterisation: $S(t) = \exp(-(t/\lambda)^k)$.

Parameters:

time (array-like)
event (array-like)

Return type:

morie.survival.lognormal_model(time, event)[source]¶

Fit a log-normal survival model via MLE.

Parameters:

time (array-like)
event (array-like)

Return type:

morie.survival.loglogistic_model(time, event)[source]¶

Fit a log-logistic survival model via MLE.

$S(t) = 1 / (1 + (t/\alpha)^\beta)$.

Parameters:

time (array-like)
event (array-like)

Return type:

morie.survival.gompertz_model(time, event)[source]¶

Fit a Gompertz survival model via MLE.

Hazard: $h(t) = b \exp(ct)$. Survival: $S(t) = \exp(-(b/c)(e^{ct} - 1))$.

Parameters:

time (array-like)
event (array-like)

Return type:

morie.survival.aft_weibull(data, duration_col, event_col, covariate_cols)[source]¶

Accelerated failure time model with Weibull baseline.

The AFT model specifies $\log T = \mu + x^T \beta + \sigma W$ where W follows a standard extreme value distribution.

Parameters:

data (DataFrame)
duration_col (str)
event_col (str)
covariate_cols (list[str])

Return type:

morie.survival.aft_lognormal(data, duration_col, event_col, covariate_cols)[source]¶

AFT model with log-normal baseline.

Parameters:

data (DataFrame)
duration_col (str)
event_col (str)
covariate_cols (list[str])

Return type:

morie.survival.aft_loglogistic(data, duration_col, event_col, covariate_cols)[source]¶

AFT model with log-logistic baseline.

Parameters:

data (DataFrame)
duration_col (str)
event_col (str)
covariate_cols (list[str])

Return type:

morie.survival.restricted_mean_survival_time(time, event, tau=None, confidence=0.95)[source]¶

Compute the restricted mean survival time (RMST).

$\text{RMST}(\tau) = \int_0^\tau \hat{S}(t) \, dt$.

Parameters:

time (array-like)
event (array-like)
tau (float or None) – Restriction time. If None, uses the maximum observed time.
confidence (float, default 0.95) – Confidence level.

Returns:

Keys: rmst, se, ci_lower, ci_upper, tau.

Return type:

morie.survival.rmst_difference(time1, event1, time2, event2, tau=None, confidence=0.95)[source]¶

Compare RMST between two groups.

Parameters:

time1 (array-like) – Group 1 survival data.
event1 (array-like) – Group 1 survival data.
time2 (array-like) – Group 2 survival data.
event2 (array-like) – Group 2 survival data.
tau (float or None) – Common restriction time.
confidence (float, default 0.95)

Returns:

Keys: rmst_diff, se, z, p_value, ci_lower, ci_upper.

Return type:

morie.survival.cumulative_incidence_function(time, event, event_of_interest=1, confidence=0.95)[source]¶

Aalen–Johansen estimator of the cumulative incidence function (CIF) for competing risks.

Parameters:

time (array-like) – Observed times.
event (array-like) – Event type indicator: 0 = censored, 1 = event of interest, 2+ = competing events.
event_of_interest (int, default 1) – Which event code is the event of interest.
confidence (float, default 0.95)

Return type:

CompetingRiskResult

morie.survival.fine_gray_weights(time, event, event_of_interest=1)[source]¶

Compute Fine–Gray IPCW weights for subdistribution hazard modelling.

These weights can be passed to a weighted Cox model to obtain Fine–Gray subdistribution hazard ratios.

Parameters:

time (array-like)
event (array-like)
event_of_interest (int, default 1)

Returns:

Observation-level weights.

Return type:

ndarray

References

Fine, J. P., & Gray, R. J. (1999). A proportional hazards model for the subdistribution of a competing risk. JASA, 94(446), 496–509.

morie.survival.concordance_index(event_times, predicted_scores, event_observed)[source]¶

Harrell’s concordance index (C-statistic) for survival models.

Parameters:

event_times (array-like) – Observed times.
predicted_scores (array-like) – Model-predicted risk scores (higher = higher risk).
event_observed (array-like) – Event indicator (1 = event).

Returns:

Concordance index in [0, 1].

Return type:

morie.survival.hazard_ratio(time, event, group, confidence=0.95)[source]¶

Estimate the hazard ratio between two groups from a simple Cox model.

Parameters:

time (array-like)
event (array-like)
group (array-like)
confidence (float, default 0.95)

Returns:

Keys: hr, ci_lower, ci_upper, p_value, log_hr, se.

Return type:

morie.survival.survival_plot_data(time, event, group=None, confidence=0.95)[source]¶

Generate a long-format DataFrame suitable for survival curve plotting.

Parameters:

time (array-like)
event (array-like)
group (array-like or None) – Group labels. If None, a single group is assumed.
confidence (float, default 0.95)

Returns:

Columns: time, survival, ci_lower, ci_upper, at_risk, group.

Return type:

DataFrame

morie.survival.survival_calibration(predicted_survival, time, event, n_groups=10)[source]¶

Calibration of predicted survival probabilities.

Divides observations into n_groups deciles of predicted risk and compares predicted vs. observed KM survival within each group.

Parameters:

predicted_survival (array-like) – Model-predicted survival probabilities at a fixed time horizon.
time (array-like)
event (array-like)
n_groups (int, default 10)

Returns:

Columns: group, predicted_mean, observed_survival, n.

Return type:

DataFrame

morie.survival.landmark_dataset(data, duration_col, event_col, landmark_time)[source]¶

Create a landmark analysis dataset.

Excludes subjects who experienced an event or were censored before landmark_time and resets the time origin.

Parameters:

data (DataFrame)
duration_col (str)
event_col (str)
landmark_time (float)

Returns:

Filtered dataset with adjusted times.

Return type:

DataFrame

morie.survival.left_truncated_km(entry_time, exit_time, event, confidence=0.95)[source]¶

Kaplan–Meier estimator with left truncation (delayed entry).

Parameters:

entry_time (array-like) – Time at which each subject enters the risk set.
exit_time (array-like) – Observed failure/censoring time.
event (array-like) – Event indicator.
confidence (float, default 0.95)

Return type:

SurvivalCurve

morie.survival.turnbull_estimator(left, right, max_iter=200, tol=1e-06)[source]¶

Turnbull’s self-consistency algorithm for interval-censored data.

Parameters:

left (array-like) – Left endpoints of observation intervals.
right (array-like) – Right endpoints (use np.inf for right-censored).
max_iter (int, default 200) – Maximum EM iterations.
tol (float, default 1e-6) – Convergence tolerance.

Returns:

(unique_times, survival_probs).

Return type:

tuple[ndarray, ndarray]

morie.survival.compare_parametric_models(time, event)[source]¶

Fit all available parametric models and compare via AIC/BIC.

Parameters:

time (array-like)
event (array-like)

Returns:

Columns: distribution, log_likelihood, aic, bic, n_events.

Return type:

DataFrame

Comprehensive hypothesis testing suite for epidemiological research.

This module provides a unified interface for the full spectrum of frequentist hypothesis tests encountered in public health and biomedical research. Every test function returns a TestResult dataclass containing the test statistic, p-value, degrees of freedom, confidence interval bounds, effect size, and method name so that downstream code can process results programmatically without parsing text output.

Categories of tests¶

Location tests (t-tests, ANOVA, non-parametric rank tests)
Association tests (chi-squared, Fisher exact, correlation)
Distribution tests (normality, homogeneity of variance, goodness of fit)
Proportion tests (one- and two-sample z-tests, Fisher exact)
Agreement tests (Cohen’s kappa, Fleiss’ kappa, ICC)

All implementations delegate heavy numerics to scipy.stats and statsmodels where possible; only formulas absent from those libraries are coded from scratch.

References

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences: (2nd ed.). Lawrence Erlbaum Associates.

Agresti, A. (2013). Categorical Data Analysis (3rd ed.). Wiley. Conover, W. J. (1999). Practical Nonparametric Statistics (3rd ed.). Wiley. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters.

Psychological Bulletin, 76(5), 378–382.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in: assessing rater reliability. Psychological Bulletin, 86(2), 420–428.

class morie.statistics.TestResult(method, test_statistic, p_value, df=None, ci_lower=None, ci_upper=None, effect_size=None, estimate=None, n=None, extra=<factory>)[source]¶

Bases: object

Standardised container for every hypothesis-test result in this module.

Parameters:

method (str) – Human-readable name of the statistical test.
test_statistic (float) – Value of the test statistic (t, F, chi-squared, U, etc.).
p_value (float) – Two-sided p-value (or one-sided where documented).
df (float | None) – Degrees of freedom (None for non-parametric tests that do not define df).
ci_lower (float | None) – Lower bound of the confidence interval for the estimated parameter.
ci_upper (float | None) – Upper bound of the confidence interval for the estimated parameter.
effect_size (float | None) – A standardised effect-size measure appropriate to the test.
estimate (float | None) – Point estimate of the quantity being tested (mean difference, odds ratio, correlation, etc.).
n (int | None) – Total sample size used in the test.
extra (dict) – Arbitrary additional outputs (e.g. per-group means, residuals).

method: str¶

test_statistic: float¶

p_value: float¶

df: float | None = None¶

ci_lower: float | None = None¶

ci_upper: float | None = None¶

effect_size: float | None = None¶

estimate: float | None = None¶

n: int | None = None¶

extra: dict¶

morie.statistics.one_sample_ttest(x, mu0=0.0, confidence=0.95)[source]¶

One-sample Student’s t-test.

Tests $H_0: \mu = \mu_0$ against $H_1: \mu \neq \mu_0$.

Parameters:

x (array-like) – Sample observations.
mu0 (float, default 0.0) – Hypothesised population mean.
confidence (float, default 0.95) – Confidence level for the CI around the sample mean.

Return type:

morie.statistics.two_sample_ttest(x, y, equal_var=True, confidence=0.95)[source]¶

Independent two-sample t-test (equal or unequal variance).

Parameters:

x (array-like) – Two independent samples.
y (array-like) – Two independent samples.
equal_var (bool, default True) – If False, use Welch’s approximation for unequal variances.
confidence (float, default 0.95) – Confidence level.

Return type:

morie.statistics.welch_ttest(x, y, confidence=0.95)[source]¶

Welch’s t-test – convenience wrapper for two_sample_ttest(equal_var=False).

Parameters:

x (array-like) – Two independent samples.
y (array-like) – Two independent samples.
confidence (float, default 0.95) – Confidence level.

Return type:

morie.statistics.paired_ttest(x, y, confidence=0.95)[source]¶

Paired-sample t-test.

Tests $H_0: \mu_d = 0$ where $d_i = x_i - y_i$.

Parameters:

x (array-like) – Paired observations (must have equal length).
y (array-like) – Paired observations (must have equal length).
confidence (float, default 0.95) – Confidence level for the mean difference CI.

Return type:

morie.statistics.one_way_anova(*groups)[source]¶

One-way between-subjects ANOVA (F-test).

Parameters:: *groups (array-like) – Two or more independent samples.
Returns:: Effect size is $\eta^2 = SS_{between} / SS_{total}$.
Return type:: TestResult

morie.statistics.two_way_anova(data, outcome, factor_a, factor_b)[source]¶

Two-way factorial ANOVA via OLS type-II sums of squares.

Parameters:

data (DataFrame) – Long-format data.
outcome (str) – Name of the dependent-variable column.
factor_a (str) – Names of the two factor columns.
factor_b (str) – Names of the two factor columns.

Returns:

The extra dict contains per-factor and interaction F and p values.

Return type:

morie.statistics.repeated_measures_anova(data, outcome, subject, within)[source]¶

One-way repeated-measures ANOVA (sphericity not assumed – uses Greenhouse–Geisser correction).

Parameters:

data (DataFrame) – Long-format data with one row per subject-condition.
outcome (str) – Name of the outcome column.
subject (str) – Name of the subject identifier column.
within (str) – Name of the within-subjects factor column.

Return type:

morie.statistics.kruskal_wallis(*groups)[source]¶

Kruskal–Wallis H-test (non-parametric one-way ANOVA).

Parameters:: *groups (array-like) – Two or more independent samples.
Returns:: Effect size is $\eta^2_H = (H - k + 1) / (N - k)$.
Return type:: TestResult

morie.statistics.friedman_test(*groups)[source]¶

Friedman test for repeated measures on ranks.

Parameters:: *groups (array-like) – Three or more matched samples (equal length).
Returns:: Effect size is Kendall’s W.
Return type:: TestResult

morie.statistics.chi2_goodness_of_fit(observed, expected=None)[source]¶

Chi-squared goodness-of-fit test.

Parameters:

observed (array-like) – Observed frequency counts.
expected (array-like or None) – Expected frequency counts. If None, uniform distribution assumed.

Returns:

Effect size is Cohen’s w.

Return type:

morie.statistics.chi2_independence(contingency_table, correction=True)[source]¶

Chi-squared test of independence for a contingency table.

Parameters:

contingency_table (array-like or DataFrame) – An r x c contingency table of observed counts.
correction (bool, default True) – Apply Yates’s continuity correction for 2x2 tables.

Returns:

Effect size is Cramer’s V.

Return type:

morie.statistics.mcnemar_test(contingency_table, exact=False)[source]¶

McNemar’s test for paired nominal data (2x2 table).

Parameters:

contingency_table (array-like) – A 2x2 contingency table [[a, b], [c, d]].
exact (bool, default False) – If True, use the exact binomial test instead of the chi-squared approximation.

Return type:

morie.statistics.cochrans_q(*groups)[source]¶

Cochran’s Q test for related samples with binary outcomes.

Parameters:: *groups (array-like) – Three or more matched binary (0/1) samples.
Return type:: TestResult

morie.statistics.pearson_correlation(x, y, confidence=0.95)[source]¶

Pearson product-moment correlation with Fisher z CI.

Parameters:

x (array-like) – Two continuous variables.
y (array-like) – Two continuous variables.
confidence (float, default 0.95) – Confidence level.

Return type:

morie.statistics.spearman_correlation(x, y, confidence=0.95)[source]¶

Spearman rank correlation.

Parameters:

x (array-like) – Two variables.
y (array-like) – Two variables.
confidence (float, default 0.95) – Confidence level.

Return type:

morie.statistics.kendall_correlation(x, y)[source]¶

Kendall’s tau-b rank correlation.

Parameters:

x (array-like) – Two variables.
y (array-like) – Two variables.

Return type:

morie.statistics.point_biserial_correlation(binary, continuous, confidence=0.95)[source]¶

Point-biserial correlation between a binary and continuous variable.

Parameters:

binary (array-like) – Binary (0/1) variable.
continuous (array-like) – Continuous variable.
confidence (float, default 0.95) – Confidence level.

Return type:

morie.statistics.partial_correlation(x, y, covariates, confidence=0.95)[source]¶

Partial Pearson correlation controlling for covariates.

Uses OLS residualisation: regress both x and y on the covariates, then correlate the residuals.

Parameters:

x (array-like) – Variables of interest.
y (array-like) – Variables of interest.
covariates (array-like or DataFrame) – Matrix of control variables (n x p).
confidence (float, default 0.95) – Confidence level.

Return type:

morie.statistics.semi_partial_correlation(x, y, covariates)[source]¶

Semi-partial (part) correlation.

Residualises only x on the covariates and correlates the residual with the raw y.

Parameters:

x (array-like) – Variables of interest.
y (array-like) – Variables of interest.
covariates (array-like or DataFrame) – Matrix of control variables.

Return type:

morie.statistics.mann_whitney_u(x, y, alternative='two-sided')[source]¶

Mann–Whitney U test (Wilcoxon rank-sum test).

Parameters:

x (array-like) – Two independent samples.
y (array-like) – Two independent samples.
alternative (str, default "two-sided") – One of "two-sided", "less", "greater".

Returns:

Effect size is rank-biserial correlation $r = 2U / (n_1 n_2) - 1$, positive when x tends to exceed y. The p-value is exact for untied samples both under 50, as wilcox.test, and the tie-corrected normal approximation otherwise.

Return type:

morie.statistics.wilcoxon_signed_rank(x, y=None, alternative='two-sided')[source]¶

Wilcoxon signed-rank test for paired samples or one sample.

Parameters:

x (array-like) – If y is None, test whether the distribution of x is symmetric about zero. Otherwise, test paired differences x - y.
y (array-like or None) – Second sample for paired test.
alternative (str, default "two-sided") – One of "two-sided", "less", "greater".

Return type:

morie.statistics.ks_test_one_sample(x, cdf='norm', args=())[source]¶

One-sample Kolmogorov–Smirnov test.

Parameters:

x (array-like) – Sample data.
cdf (str, default "norm") – Name of a scipy.stats distribution (e.g. "norm", "expon").
args (tuple) – Extra arguments to the CDF (loc, scale, etc.).

Return type:

morie.statistics.ks_test_two_sample(x, y)[source]¶

Two-sample Kolmogorov–Smirnov test.

Parameters:

x (array-like) – Two independent samples.
y (array-like) – Two independent samples.

Return type:

morie.statistics.anderson_darling(x, dist='norm')[source]¶

Anderson–Darling goodness-of-fit test.

Native implementation for the two composite null hypotheses with published percentage points: the normal distribution with unknown mean and variance, and the exponential distribution with unknown mean.

\[A^2 = -n - n^{-1} \sum (2i-1)[\ln F(z_i) + \ln(1 - F(z_{n+1-i}))]\]

Because the parameters are estimated from the same sample, $A^2$ is not referred to its own null distribution: the modified statistic is $A^* = A^2(1 + 0.75/n + 2.25/n^2)$ in the normal case and $A^* = A^2(1 + 0.3/n)$ in the exponential case.

Parameters:

x (array-like) – Sample data.
dist (str, default "norm") – Either "norm" or "expon".

Returns:

test_statistic is the modified $A^*$. extra carries the unmodified a_squared and p_bounded ("upper"/"lower") when the statistic falls outside the tabulated range, in which case the p-value is the nearest bound rather than an exact value.

Return type:

References

Gibbons, J. D. & Chakraborti, S. (2010). Nonparametric Statistical Inference, 5th edn. CRC Press. Section 4.7 and Table 4.7.1.

Stephens, M. A. (1986). Tests based on EDF statistics. In R. B. D’Agostino & M. A. Stephens (eds), Goodness-of-Fit Techniques. Marcel Dekker.

morie.statistics.levene_test(*groups, center='median')[source]¶

Levene’s test for equality of variances.

Parameters:

*groups (array-like) – Two or more samples.
center (str, default "median") – "median" (Brown–Forsythe), "mean", or "trimmed".

Return type:

morie.statistics.bartlett_test(*groups)[source]¶

Bartlett’s test for equality of variances.

Parameters:: *groups (array-like) – Two or more samples.
Return type:: TestResult

morie.statistics.runs_test(x, cutoff=None)[source]¶

Wald–Wolfowitz runs test for randomness.

Parameters:

x (array-like) – Numeric sequence.
cutoff (float or None) – If None, uses the median to dichotomise x.

Return type:

morie.statistics.shapiro_wilk(x)[source]¶

Shapiro–Wilk test for normality.

Parameters:: x (array-like) – Sample data (n <= 5000 recommended by scipy).
Return type:: TestResult

morie.statistics.dagostino_pearson(x)[source]¶

D’Agostino–Pearson omnibus normality test.

Parameters:: x (array-like) – Sample data (n >= 20 recommended).
Return type:: TestResult

morie.statistics.jarque_bera(x)[source]¶

Jarque–Bera test for normality (based on skewness and kurtosis).

Parameters:: x (array-like) – Sample data.
Return type:: TestResult

morie.statistics.lilliefors_test(x, dist='norm')[source]¶

Lilliefors goodness-of-fit test.

The Kolmogorov–Smirnov statistic referred to Lilliefors’s null distribution rather than Kolmogorov’s. When the parameters are estimated from the same sample the ordinary K–S critical values are badly conservative – the point Lilliefors (1967) established – so the correct points come from separate simulations, tabulated for the normal case and for the exponential case.

Parameters:

x (array-like) – Sample data.
dist (str, default "norm") – "norm" (mean and variance unknown) or "expon" (mean unknown).

Returns:

extra["p_bounded"] is "upper"/"lower" when D falls outside the tabulated range; the p-value is then the nearest bound (0.10 or 0.001) rather than an exact value.

Return type:

References

Lilliefors, H. W. (1967). On the Kolmogorov-Smirnov test for normality with mean and variance unknown. Journal of the American Statistical Association, 62(318), 399–402.

Lilliefors, H. W. (1969). On the Kolmogorov-Smirnov test for the exponential distribution with mean unknown. Journal of the American Statistical Association, 64(325), 387–389.

Gibbons, J. D. & Chakraborti, S. (2010). Nonparametric Statistical Inference, 5th edn. CRC Press. Sections 4.5–4.6, Tables O and T.

morie.statistics.one_proportion_ztest(count, nobs, value=0.5, confidence=0.95)[source]¶

One-sample z-test for a proportion.

Parameters:

count (int) – Number of successes.
nobs (int) – Number of observations.
value (float, default 0.5) – Hypothesised proportion under H_0.
confidence (float, default 0.95) – Confidence level for the Wilson score interval.

Return type:

morie.statistics.two_proportion_ztest(count1, nobs1, count2, nobs2, confidence=0.95)[source]¶

Two-sample z-test for the difference between two proportions.

Parameters:

count1 (int) – Successes and observations in sample 1.
nobs1 (int) – Successes and observations in sample 1.
count2 (int) – Successes and observations in sample 2.
nobs2 (int) – Successes and observations in sample 2.
confidence (float, default 0.95) – Confidence level.

Return type:

morie.statistics.fisher_exact_test(contingency_table, alternative='two-sided')[source]¶

Fisher’s exact test for a 2x2 contingency table.

Parameters:

contingency_table (array-like) – A 2x2 table of counts.
alternative (str, default "two-sided") – "two-sided", "less", or "greater".

Returns:

estimate is the odds ratio.

Return type:

morie.statistics.cohens_kappa(rater1, rater2, confidence=0.95)[source]¶

Cohen’s kappa for inter-rater agreement between two raters.

Parameters:

rater1 (array-like) – Categorical ratings from two raters (same length).
rater2 (array-like) – Categorical ratings from two raters (same length).
confidence (float, default 0.95) – Confidence level.

Return type:

morie.statistics.fleiss_kappa(ratings_matrix)[source]¶

Fleiss’ kappa for agreement among multiple raters.

Parameters:: ratings_matrix (array-like) – An n x k matrix where rows are subjects and columns are rating categories. Cell (i, j) is the number of raters who assigned subject i to category j.
Return type:: TestResult

References

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.

morie.statistics.intraclass_correlation(data, targets, raters, ratings, icc_type='ICC3k')[source]¶

Intraclass correlation coefficient (ICC).

Implements the Shrout & Fleiss (1979) taxonomy:

ICC1: One-way random, single measures
ICC1k: One-way random, average measures
ICC2: Two-way random, single measures
ICC2k: Two-way random, average measures
ICC3: Two-way mixed, single measures
ICC3k: Two-way mixed, average measures

Parameters:

data (DataFrame) – Long-format data.
targets (str) – Column identifying subjects/targets.
raters (str) – Column identifying raters.
ratings (str) – Column containing ratings (numeric).
icc_type (str, default "ICC3k") – Which ICC form to compute.

Return type:

References

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.

morie.statistics.normality_battery(x)[source]¶

Run all available normality tests on a single sample.

Parameters:: x (array-like) – Sample data.
Returns:: Results from Shapiro–Wilk, D’Agostino–Pearson, Jarque–Bera, and Lilliefors tests.
Return type:: list[TestResult]

morie.statistics.variance_equality_battery(*groups)[source]¶

Run Levene’s (median) and Bartlett’s tests for homogeneity of variance.

Parameters:: *groups (array-like) – Two or more samples.
Return type:: list[TestResult]

morie.statistics.correlation_matrix(data, method='pearson')[source]¶

Compute a pairwise correlation matrix with p-values.

Parameters:

data (DataFrame) – Numeric columns only.
method (str, default "pearson") – "pearson", "spearman", or "kendall".

Returns:

MultiIndex columns with ("r", col) and ("p", col) levels.

Return type:

DataFrame

morie.statistics.auto_test(x, y=None, paired=False, confidence=0.95)[source]¶

Automatically select and run the most appropriate test.

Decision logic:

If y is None – one-sample t-test against zero.
If paired=True – paired t-test (if normal differences) or Wilcoxon.
If two independent samples – check normality and variance equality; choose between Student’s t, Welch’s t, or Mann–Whitney U.

Parameters:

x (array-like) – First sample.
y (array-like or None) – Second sample (if applicable).
paired (bool, default False) – Whether samples are paired.
confidence (float, default 0.95) – Confidence level.

Return type:

Comprehensive bootstrap and resampling inference methods.

Provides nonparametric and parametric bootstrap, jackknife, permutation tests, subsampling, and cross-validation resampling for statistical inference in epidemiological and observational studies.

All methods support stratified resampling, cluster-aware resampling, and survey-weighted variants for complex survey designs.

class morie.bootstrap_methods.BootstrapResult(estimate, se, ci_lower, ci_upper, bias, n_boot, method, ci_method, boot_distribution, original_estimate, acceleration=0.0)[source]¶

Bases: object

Result from a bootstrap procedure.

Parameters:

estimate (float)
se (float)
ci_lower (float)
ci_upper (float)
bias (float)
n_boot (int)
method (str)
ci_method (str)
boot_distribution (marr)
original_estimate (float)
acceleration (float)

estimate: float¶

se: float¶

ci_lower: float¶

ci_upper: float¶

bias: float¶

n_boot: int¶

method: str¶

ci_method: str¶

boot_distribution: marr¶

original_estimate: float¶

acceleration: float = 0.0¶

summary()[source]¶

Return type:: dict

class morie.bootstrap_methods.JackknifeResult(estimate, se, ci_lower, ci_upper, bias, n, jackknife_estimates, pseudovalues, influence_values)[source]¶

Bases: object

Result from jackknife estimation.

Parameters:

estimate (float)
se (float)
ci_lower (float)
ci_upper (float)
bias (float)
n (int)
jackknife_estimates (marr)
pseudovalues (marr)
influence_values (marr)

estimate: float¶

se: float¶

ci_lower: float¶

ci_upper: float¶

bias: float¶

n: int¶

jackknife_estimates: marr¶

pseudovalues: marr¶

influence_values: marr¶

summary()[source]¶

Return type:: dict

class morie.bootstrap_methods.PermutationTestResult(observed_statistic, p_value, null_distribution, n_permutations, alternative, ci_lower=nan, ci_upper=nan)[source]¶

Bases: object

Result from a permutation test.

Parameters:

observed_statistic (float)
p_value (float)
null_distribution (marr)
n_permutations (int)
alternative (str)
ci_lower (float)
ci_upper (float)

observed_statistic: float¶

p_value: float¶

null_distribution: marr¶

n_permutations: int¶

alternative: str¶

ci_lower: float = nan¶

ci_upper: float = nan¶

class morie.bootstrap_methods.CrossValidationResult(scores, mean_score, se_score, ci_lower, ci_upper, n_folds, metric, fold_sizes)[source]¶

Bases: object

Result from cross-validation.

Parameters:

scores (marr)
mean_score (float)
se_score (float)
ci_lower (float)
ci_upper (float)
n_folds (int)
metric (str)
fold_sizes (list[int])

scores: marr¶

mean_score: float¶

se_score: float¶

ci_lower: float¶

ci_upper: float¶

n_folds: int¶

metric: str¶

fold_sizes: list[int]¶

morie.bootstrap_methods.bootstrap(data, statistic, n_boot=2000, ci_level=0.95, ci_method='bca', seed=42, stratify=None, cluster=None)[source]¶

Nonparametric bootstrap inference.

Parameters:

data (array-like) – Input data (1D or 2D).
statistic (callable) – Function that computes the statistic from a data array.
n_boot (int) – Number of bootstrap replicates.
ci_level (float) – Confidence interval level.
ci_method (str) – CI method: ‘percentile’, ‘normal’, ‘basic’, ‘bca’, ‘studentized’.
seed (int) – Random seed.
stratify (array-like, optional) – Stratification variable for stratified bootstrap.
cluster (array-like, optional) – Cluster variable for cluster bootstrap.

Return type:

Examples

>>> from morie.fn import _array_core as np
>>> x = np.array([3.1, 2.4, 4.0, 3.3, 2.9, 3.8, 3.5, 2.7])
>>> b = bootstrap(x, lambda a: float(np.mean(a)), n_boot=200, seed=1)
>>> (round(b.estimate, 4), round(b.ci_lower, 4), round(b.ci_upper, 4), b.ci_method)
(3.2125, 2.7232, 3.5087, 'bca')

morie.bootstrap_methods.parametric_bootstrap(data, statistic, distribution='normal', n_boot=2000, ci_level=0.95, seed=42, **dist_params)[source]¶

Parametric bootstrap.

Generates bootstrap samples from a fitted parametric distribution rather than resampling from the data.

Parameters:

data (array-like) – Original data (used to fit the distribution).
statistic (callable) – Function that computes the statistic.
distribution (str) – Distribution: ‘normal’, ‘poisson’, ‘binomial’, ‘exponential’, ‘gamma’.
n_boot (int) – Number of bootstrap replicates.
ci_level (float) – Confidence interval level.
seed (int) – Random seed.

Return type:

morie.bootstrap_methods.wild_bootstrap(y, X, statistic_idx=1, n_boot=999, ci_level=0.95, weight_distribution='rademacher', seed=42)[source]¶

Wild bootstrap for linear regression with heteroskedasticity.

Parameters:

y (array-like) – Response variable.
X (array-like) – Design matrix (including intercept if desired).
statistic_idx (int) – Index of the coefficient to test.
n_boot (int) – Number of bootstrap replicates.
ci_level (float) – Confidence interval level.
weight_distribution (str) – ‘rademacher’ (±1 with equal probability) or ‘mammen’.
seed (int) – Random seed.

Return type:

morie.bootstrap_methods.block_bootstrap(data, statistic, block_size, n_boot=2000, ci_level=0.95, method='circular', seed=42)[source]¶

Block bootstrap for dependent data.

Parameters:

data (array-like) – Time series or dependent data.
statistic (callable) – Function that computes the statistic.
block_size (int) – Length of each block.
n_boot (int) – Number of bootstrap replicates.
ci_level (float) – Confidence interval level.
method (str) – ‘moving’ (non-overlapping), ‘circular’, or ‘stationary’.
seed (int) – Random seed.

Return type:

morie.bootstrap_methods.jackknife(data, statistic, ci_level=0.95)[source]¶

Delete-one jackknife estimation.

Parameters:

data (array-like) – Input data.
statistic (callable) – Function that computes the statistic.
ci_level (float) – Confidence interval level.

Return type:

JackknifeResult

morie.bootstrap_methods.delete_d_jackknife(data, statistic, d=2, ci_level=0.95, max_subsets=5000, seed=42)[source]¶

Delete-d jackknife (generalized jackknife).

Parameters:

data (array-like) – Input data.
statistic (callable) – Function that computes the statistic.
d (int) – Number of observations to delete.
ci_level (float) – Confidence interval level.
max_subsets (int) – Maximum number of subsets to evaluate.
seed (int) – Random seed.

Return type:

JackknifeResult

morie.bootstrap_methods.permutation_test(group1, group2, statistic='mean_diff', n_permutations=9999, alternative='two-sided', seed=42)[source]¶

Two-sample permutation test.

Parameters:

group1 (array-like) – First group.
group2 (array-like) – Second group.
statistic (str or callable) – Test statistic: ‘mean_diff’, ‘median_diff’, ‘t_stat’, or a callable(g1, g2) -> float.
n_permutations (int) – Number of permutations.
alternative (str) – ‘two-sided’, ‘greater’, ‘less’.
seed (int) – Random seed.

Return type:

PermutationTestResult

morie.bootstrap_methods.paired_permutation_test(x, y, statistic='mean_diff', n_permutations=9999, alternative='two-sided', seed=42)[source]¶

Paired permutation test (sign-flipping).

Parameters:

x (array-like) – First measurement.
y (array-like) – Second measurement (paired with x).
statistic (str) – ‘mean_diff’ or ‘median_diff’.
n_permutations (int) – Number of permutations.
alternative (str) – ‘two-sided’, ‘greater’, ‘less’.
seed (int) – Random seed.

Return type:

PermutationTestResult

morie.bootstrap_methods.subsampling(data, statistic, subsample_size=None, n_subsamples=1000, ci_level=0.95, seed=42)[source]¶

Subsampling inference (Politis, Romano & Wolf).

Unlike the bootstrap, subsampling draws without replacement at a smaller sample size, providing valid inference under weaker conditions.

Parameters:

data (array-like) – Input data.
statistic (callable) – Function that computes the statistic.
subsample_size (int, optional) – Size of each subsample. Default: int(n^0.7).
n_subsamples (int) – Number of subsamples.
ci_level (float) – Confidence interval level.
seed (int) – Random seed.

Return type:

morie.bootstrap_methods.bootstrap_632(X, y, model_fn, score_fn, n_boot=200, seed=42)[source]¶

The .632 bootstrap estimator for prediction error.

Parameters:

X (array-like, shape (n, p)) – Feature matrix.
y (array-like, shape (n,)) – Response.
model_fn (callable) – Function that fits a model: model_fn(X_train, y_train) -> model.
score_fn (callable) – Function that scores predictions: score_fn(y_true, y_pred) -> float.
n_boot (int) – Number of bootstrap replicates.
seed (int) – Random seed.

Returns:

With keys: apparent_error, bootstrap_error, error_632, error_632plus.

Return type:

morie.bootstrap_methods.cross_validate(X, y, model_fn, score_fn, n_folds=10, stratify=None, groups=None, seed=42)[source]¶

K-fold cross-validation.

Parameters:

X (array-like, shape (n, p)) – Feature matrix.
y (array-like, shape (n,)) – Response.
model_fn (callable) – model_fn(X_train, y_train) -> model with .predict().
score_fn (callable) – score_fn(y_true, y_pred) -> float.
n_folds (int) – Number of folds.
stratify (array-like, optional) – Stratification variable for stratified CV.
groups (array-like, optional) – Group variable for grouped CV (no group split across folds).
seed (int) – Random seed.

Return type:

CrossValidationResult

morie.bootstrap_methods.repeated_cv(X, y, model_fn, score_fn, n_folds=10, n_repeats=10, seed=42)[source]¶

Repeated K-fold cross-validation.

Parameters:

X (array-like) – Feature matrix.
y (array-like) – Response.
model_fn (callable) – Model fitting function.
score_fn (callable) – Scoring function.
n_folds (int) – Number of folds per repeat.
n_repeats (int) – Number of repetitions.
seed (int) – Random seed.

Return type:

CrossValidationResult

morie.bootstrap_methods.leave_one_out_cv(X, y, model_fn, score_fn)[source]¶

Leave-one-out cross-validation.

Parameters:

X (array-like) – Feature matrix.
y (array-like) – Response.
model_fn (callable) – Model fitting function.
score_fn (callable) – Scoring function.

Return type:

CrossValidationResult

Multiple testing correction and multiplicity-adjusted inference.

Provides comprehensive p-value adjustment methods, family-wise error rate (FWER) control, false discovery rate (FDR) control, and simultaneous confidence interval methods for multiple comparisons in epidemiological and biostatistical analyses.

All functions accept arrays of p-values and return adjusted p-values or decision boundaries. Compatible with the MORIE pipeline and can be applied to any collection of hypothesis tests from the analysis modules.

class morie.multiple_testing.AdjustedPValues(original, adjusted, method, alpha, rejected, n_rejected, n_tests, labels=None)[source]¶

Bases: object

Container for multiplicity-adjusted p-values.

Parameters:

original (marr)
adjusted (marr)
method (str)
alpha (float)
rejected (marr)
n_rejected (int)
n_tests (int)
labels (list[str] | None)

original: marr¶

adjusted: marr¶

method: str¶

alpha: float¶

rejected: marr¶

n_rejected: int¶

n_tests: int¶

labels: list[str] | None = None¶

summary()[source]¶

Return a summary DataFrame.

Return type:: DataFrame

class morie.multiple_testing.GatekeepingResult(stages, overall_rejected, method, alpha)[source]¶

Bases: object

Result from a gatekeeping or hierarchical testing procedure.

Parameters:

stages (list[dict])
overall_rejected (marr)
method (str)
alpha (float)

stages: list[dict]¶

overall_rejected: marr¶

method: str¶

alpha: float¶

class morie.multiple_testing.ClosedTestResult(original_p, adjusted_p, rejected, intersection_tests, method)[source]¶

Bases: object

Result from a closed testing procedure.

Parameters:

original_p (marr)
adjusted_p (marr)
rejected (marr)
intersection_tests (dict)
method (str)

original_p: marr¶

adjusted_p: marr¶

rejected: marr¶

intersection_tests: dict¶

method: str¶

morie.multiple_testing.bonferroni(p_values, alpha=0.05, labels=None)[source]¶

Bonferroni correction (FWER control).

Parameters:

p_values (array-like) – Raw p-values.
alpha (float) – Significance level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.sidak(p_values, alpha=0.05, labels=None)[source]¶

Sidak correction (FWER control, slightly less conservative than Bonferroni).

Parameters:

p_values (array-like) – Raw p-values.
alpha (float) – Significance level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.holm(p_values, alpha=0.05, labels=None)[source]¶

Holm step-down procedure (FWER control).

Uniformly more powerful than Bonferroni.

Parameters:

p_values (array-like) – Raw p-values.
alpha (float) – Significance level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.hochberg(p_values, alpha=0.05, labels=None)[source]¶

Hochberg step-up procedure (FWER control under PRDS).

Parameters:

p_values (array-like) – Raw p-values.
alpha (float) – Significance level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.hommel(p_values, alpha=0.05, labels=None)[source]¶

Hommel procedure (FWER control, most powerful single-step method under independence).

Parameters:

p_values (array-like) – Raw p-values.
alpha (float) – Significance level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.holm_sidak(p_values, alpha=0.05, labels=None)[source]¶

Holm-Sidak step-down procedure.

Parameters:

p_values (array-like) – Raw p-values.
alpha (float) – Significance level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.benjamini_hochberg(p_values, alpha=0.05, labels=None)[source]¶

Benjamini-Hochberg procedure (FDR control).

Controls the false discovery rate at level alpha under independence or positive regression dependency (PRDS).

Parameters:

p_values (array-like) – Raw p-values.
alpha (float) – Target FDR level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.bh(p_values, alpha=0.05, labels=None)¶

Benjamini-Hochberg procedure (FDR control).

Controls the false discovery rate at level alpha under independence or positive regression dependency (PRDS).

Parameters:

p_values (array-like) – Raw p-values.
alpha (float) – Target FDR level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.benjamini_yekutieli(p_values, alpha=0.05, labels=None)[source]¶

Benjamini-Yekutieli procedure (FDR control under arbitrary dependence).

Parameters:

p_values (array-like) – Raw p-values.
alpha (float) – Target FDR level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.by(p_values, alpha=0.05, labels=None)¶

Benjamini-Yekutieli procedure (FDR control under arbitrary dependence).

Parameters:

p_values (array-like) – Raw p-values.
alpha (float) – Target FDR level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.storey_q(p_values, alpha=0.05, lambda_param=0.5, labels=None)[source]¶

Storey’s q-value procedure (adaptive FDR control).

Estimates the proportion of true null hypotheses (pi0) and uses it to achieve higher power than BH when many nulls are false.

Parameters:

p_values (array-like) – Raw p-values.
alpha (float) – Target FDR level.
lambda_param (float) – Tuning parameter for pi0 estimation.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.local_fdr(p_values, labels=None)[source]¶

Estimate local false discovery rate from p-values.

Uses an empirical Bayes approach with a two-component mixture model.

Parameters:

p_values (array-like) – Raw p-values.
labels (list[str], optional) – Labels for each test.

Returns:

With columns: p_value, z_score, local_fdr, label (if provided).

Return type:

pd.DataFrame

morie.multiple_testing.permutation_fwer(test_statistics, null_distribution, alternative='two-sided', alpha=0.05, labels=None)[source]¶

FWER control via max-T permutation test (Westfall-Young).

Parameters:

test_statistics (array-like, shape (m,)) – Observed test statistics.
null_distribution (array-like, shape (n_perm, m)) – Null distribution from permutations.
alternative (str) – One of ‘two-sided’, ‘greater’, ‘less’.
alpha (float) – Significance level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.permutation_fdr(p_values, null_p_values, alpha=0.05, labels=None)[source]¶

FDR control via permutation null distribution.

Parameters:

p_values (array-like, shape (m,)) – Observed p-values.
null_p_values (array-like, shape (n_perm, m)) – P-values from permuted data.
alpha (float) – Target FDR level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.fisher_combined(p_values)[source]¶

Fisher’s method for combining independent p-values.

Parameters:: p_values (array-like) – Independent p-values.
Returns:: (test_statistic, combined_p_value).
Return type:: tuple[float, float]

morie.multiple_testing.stouffer_combined(p_values, weights=None)[source]¶

Stouffer’s method for combining independent p-values.

Parameters:

p_values (array-like) – Independent p-values.
weights (array-like, optional) – Weights for each p-value. If None, equal weights are used.

Returns:

(z_statistic, combined_p_value).

Return type:

morie.multiple_testing.tippett_combined(p_values)[source]¶

Tippett’s method: combine via minimum p-value.

Parameters:: p_values (array-like) – Independent p-values.
Returns:: (min_p, combined_p).
Return type:: tuple[float, float]

morie.multiple_testing.simes_combined(p_values)[source]¶

Simes’ test for the global null hypothesis.

Parameters:: p_values (array-like) – P-values (may be dependent under PRDS).
Returns:: (simes_statistic, combined_p).
Return type:: tuple[float, float]

morie.multiple_testing.harmonic_mean_p(p_values)[source]¶

Harmonic mean p-value for combining dependent tests.

Parameters:: p_values (array-like) – P-values (can be dependent).
Returns:: Harmonic mean p-value.
Return type:: float

morie.multiple_testing.cauchy_combination(p_values, weights=None)[source]¶

Cauchy combination test (Liu & Xie 2020).

Robust to arbitrary correlation structures.

Parameters:

p_values (array-like) – P-values.
weights (array-like, optional) – Non-negative weights summing to 1.

Returns:

(test_statistic, combined_p).

Return type:

morie.multiple_testing.hierarchical_bonferroni(p_values_by_family, alpha=0.05, propagate_alpha=True)[source]¶

Hierarchical (serial gatekeeping) Bonferroni procedure.

Tests are organized in families (stages). A family is tested only if at least one hypothesis in the previous family is rejected.

Parameters:

p_values_by_family (list of array-like) – P-values grouped by family/stage.
alpha (float) – Overall FWER level.
propagate_alpha (bool) – If True, unused alpha is propagated to next family.

Return type:

GatekeepingResult

morie.multiple_testing.fixed_sequence(p_values, alpha=0.05, labels=None)[source]¶

Fixed-sequence (predetermined order) testing procedure.

Tests are evaluated in order; the procedure stops at the first non-rejection. No multiplicity adjustment needed for tests that are reached.

Parameters:

p_values (array-like) – P-values in the predetermined testing order.
alpha (float) – Significance level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.fallback_procedure(p_values, weights, alpha=0.05, labels=None)[source]¶

Fallback (fixed-sequence with alpha spending) procedure.

Parameters:

p_values (array-like) – P-values in testing order.
weights (array-like) – Alpha allocation weights (must sum to 1).
alpha (float) – Overall significance level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.estimate_pi0(p_values, method='storey')[source]¶

Estimate the proportion of true null hypotheses (pi0).

Parameters:

p_values (array-like) – Raw p-values.
method (str) – Estimation method: ‘storey’ (default), ‘bootstrap’, ‘two_step’.

Returns:

Estimated pi0 in [0, 1].

Return type:

morie.multiple_testing.adjust_p_values(p_values, method='bh', alpha=0.05, labels=None)[source]¶

Convenience dispatcher for p-value adjustment methods.

Parameters:

p_values (array-like) – Raw p-values.
method (str) – Adjustment method. One of: ‘bonferroni’, ‘sidak’, ‘holm’, ‘hochberg’, ‘hommel’, ‘holm_sidak’, ‘bh’, ‘by’, ‘storey’.
alpha (float) – Significance level.
labels (list[str], optional) – Labels for each test.

Return type:

morie.multiple_testing.n_effective_tests(p_values=None, correlation_matrix=None, method='galwey')[source]¶

Estimate the effective number of independent tests.

Useful for determining appropriate Bonferroni correction when tests are correlated.

Parameters:

p_values (array-like, optional) – P-values (used for some methods).
correlation_matrix (array-like, optional) – Correlation matrix between test statistics.
method (str) – ‘galwey’ (eigenvalue-based), ‘li_ji’ (Li & Ji 2005), ‘nyholt’ (Nyholt 2004).

Returns:

Estimated number of effective independent tests.

Return type:

Comprehensive effect-size calculations for epidemiological research.

This module provides every major family of effect-size estimators used in biomedical and social-science research, each with analytic or bootstrap confidence intervals. Functions are organised by the type of comparison:

Standardised mean differences: Cohen’s d, Hedges’ g, Glass’s delta
Common-language effect sizes: CLES / probability of superiority
Correlation-based: r, R-squared, eta-squared, partial eta-squared, omega-squared, epsilon-squared
Contingency-table measures: odds ratio, risk ratio, risk difference, NNT, NNH, rate ratio, incidence rate difference
Association measures: Cohen’s w, Cramer’s V, phi coefficient
Non-parametric: rank-biserial correlation, Cliff’s delta, Vargha–Delaney A
Regression: standardised coefficients, coefficient of variation
Conversion: d <-> r, OR <-> d, etc.
Meta-analysis: fixed- and random-effects pooling, I-squared, prediction intervals

References

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences: (2nd ed.). Lawrence Erlbaum Associates.
Hedges, L. V., & Olkin, I. (1985). Statistical Methods for Meta-Analysis.: Academic Press.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009).: Introduction to Meta-Analysis. Wiley.
Vargha, A., & Delaney, H. D. (2000). A critique and improvement of the CL: common language effect size statistics of McGraw and Wong. JEBS, 25(2), 101–132.

class morie.effect_sizes.EffectSizeResult(measure, estimate, ci_lower=None, ci_upper=None, se=None, n=None, extra=<factory>)[source]¶

Bases: object

Standardised result for every effect-size calculation.

Parameters:

measure (str) – Name of the effect-size statistic.
estimate (float) – Point estimate.
ci_lower (float | None) – Lower confidence bound.
ci_upper (float | None) – Upper confidence bound.
se (float | None) – Standard error (analytic or bootstrap).
n (int | None) – Sample size used.
extra (dict) – Additional outputs.

measure: str¶

estimate: float¶

ci_lower: float | None = None¶

ci_upper: float | None = None¶

se: float | None = None¶

n: int | None = None¶

extra: dict¶

morie.effect_sizes.cohens_d(x, y, confidence=0.95)[source]¶

Cohen’s d for independent samples (pooled SD denominator).

\[d = \frac{\bar{x} - \bar{y}}{s_p}\]

Parameters:

x (array-like) – Two independent samples.
y (array-like) – Two independent samples.
confidence (float, default 0.95)

Return type:

Examples

>>> r = cohens_d([2.1, 2.5, 3.0, 2.8, 3.3], [1.4, 1.9, 2.2, 1.7, 2.0])
>>> (r.measure, round(r.estimate, 4), round(r.ci_lower, 4) < r.estimate < round(r.ci_upper, 4))
("Cohen's d", 2.3009, True)

morie.effect_sizes.hedges_g(x, y, confidence=0.95)[source]¶

Hedges’ g – bias-corrected Cohen’s d.

Applies the exact correction factor $J(m) = \Gamma(m/2) / (\sqrt{m/2}\,\Gamma((m-1)/2))$, $m = n_1+n_2-2$ (Hedges 1981); $1 - 3/(4m-1)$ is its large-m approximation.

Parameters:

x (array-like)
y (array-like)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.glass_delta(x, y, control='y', confidence=0.95)[source]¶

Glass’s delta – uses the control group SD as denominator.

Parameters:

x (array-like)
y (array-like)
control (str, default "y") – Which group is the control: "x" or "y".
confidence (float, default 0.95)

Return type:

morie.effect_sizes.cles(x, y, confidence=0.95)[source]¶

Common Language Effect Size (probability of superiority).

Estimates $P(X > Y)$ for randomly drawn observations from each group.

Parameters:

x (array-like)
y (array-like)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.r_effect_size(x, y, confidence=0.95)[source]¶

Pearson r as an effect size with Fisher z CI.

Parameters:

x (array-like)
y (array-like)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.r_squared(x, y)[source]¶

Coefficient of determination R-squared.

Parameters:

x (array-like)
y (array-like)

Return type:

morie.effect_sizes.eta_squared(ss_effect, ss_total)[source]¶

Eta-squared from ANOVA sums of squares.

$\eta^2 = SS_{effect} / SS_{total}$

Parameters:

ss_effect (float) – Sum of squares for the effect.
ss_total (float) – Total sum of squares.

Return type:

morie.effect_sizes.partial_eta_squared(ss_effect, ss_error)[source]¶

Partial eta-squared.

$\eta^2_p = SS_{effect} / (SS_{effect} + SS_{error})$

Parameters:

ss_effect (float)
ss_error (float)

Return type:

morie.effect_sizes.omega_squared(ss_effect, ss_total, df_effect, ms_error)[source]¶

Omega-squared – less biased than eta-squared.

$\omega^2 = (SS_{effect} - df_{effect} \cdot MS_{error}) / (SS_{total} + MS_{error})$

Parameters:

ss_effect (float)
ss_total (float)
df_effect (int)
ms_error (float)

Return type:

morie.effect_sizes.epsilon_squared(ss_effect, ss_total, df_effect, ms_error)[source]¶

Epsilon-squared (Kelley, 1935).

$\varepsilon^2 = (SS_{effect} - df_{effect} \cdot MS_{error}) / SS_{total}$

Parameters:

ss_effect (float)
ss_total (float)
df_effect (int)
ms_error (float)

Return type:

morie.effect_sizes.odds_ratio(a, b, c, d, confidence=0.95)[source]¶

Odds ratio for a 2x2 table [[a, b], [c, d]].

$OR = (a \cdot d) / (b \cdot c)$

Parameters:

a (int) – Cell counts of the 2x2 table.
b (int) – Cell counts of the 2x2 table.
c (int) – Cell counts of the 2x2 table.
d (int) – Cell counts of the 2x2 table.
confidence (float, default 0.95)

Return type:

morie.effect_sizes.risk_ratio(a, b, c, d, confidence=0.95)[source]¶

Risk ratio (relative risk) for a 2x2 table.

$RR = [a/(a+b)] / [c/(c+d)]$

Parameters:

a (int)
b (int)
c (int)
d (int)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.risk_difference(a, b, c, d, confidence=0.95)[source]¶

Risk difference (attributable risk) for a 2x2 table.

$RD = a/(a+b) - c/(c+d)$

Parameters:

a (int)
b (int)
c (int)
d (int)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.number_needed_to_treat(a, b, c, d, confidence=0.95)[source]¶

Number needed to treat (NNT).

$NNT = 1 / |RD|$

Parameters:

a (int)
b (int)
c (int)
d (int)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.number_needed_to_harm(a, b, c, d, confidence=0.95)[source]¶

Number needed to harm (NNH) – same as NNT but with reversed sign convention.

Parameters:

a (int)
b (int)
c (int)
d (int)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.rate_ratio(events1, person_time1, events2, person_time2, confidence=0.95)[source]¶

Incidence rate ratio.

$IRR = (e_1/PT_1) / (e_2/PT_2)$

Parameters:

events1 (int, float) – Events and person-time in group 1.
person_time1 (int, float) – Events and person-time in group 1.
events2 (int, float) – Events and person-time in group 2.
person_time2 (int, float) – Events and person-time in group 2.
confidence (float, default 0.95)

Return type:

morie.effect_sizes.incidence_rate_difference(events1, person_time1, events2, person_time2, confidence=0.95)[source]¶

Incidence rate difference.

$IRD = (e_1/PT_1) - (e_2/PT_2)$

Parameters:

events1 (int, float)
person_time1 (int, float)
events2 (int, float)
person_time2 (int, float)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.cohens_w(observed, expected=None)[source]¶

Cohen’s w for chi-squared.

$w = \sqrt{\chi^2 / N}$

Parameters:

observed (array-like) – Observed frequencies.
expected (array-like or None) – Expected frequencies (uniform if None).

Return type:

morie.effect_sizes.cohens_f(eta2)[source]¶

Cohen’s f from eta-squared.

$f = \sqrt{\eta^2 / (1 - \eta^2)}$

Parameters:: eta2 (float) – Eta-squared value.
Return type:: EffectSizeResult

morie.effect_sizes.cramers_v(contingency_table, confidence=0.95)[source]¶

Cramer’s V for a contingency table.

$V = \sqrt{\chi^2 / (N \cdot (k - 1))}$ where $k = \min(r, c)$.

Parameters:

contingency_table (array-like or DataFrame)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.phi_coefficient(contingency_table)[source]¶

Phi coefficient for a 2x2 contingency table.

$\phi = \sqrt{\chi^2 / N}$

Parameters:: contingency_table (array-like)
Return type:: EffectSizeResult

morie.effect_sizes.rank_biserial_correlation(x, y, confidence=0.95)[source]¶

Rank-biserial correlation for two independent samples.

Glass (1965): positive when x tends to exceed y (equal to Cliff’s delta).

$r = 2U / (n_1 n_2) - 1$ where U is the Mann–Whitney statistic of x.

Parameters:

x (array-like)
y (array-like)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.cliffs_delta(x, y, confidence=0.95)[source]¶

Cliff’s delta (non-parametric effect size).

$\delta = (\#(x_i > y_j) - \#(x_i < y_j)) / (n_x n_y)$

Parameters:

x (array-like)
y (array-like)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.vargha_delaney_a(x, y, confidence=0.95)[source]¶

Vargha–Delaney A statistic.

$A = U / (n_1 n_2)$ where U is the Mann–Whitney statistic (probability that a randomly chosen x exceeds a randomly chosen y).

Parameters:

x (array-like)
y (array-like)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.standardized_coefficients(X, y)[source]¶

Compute standardised regression coefficients (beta weights).

Standardises X and y to zero mean and unit variance before OLS.

Parameters:

X (array-like or DataFrame) – Predictor matrix (n x p).
y (array-like or Series) – Outcome variable.

Returns:

Columns: variable, beta, se, t, p_value.

Return type:

DataFrame

morie.effect_sizes.coefficient_of_variation(x)[source]¶

Coefficient of variation (CV).

$CV = s / \bar{x}$

Parameters:: x (array-like)
Return type:: EffectSizeResult

morie.effect_sizes.variance_ratio(x, y, confidence=0.95)[source]¶

Variance ratio (F-test for equality of variances).

Parameters:

x (array-like)
y (array-like)
confidence (float, default 0.95)

Return type:

morie.effect_sizes.d_to_r(d, n1=None, n2=None)[source]¶

Convert Cohen’s d to Pearson r.

$r = d / \sqrt{d^2 + a}$ where $a = (n_1 + n_2)^2 / (n_1 n_2)$ or a = 4 when sample sizes are unknown.

Parameters:

d (float)
n1 (int or None)
n2 (int or None)

Return type:

morie.effect_sizes.r_to_d(r)[source]¶

Convert Pearson r to Cohen’s d.

$d = 2r / \sqrt{1 - r^2}$

Parameters:: r (float)
Return type:: float

morie.effect_sizes.or_to_d(or_val)[source]¶

Convert odds ratio to Cohen’s d (Hasselblad & Hedges, 1995).

$d = \log(OR) \cdot \sqrt{3} / \pi$

Parameters:: or_val (float) – Odds ratio.
Return type:: float

morie.effect_sizes.d_to_or(d)[source]¶

Convert Cohen’s d to an odds ratio.

$OR = \exp(d \pi / \sqrt{3})$

Parameters:: d (float)
Return type:: float

morie.effect_sizes.or_to_r(or_val)[source]¶

Convert odds ratio to Pearson r via d.

Parameters:: or_val (float)
Return type:: float

morie.effect_sizes.r_to_or(r)[source]¶

Convert Pearson r to odds ratio via d.

Parameters:: r (float)
Return type:: float

morie.effect_sizes.d_to_nnt(d, base_rate=0.5)[source]¶

Convert Cohen’s d to NNT given a base rate.

Uses the Kraemer & Kupfer (2006) formula: $NNT = 1 / (\Phi(d/2 + \Phi^{-1}(CER)) - CER)$

Parameters:

d (float)
base_rate (float, default 0.5) – Control event rate.

Return type:

morie.effect_sizes.fixed_effects_meta(estimates, standard_errors, confidence=0.95)[source]¶

Fixed-effects (inverse-variance weighted) meta-analytic pooling.

Parameters:

estimates (array-like) – Effect-size estimates from k studies.
standard_errors (array-like) – Standard errors of the estimates.
confidence (float, default 0.95)

Return type:

morie.effect_sizes.random_effects_meta(estimates, standard_errors, confidence=0.95, method='DL')[source]¶

Random-effects meta-analytic pooling.

Parameters:

estimates (array-like)
standard_errors (array-like)
confidence (float, default 0.95)
method (str, default "DL") – Tau-squared estimator: "DL" (DerSimonian–Laird), "PM" (Paule–Mandel) or "REML".

Return type:

References

DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7(3), 177–188.

morie.effect_sizes.i_squared(estimates, standard_errors)[source]¶

Compute Higgins’ I-squared heterogeneity statistic.

$I^2 = \max(0, (Q - (k-1))/Q) \times 100$

Parameters:

estimates (array-like)
standard_errors (array-like)

Returns:

Percentage (0–100).

Return type:

morie.effect_sizes.prediction_interval(estimates, standard_errors, confidence=0.95)[source]¶

Prediction interval for a new study from a random-effects meta-analysis.

Parameters:

estimates (array-like)
standard_errors (array-like)
confidence (float, default 0.95)

Returns:

(lower, upper) prediction interval bounds.

Return type:

morie.effect_sizes.bootstrap_effect_size_ci(func, *arrays, n_boot=2000, confidence=0.95, seed=42)[source]¶

Generic bootstrap CI wrapper for any effect-size function.

Parameters:

func (callable) – Function that takes one or more arrays and returns a scalar.
*arrays (array-like) – Input arrays to bootstrap.
n_boot (int, default 2000)
confidence (float, default 0.95)
seed (int, default 42)

Return type:

Datasets¶

Dataset-agnostic analysis engine: ingest, profile, and plan analyses for any dataset.

This module provides tools to ingest datasets in multiple formats (CSV, TSV, Excel, Parquet, JSON) without prior schema knowledge, infer levels of measurement (NOIR: Nominal, Ordinal, Interval, Ratio) for each column, auto-detect key variable roles (treatment, outcome, weight, stratum, cluster), produce comprehensive dataset profiles, and suggest analysis plans.

The measurement-level inference follows Stevens’ (1946) taxonomy as refined for applied statistical practice:

Nominal — unordered categories (e.g., province, gender)
Ordinal — ordered categories (e.g., Likert scales, education level)
Interval — numeric with no true zero (e.g., temperature, year, index)
Ratio — numeric with true zero (e.g., count, weight, income, eBAC)

References

Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677–680. https://doi.org/10.1126/science.103.2684.677

Velleman, P. F., & Wilkinson, L. (1993). Nominal, ordinal, interval, and ratio typologies are misleading. The American Statistician, 47(1), 65–72. https://doi.org/10.1080/00031305.1993.10475938

class morie.dataset.MeasurementLevel(*values)[source]¶

Bases: Enum

Stevens’ levels of measurement (NOIR taxonomy).

Level	Properties	Example variables
NOMINAL	Identity only (= / !=)	Province, gender, treatment indicator
ORDINAL	Identity + order (< / >)	Likert scale, education level, income bracket
INTERVAL	Identity + order + equal intervals, no true zero	Temperature (C), year, standardized score
RATIO	Identity + order + equal intervals + true zero	Count, weight, income, eBAC

References

Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677–680.

NOMINAL = 'nominal'¶

ORDINAL = 'ordinal'¶

INTERVAL = 'interval'¶

RATIO = 'ratio'¶

class morie.dataset.ColumnProfile(name, dtype, level, n_unique, missing_pct, is_binary, is_constant, suggested_role, summary_stats=<factory>)[source]¶

Bases: object

Profile of a single column in the dataset.

Variables:

name (str) – Column name.
dtype (str) – Pandas dtype as a string.
level (MeasurementLevel) – Inferred Stevens measurement level.
n_unique (int) – Number of distinct non-null values.
missing_pct (float) – Percentage of missing values (0.0–100.0).
is_binary (bool) – True if column has exactly two unique non-null values.
is_constant (bool) – True if column has at most one unique non-null value.
suggested_role (str) – One of: "treatment", "outcome", "covariate", "weight", "id", "stratum", "cluster", or "unknown".
summary_stats (dict) – For numeric columns: {mean, std, min, q25, median, q75, max}. For categorical columns: {top_counts} as a dict of value->count.

Parameters:

name (str)
dtype (str)
level (MeasurementLevel)
n_unique (int)
missing_pct (float)
is_binary (bool)
is_constant (bool)
suggested_role (str)
summary_stats (dict)

name: str¶

dtype: str¶

level: MeasurementLevel¶

n_unique: int¶

missing_pct: float¶

is_binary: bool¶

is_constant: bool¶

suggested_role: str¶

summary_stats: dict¶

class morie.dataset.DatasetProfile(n_rows, n_cols, columns, suggested_treatment=None, suggested_outcome=None, suggested_weights=None)[source]¶

Bases: object

Full profile of a dataset including all column profiles and summary diagnostics.

Variables:

n_rows (int) – Number of rows.
n_cols (int) – Number of columns.
columns (dict[str, ColumnProfile]) – Per-column profiles keyed by column name.
suggested_treatment (str | None) – Best-guess treatment column, or None if not detected.
suggested_outcome (str | None) – Best-guess outcome column, or None if not detected.
suggested_weights (str | None) – Best-guess survey-weight column, or None if not detected.

Parameters:

n_rows (int)
n_cols (int)
columns (dict[str, ColumnProfile])
suggested_treatment (str | None)
suggested_outcome (str | None)
suggested_weights (str | None)

n_rows: int¶

n_cols: int¶

columns: dict[str, ColumnProfile]¶

suggested_treatment: str | None = None¶

suggested_outcome: str | None = None¶

suggested_weights: str | None = None¶

to_dict()[source]¶

Serialize the profile to a plain dictionary.

Returns:: Nested dictionary representation suitable for JSON serialization.
Return type:: dict

summary_table()[source]¶

Return a human-readable summary table.

Uses rich Table if available; falls back to a plain-text tabular layout otherwise.

Returns:: Formatted summary string.
Return type:: str

morie.dataset.infer_measurement_level(series, *, ordinal_threshold=10)[source]¶

Infer the Stevens measurement level (NOIR) for a single column.

Detection rules (applied in order):

dtype object / category + ordinal name heuristic + n_unique <= ordinal_threshold: ORDINAL
dtype object / category (default): NOMINAL
dtype bool: NOMINAL
dtype int/float + is_binary (n_unique <= 2): NOMINAL (binary)
dtype int/float + n_unique <= 20 + ordinal name heuristic: ORDINAL
dtype float + interval name heuristic: INTERVAL
dtype float: RATIO
dtype int + min >= 0: RATIO
dtype int + min < 0: INTERVAL
datetime: INTERVAL

Parameters:

series (pandas.Series) – Column data to classify.
ordinal_threshold (int, optional) – Maximum number of unique values for a categorical column to be considered ordinal (default 10).

Returns:

Inferred measurement level.

Return type:

MeasurementLevel

Examples

>>> from morie.fn import _frame_core as pd
>>> infer_measurement_level(pd.Series(["M", "F", "M", "F"]))
<MeasurementLevel.NOMINAL: 'nominal'>

>>> infer_measurement_level(pd.Series([1, 2, 3, 4, 5], name="likert_q1"))
<MeasurementLevel.ORDINAL: 'ordinal'>

morie.dataset.profile_dataset(df, *, hint_treatment=None, hint_outcome=None, hint_weights=None, ordinal_threshold=10, binary_threshold=2)[source]¶

Fully profile a DataFrame without prior schema knowledge.

Walks every column, infers its measurement level and epidemiological role, computes summary statistics, and auto-detects likely treatment, outcome, and survey-weight columns. User-supplied hints override heuristic detection.

Parameters:

df (pandas.DataFrame) – Input data.
hint_treatment (str, optional) – If provided, use this column as the treatment indicator.
hint_outcome (str, optional) – If provided, use this column as the outcome.
hint_weights (str, optional) – If provided, use this column as the survey weight.
ordinal_threshold (int, optional) – Maximum unique values for categorical columns to be classified as ordinal (default 10).
binary_threshold (int, optional) – Maximum unique values for a column to be considered binary (default 2).

Returns:

Complete dataset profile.

Return type:

DatasetProfile

Raises:

TypeError – If df is not a pandas DataFrame.
ValueError – If df has zero rows or zero columns.

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({
...     "treatment": [0, 1, 0, 1],
...     "outcome": [1.2, 3.4, 2.1, 4.5],
...     "age": [25, 30, 35, 40],
... })
>>> profile = profile_dataset(df)
>>> profile.n_rows
4
>>> profile.suggested_treatment
'treatment'

morie.dataset.load_dataset(path, *, encoding='utf-8', **read_kwargs)[source]¶

Load a dataset from CSV, TSV, Excel, Parquet, or JSON file.

File format is detected from the extension. Supported extensions: .csv, .tsv, .xlsx / .xls, .parquet / .pq, .json / .jsonl.

Parameters:

path (str or Path) – File path to the dataset.
encoding (str, optional) – Character encoding for text-based formats (default "utf-8").
**read_kwargs – Additional keyword arguments forwarded to the pandas reader.

Returns:

Loaded dataset.

Return type:

Raises:

FileNotFoundError – If path does not exist.
ValueError – If the file extension is not recognized.

Examples

>>> import os, tempfile
>>> path = os.path.join(tempfile.mkdtemp(), "data.csv")
>>> with open(path, "w") as fh:
...     print("id,score", "1,3.5", "2,4.0", sep=chr(10), file=fh)
>>> df = load_dataset(path)
>>> df.shape
(2, 2)

A workbook takes its sheet by name:

>>> df = load_dataset("survey.xlsx", sheet_name="wave1")

morie.dataset.suggest_analysis_plan(profile)[source]¶

Suggest an ordered analysis plan based on the detected variable types.

Uses the inferred measurement levels, binary indicators, and detected treatment/outcome/weight columns to recommend epidemiological analyses.

Parameters:

profile (DatasetProfile) – Output from profile_dataset().

Returns:

Ordered list of suggested analyses. Each dict contains:

analysis (str): method identifier (e.g., "ipw_ate")
rationale (str): plain-English reason this analysis is suggested
required_vars (dict): mapping of role -> column name(s)

Return type:

list[dict[str, Any]]

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({"treatment": [0, 1, 0, 1], "outcome": [1.2, 3.4, 2.1, 4.5]})
>>> plan = suggest_analysis_plan(profile_dataset(df))
>>> plan[0]["analysis"]
'descriptive_profile'

morie.data.morie_db()[source]¶

Return path to morie.db – checks package-bundled location first, then cache.

Package-bundled DB ships with the installed package (via LFS/setuptools). Cache location is used for development and is gitignored.

Return type:: Path

morie.data.cache_connect(db_path=None)[source]¶

Open (or create) the MORIE SQLite cache database.

Parameters:: db_path (str | Path | None)
Return type:: Connection

morie.data.cache_store(df, table, db_path=None, quiet=False)[source]¶

Write a DataFrame to the SQLite cache, replacing any existing table.

Parameters:

df (DataFrame)
table (str)
db_path (str | Path | None)
quiet (bool)

Return type:

morie.data.cache_load(table, db_path=None)[source]¶

Load a table from the SQLite cache. Returns None if not cached.

Parameters:

table (str)
db_path (str | Path | None)

Return type:

DataFrame | None

morie.data.cache_list(db_path=None)[source]¶

List all cached tables with row counts.

Parameters:: db_path (str | Path | None)
Return type:: list[dict[str, Any]]

morie.data.wayback_snapshot_url(url, timestamp=None, timeout=15)[source]¶

The closest Internet Archive snapshot of url (https), or None.

Mirrors rmoriebricklayer::wbsnpurl(): asks archive.org/wayback/available and returns the snapshot URL only when the archive reports it available.

Parameters:

url (str)
timestamp (str | None)
timeout (int)

Return type:

str | None

morie.data.download_with_wayback(url, timeout=60)[source]¶

Fetch url; if the live site fails, the closest Wayback snapshot.

Returns the bytes and the URL they came from. Raises the live error when no snapshot exists either.

Parameters:

url (str)
timeout (int)

Return type:

tuple[bytes, str]

morie.data.fetch_ckan_to_cache(dataset_key='cpads', limit=32000, db_path=None, timeout=60, max_records=None)[source]¶

Fetch a dataset from CKAN and store it in the SQLite cache.

Parameters:

dataset_key (str) – Key in CKAN_DATASETS (e.g., “cpads”, “csads”, “csus”).
limit (int) – Records per page of the CKAN DataStore API.
max_records (int, optional) – Stop once this many records have been fetched (morie download-bootstrap --limit).
db_path (str | Path | None) – Override cache database path.
timeout (int) – HTTP timeout in seconds.

Returns:

The fetched (and optionally canonicalized) DataFrame.

Return type:

pd.DataFrame

exception morie.data.RObjectSavedError(key, path)[source]¶

Bases: NotImplementedError

The dataset is an R object (RData/rds): Python cannot open it, but it is saved at path for R.

Parameters:

key (str)
path (Path)

morie.data.list_rmoriedata(timeout=120)[source]¶

The rmoriedata catalog: one dict per slug (slug, source_path, kind, n_rows, n_cols).

Parameters:: timeout (int)
Return type:: list[dict]

morie.data.load_rmoriedata(slug, timeout=120)[source]¶

Load one rmoriedata table by slug (list_rmoriedata() lists them).

Reads the CSV (or CSV.gz) that rmoriedata ships, applying the column names from its schema exactly as rmoriedata::mdload() does.

Parameters:

slug (str)
timeout (int)

Return type:

DataFrame

morie.data.fetch_cihi_indicator_library(timeout=120)[source]¶

CIHI’s Indicator Library, all indicators: the public workbook, fetched once.

The workbook is 72 MB and 809,000 rows by 33 columns. Loading it as a worksheet object needs more than 10 GB, so it is streamed row by row into a CSV next to it (once) and read from there. A copy you already have under MORIE_DATA_DIR is used instead of downloading.

Parameters:: timeout (int)
Return type:: DataFrame

morie.data.cached_cpads()[source]¶

The real CPADS PUMF already in the built-in store (morie pull cpads), or None. No network.

Return type:: DataFrame | None

morie.data.load_cpads(db_path=None, timeout=60)[source]¶

Load CPADS data: try local files, then cache, then CKAN API.

Resolution order: 1. Local RDS (via R bridge) or CSV files in standard locations 2. SQLite cache (data/cache/morie.db) 3. CKAN API fetch -> cache -> return

Parameters:

db_path (str | Path | None)
timeout (int)

Return type:

DataFrame

morie.data.synthetic_mapq_panel(n=400, seed=2026)[source]¶

Deterministic synthetic MAPQII panel: a toy stand-in for TKARONTOMAPQ.xlsx.

The same design as rmorie’s .morie_mapq_synth_panel(): 20 Likert items (1-5) in four subscales (EE, EA, UA, ER) with a planted one-factor-per-subscale structure, gender_male and age, subscale scores, and a Knowledge Scale score ks_score driven by epistemic attitudes, gender and age, so the psychometric and DML stages have a real signal to recover. Participant-level MAPQ data are not distributed; this panel is.

Parameters:

n (int)
seed (int)

Return type:

DataFrame

morie.data.list_datasets(db_path=None, *, hosted=True)[source]¶

List all datasets with their cache status.

Returns a list of dicts with keys: key, name, source, survey, year, type, cached (bool), rows (int or None).

When db_path is explicitly provided, only that database is queried (the built-in DB is skipped). This keeps unit tests deterministic.

Parameters:

db_path (str | Path | None)
hosted (bool)

Return type:

list[dict]

morie.data.dataset_route(entry)[source]¶

How a catalog entry is obtained: a portal it downloads from, rmoriedata, or your own file.

Parameters:: entry (dict)
Return type:: str

morie.data.dataset_info(key)[source]¶

Return full metadata for a dataset: a catalog key, or a curated db/table key.

Parameters:: key (str)
Return type:: dict

class morie.data.DatasetRegistry(data_dir='data/datasets/')[source]¶

Bases: object

A registry to manage, catalog, and load secure epidemiological datasets.

Parameters:: data_dir (str)

__init__(data_dir='data/datasets/')[source]¶

Initialize the dataset registry.

Parameters:: data_dir (str) – The root file directory containing secure datasets, defaults to “data/datasets/”.

register_dataset(name, metadata)[source]¶

Register a dataset in the internal catalog.

Parameters:

name (str) – Unique identifier for the dataset.
metadata (dict) – A dictionary mapping metadata values (like name, path, format).

list_datasets()[source]¶

Return a copy of the registered dataset catalog.

Returns:: Dataset metadata keyed by registry identifier.
Return type:: dict[str, dict[str, Any]]

get_dataset_metadata(name)[source]¶

Return a copy of the metadata for one registered dataset.

Parameters:: name (str)
Return type:: dict[str, Any]

fetch_ckan_records(name, *, limit=5, query=None, timeout=30)[source]¶

Fetch records for a dataset backed by a CKAN DataStore resource.

Parameters:

name (str)
limit (int)
query (str | None)
timeout (int)

Return type:

fetch_ckan_dataframe(name, *, limit=100, query=None, timeout=30)[source]¶

Fetch a CKAN result set and return the records as a DataFrame.

Parameters:

name (str)
limit (int)
query (str | None)
timeout (int)

Return type:

DataFrame

cpads_contract()[source]¶

Return the canonical local-private CPADS contract.

Return type:: dict[str, Any]

register_local_cpads(path, *, name='cpads_local')[source]¶

Register a user-provided local CPADS file.

Parameters:

path (str | Path)
name (str)

Return type:

validate_cpads_frame(frame, *, strict=True)[source]¶

Validate a DataFrame against the canonical CPADS variable contract.

Parameters:

frame (DataFrame)
strict (bool)

Return type:

load(name)[source]¶

Load a registered dataset securely.

Parameters:

name (str) – Unique identifier for the dataset in the catalog.

Raises:

ValueError – If the dataset name is not in the registry catalog.
FileNotFoundError – If the underlying file could not be queried or synced locally.
NotImplementedError – If the specified file format is not supported.

Returns:

A pandas DataFrame containing the dataset content.

Return type:

morie.data.dataset_recommendation(key, entry=None)[source]¶

Human-readable guidance on how to obtain a catalogued dataset.

Surfaced in load_dataset() errors and check_datasets() output, so a user who hits an unavailable dataset is told where it comes from and what to do – morie ships code, not the data, but it can always point at the source.

Parameters:

key (str) – Catalogue key (e.g. "cchs22", "siu").
entry (dict, optional) – The catalogue entry; looked up from DATASET_CATALOG when omitted.

Returns:

A multi-line recommendation.

Return type:

morie.data.check_datasets(*, probe_remote=False, db_path=None, timeout=15)[source]¶

Audit the availability of every catalogued dataset.

Classifies each entry in DATASET_CATALOG by where it loads from:

bundled – shipped in the built-in morie database.
cached – present in the user cache.
local – a local file exists at local_path.
remote – not available locally, but a CKAN resource id or a fetcher is registered, so it can be fetched on demand.
MISSING – no working source; the entry needs attention (a missing file together with an empty ckan_resource_id and no fetcher).

With probe_remote=True every remote dataset is actually fetched once, to confirm the link is still live (remote-ok) or not (remote-dead).

Returns:

A per-dataset table and per-tier counts; the headline value is the number of datasets that load with no attention needed.

Return type:

Parameters:

probe_remote (bool)
db_path (str | Path | None)
timeout (int)

morie.data.load_dataset(key, *, db_path=None, timeout=60)[source]¶

Load a dataset by catalog key.

Resolution order: 1. Built-in morie.db (ships with package) 2. User cache data/cache/morie.db 3. Local file (ingest to cache on the fly) 4. CKAN API (if resource ID available) 5. Error

Supports fuzzy matching: load_dataset("cpads") resolves to ocp21.

Parameters:

key (str)
db_path (str | Path | None)
timeout (int)

Return type:

DataFrame

BigQuery-derived dataset analysis over a local SQLite catalog.

This module gives the morie analytics surface a typed entry point into a SQLite-mirrored slice of public BigQuery datasets – useful for local, network-free exploration and CI work.

Two access paths:

Local – point at the on-disk SQLite file directly. Fast, no network. Works against the bundled morie_datasets.db plus any user-supplied SQLite mirrors of public BigQuery slices.

Remote – a SQL-over-HTTP endpoint configured via the MORIE_REMOTE_URL env var. Useful for exploration without copying GBs of SQLite locally.

The high-level helpers (bq_describe, bq_columns, bq_summary, bq_correlate) auto-pick local first, fall through to remote when configured.

class morie.bq.TableInfo(db, table, row_count, columns, sample)[source]¶

Bases: object

High-level summary of one table inside a BigQuery-mirror database.

Parameters:

db (str)
table (str)
row_count (int)
columns (list[dict[str, str]])
sample (list[dict[str, Any]])

db: str¶

table: str¶

row_count: int¶

columns: list[dict[str, str]]¶

sample: list[dict[str, Any]]¶

class morie.bq.ColumnSummary(db, table, column, n, n_missing, mean, median, std, min, max)[source]¶

Bases: object

Univariate descriptive stats for one numeric column.

Parameters:

db (str)
table (str)
column (str)
n (int)
n_missing (int)
mean (float | None)
median (float | None)
std (float | None)
min (float | None)
max (float | None)

db: str¶

table: str¶

column: str¶

n: int¶

n_missing: int¶

mean: float | None¶

median: float | None¶

std: float | None¶

min: float | None¶

max: float | None¶

morie.bq.bq_query(db, sql)[source]¶

Run an arbitrary SQL query against the BigQuery-mirror database db.

Auto-picks local SQLite if available; falls through to the configured remote endpoint otherwise. Returns a list of row dicts.

Examples

Needs MORIE_REMOTE_URL (a query endpoint) or MORIE_LOCAL_DB_DIR (a local SQLite mirror):

>>> rows = bq_query("nyc_311_historic", "SELECT borough, count(*) FROM nyc_311_historic GROUP BY 1")

Parameters:

db (str)
sql (str)

Return type:

list[dict[str, Any]]

morie.bq.bq_tables(db)[source]¶

List every table in the database.

Parameters:: db (str)
Return type:: list[str]

morie.bq.bq_columns(db, table=None)[source]¶

Schema for table (defaults to the first user table in the db).

Parameters:

db (str)
table (str | None)

Return type:

list[dict[str, str]]

morie.bq.bq_describe(db, table=None)[source]¶

High-level summary: row count, schema, first 5 rows.

Parameters:

db (str)
table (str | None)

Return type:

TableInfo

morie.bq.bq_summary(db, table, column)[source]¶

Univariate stats for a numeric column. NaN-safe.

Parameters:

db (str)
table (str)
column (str)

Return type:

ColumnSummary

morie.bq.bq_correlate(db, table, x, y)[source]¶

Pearson correlation between two numeric columns (NaN-safe).

Parameters:

db (str)
table (str)
x (str)
y (str)

Return type:

float | None

OTIS — Offender Tracking Information System¶

morie.otis – Ontario Restrictive Confinement (OTIS) analysis.

Correctional/sociolegal analysis for Ontario placement data (2023-2025). Orchestrates existing MORIE causal inference modules for the correctional domain: DML-IRM, propensity score matching, AIPW, mixed-effects models.

Short-name API (≤6 chars):

rplace() – Regional placement analysis by age/sex/year
astcmb() – Alert-state combination encoding (8 binary -> complexity)
volat() – Regional volatility/movement metric
rctrnd() – Restrictive confinement trends over time
otdesc() – Full OTIS descriptive statistics suite
otdml() – Run DML IRM (ATE/ATT) on OTIS data

Data: data/cache/correctional_stats_report_environment1b.RData (via R): data/cache/dt_expanded.rds (1.9M rows × 13 cols)

References

Ontario Ministry of the Solicitor General (2025). Restrictive Confinement Detailed Dataset. data.ontario.ca.
Jahn v. Ontario (2020). Settlement Agreement – Inmate Data Disclosure.
Chernozhukov et al. (2018). Double/debiased machine learning for treatment and structural parameters. Econometrics Journal.

morie.otis.project_root()[source]¶

Resolve the repo root for OTIS data/output paths.

Walks ancestors of this file for a pyproject.toml marker so the same code works from a source checkout, a git worktree, or an installed wheel. Falls back to the current working directory.

Replaces the old parents[5] hard-coded depth, which resolved to /Volumes from this worktree layout.

Return type:: Path

class morie.otis.RplRes(counts, props, year, sex=None)[source]¶

Bases: object

Regional placement result.

Parameters:

counts (DataFrame)
props (DataFrame)
year (int)
sex (str | None)

counts: DataFrame¶

props: DataFrame¶

year: int¶

sex: str | None = None¶

class morie.otis.AstRes(data, summary)[source]¶

Bases: object

Alert-state combination result.

Parameters:

data (DataFrame)
summary (DataFrame)

data: DataFrame¶

summary: DataFrame¶

class morie.otis.VolRes(data, mean, median)[source]¶

Bases: object

Volatility result.

Parameters:

data (DataFrame)
mean (float)
median (float)

data: DataFrame¶

mean: float¶

median: float¶

class morie.otis.OtDmlR(ate, ate_se, ate_pval, att, att_se, att_pval, n, method='IRM')[source]¶

Bases: object

OTIS DML result.

Parameters:

ate (float)
ate_se (float)
ate_pval (float)
att (float)
att_se (float)
att_pval (float)
n (int)
method (str)

ate: float¶

ate_se: float¶

ate_pval: float¶

att: float¶

att_se: float¶

att_pval: float¶

n: int¶

method: str = 'IRM'¶

morie.otis.rplace(df, year, sex=None, *, id_col='unique_individual_id', age_col='age_category', region_col='region_at_time_of_placement', year_col='end_fiscal_year', gender_col='gender')[source]¶

Regional placement analysis by age group.

Computes count matrix S and proportion matrix P of placements across regions, stratified by age group and optionally by sex.

Parameters:

df (DataFrame) – Correctional placement data.
year (int) – Fiscal year to analyze.
sex (str, optional) – Filter by gender (“Male” or “Female”).
id_col (str)
age_col (str)
region_col (str)
year_col (str)
gender_col (str)

Return type:

RplRes

morie.otis.astcmb(df, *, alert_cols=('mental_health_alert', 'suicide_risk_alert', 'suicide_watch_alert'), id_col='unique_individual_id', year_col='end_fiscal_year', np_col='number_of_placements')[source]¶

Alert-state combination encoding.

Encodes 3 binary alert indicators into 8 possible combinations (a1-a8) and computes a complexity index (count of distinct combinations per person-year).

Parameters:

df (DataFrame) – Must contain alert columns (Yes/No or 1/0).
alert_cols (tuple[str, ...])
id_col (str)
year_col (str)
np_col (str)

Return type:

AstRes

morie.otis.volat(df, *, id_col='unique_individual_id', year_col='end_fiscal_year', regA_col='region_at_time_of_placement', regB_col='region_most_recent_placement')[source]¶

Regional volatility/movement metric.

Counts the number of distinct regions an individual was placed in (combining initial and most recent placement regions).

Parameters:

df (DataFrame)
id_col (str)
year_col (str)
regA_col (str)
regB_col (str)

Return type:

VolRes

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({
...     "unique_individual_id": [1, 1, 2, 2, 3], "end_fiscal_year": [2020, 2021, 2020, 2021, 2021],
...     "region_at_time_of_placement": ["A", "A", "B", "B", "C"], "region_most_recent_placement": ["A", "B", "B", "B", "C"],
... })
>>> r = volat(df)
>>> (r.mean, r.median)
(1.2, 1.0)

morie.otis.rctrnd(df, *, id_col='unique_individual_id', year_col='end_fiscal_year', region_col='region_at_time_of_placement')[source]¶

Restrictive confinement trends over time.

Returns per-year counts by region.

Parameters:

df (DataFrame)
id_col (str)
year_col (str)
region_col (str)

Returns:

DataFrame with columns

Return type:

year, region, n_individuals, n_placements

morie.otis.otdesc(df, *, id_col='unique_individual_id', year_col='end_fiscal_year')[source]¶

Full OTIS descriptive statistics suite.

Returns:

dict with keys (n_total, n_by_year, n_by_region, n_by_age,)
n_by_gender, placement_dist

Parameters:

df (DataFrame)
id_col (str)
year_col (str)

Return type:

morie.otis.otdml(df, outcome='Y', treatment='D', covariates=None, *, cluster=None, n_folds=3, seed=123)[source]¶

Run DML IRM (ATE/ATT) on OTIS data.

Wraps morie.effects.estimate_irm() for the correctional domain.

Parameters:

df (DataFrame) – Must contain outcome, treatment, and covariate columns.
outcome (str) – Outcome variable name (default “Y” = suicide risk).
treatment (str) – Treatment variable name (default “D” = mental health alert).
covariates (list of str, optional) – Covariate column names. Defaults to standard OTIS set.
cluster (str, optional) – Cluster variable for clustered standard errors.
n_folds (int)
seed (int)

Return type:

OtDmlR

morie.otis.regional_placement(df, year, sex=None, *, id_col='unique_individual_id', age_col='age_category', region_col='region_at_time_of_placement', year_col='end_fiscal_year', gender_col='gender')¶

Regional placement analysis by age group.

Computes count matrix S and proportion matrix P of placements across regions, stratified by age group and optionally by sex.

Parameters:

df (DataFrame) – Correctional placement data.
year (int) – Fiscal year to analyze.
sex (str, optional) – Filter by gender (“Male” or “Female”).
id_col (str)
age_col (str)
region_col (str)
year_col (str)
gender_col (str)

Return type:

RplRes

morie.otis.alert_state_combo(df, *, alert_cols=('mental_health_alert', 'suicide_risk_alert', 'suicide_watch_alert'), id_col='unique_individual_id', year_col='end_fiscal_year', np_col='number_of_placements')¶

Alert-state combination encoding.

Encodes 3 binary alert indicators into 8 possible combinations (a1-a8) and computes a complexity index (count of distinct combinations per person-year).

Parameters:

df (DataFrame) – Must contain alert columns (Yes/No or 1/0).
alert_cols (tuple[str, ...])
id_col (str)
year_col (str)
np_col (str)

Return type:

AstRes

morie.otis.volatility(df, *, id_col='unique_individual_id', year_col='end_fiscal_year', regA_col='region_at_time_of_placement', regB_col='region_most_recent_placement')¶

Regional volatility/movement metric.

Counts the number of distinct regions an individual was placed in (combining initial and most recent placement regions).

Parameters:

df (DataFrame)
id_col (str)
year_col (str)
regA_col (str)
regB_col (str)

Return type:

VolRes

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({
...     "unique_individual_id": [1, 1, 2, 2, 3], "end_fiscal_year": [2020, 2021, 2020, 2021, 2021],
...     "region_at_time_of_placement": ["A", "A", "B", "B", "C"], "region_most_recent_placement": ["A", "B", "B", "B", "C"],
... })
>>> r = volat(df)
>>> (r.mean, r.median)
(1.2, 1.0)

morie.otis.rc_trends(df, *, id_col='unique_individual_id', year_col='end_fiscal_year', region_col='region_at_time_of_placement')¶

Restrictive confinement trends over time.

Returns per-year counts by region.

Parameters:

df (DataFrame)
id_col (str)
year_col (str)
region_col (str)

Returns:

DataFrame with columns

Return type:

year, region, n_individuals, n_placements

morie.otis.otis_descriptives(df, *, id_col='unique_individual_id', year_col='end_fiscal_year')¶

Full OTIS descriptive statistics suite.

Returns:

dict with keys (n_total, n_by_year, n_by_region, n_by_age,)
n_by_gender, placement_dist

Parameters:

df (DataFrame)
id_col (str)
year_col (str)

Return type:

morie.otis.otis_dml(df, outcome='Y', treatment='D', covariates=None, *, cluster=None, n_folds=3, seed=123)¶

Run DML IRM (ATE/ATT) on OTIS data.

Wraps morie.effects.estimate_irm() for the correctional domain.

Parameters:

df (DataFrame) – Must contain outcome, treatment, and covariate columns.
outcome (str) – Outcome variable name (default “Y” = suicide risk).
treatment (str) – Treatment variable name (default “D” = mental health alert).
covariates (list of str, optional) – Covariate column names. Defaults to standard OTIS set.
cluster (str, optional) – Cluster variable for clustered standard errors.
n_folds (int)
seed (int)

Return type:

OtDmlR

OTIS analysis surfaces – RichResult wrappers around morie.otis primitives.

morie.otis provides 6 result-emitting callables that return specialized dataclasses (RplRes / AstRes / VolRes / OtDmlR / etc.). This module wraps each in a RichResult so OTIS analyses get the same multi-section output that Welch / Mann–Whitney / SIU analyses already emit.

OTIS here = the public “Data on Inmates in Ontario” detailed records (Ontario Ministry of the Solicitor General), extracted from the Offender Tracking Information System (OTIS); open data under the Open Government Licence – Ontario. The canonical 76,934-row table lives in the repo cache at data/cache/correctional_stats_report_environment.RData as the R object df. Use load_otis() (CSV mirror) or call the Rscript exporter at scripts/export_otis_csv.R to refresh the CSV.

Public API:: rplace(df, year, sex=None) -> RichResult astcmb(df) -> RichResult volat(df) -> RichResult rctrnd(df) -> RichResult otdesc(df, year=None, sex=None) -> RichResult otdml(df, treatment, outcome, …) -> RichResult all_analyses(df, year, *, out_dir=None) -> dict[str, RichResult]

morie.otis_analyze.load_otis(csv_path=None)[source]¶

Load the canonical OTIS DataFrame.

Defaults to data/cache/otis_main.csv (Rscript-exported from the correctional_stats_report_environment.RData fixture). Pass a custom csv_path to load a refreshed snapshot.

Returns a 10-column DataFrame:: end_fiscal_year, unique_individual_id, region_at_time_of_placement, region_most_recent_placement, gender, age_category, mental_health_alert, suicide_risk_alert, suicide_watch_alert, number_of_placements

Parameters:: csv_path (Path | str | None)
Return type:: DataFrame

morie.otis_analyze.rplace(df, year, sex=None, **kw)[source]¶

Regional placement analysis as RichResult.

Parameters:

df (DataFrame)
year (int)
sex (str | None)
kw (Any)

Return type:

morie.otis_analyze.astcmb(df, **kw)[source]¶

Alert-state combination analysis as RichResult.

Parameters:

df (DataFrame)
kw (Any)

Return type:

morie.otis_analyze.volat(df, **kw)[source]¶

Regional volatility metric as RichResult.

Parameters:

df (DataFrame)
kw (Any)

Return type:

morie.otis_analyze.rctrnd(df, **kw)[source]¶

Restrictive-confinement trends as RichResult.

Parameters:

df (DataFrame)
kw (Any)

Return type:

morie.otis_analyze.otdesc(df, **kw)[source]¶

Full OTIS descriptive suite as RichResult.

_o.otdesc does not accept year/sex filters – apply those upstream via df = df[df.end_fiscal_year == year] if needed.

Parameters:

df (DataFrame)
kw (Any)

Return type:

morie.otis_analyze.otdml(df, *, treatment, outcome, covariates, **kw)[source]¶

OTIS DML IRM (ATE/ATT) as RichResult.

Parameters:

df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
kw (Any)

Return type:

morie.otis_analyze.all_analyses(df, year, *, sex=None, out_dir=None)[source]¶

Run rplace / astcmb / volat / rctrnd / otdesc on df and write to disk.

otdml is excluded from the bundle because it requires the user to specify (treatment, outcome, covariates) – call it separately.

Parameters:

df (DataFrame)
year (int)
sex (str | None)
out_dir (Path | None)

Return type:

morie.otis_all_analyze – RichResult-emitting analyses for ALL 28 OTIS datasets.

This module is the comprehensive OTIS analysis surface. It pairs with morie.otis_datasets (the registry/loader) and extends the narrower morie.otis_analyze (which only covered the restricted- confinement subset).

For each dataset id (b01..d07), this module exposes:: analyze_<id>(df=None) -> RichResult

If df is None, the analysis loads from data/datasets/OTIS/<csv_filename>.

Plus a top-level driver:: analyze_all(*, out_dir=None) -> dict[str, RichResult]

which runs every dataset’s analysis, writes per-dataset .txt and .json under data/manifest/outputs/otis/, and returns the in-memory results.

Plus the Ruhela-formulation (RF) high-level entry points:

analyze_a01_ruhela_formulations(df=None) – full DLRM on a01: – alias: analyze_a01_dlrm
analyze_b01_ruhela_formulations(df=None) – full DLRM on b01: – alias: analyze_b01_dlrm
analyze_b02_ruhela_formulations(df=None) – gender -> seg days on b02: – alias: analyze_b02_dlrm

analyze_a01_with_csi_context() – a01 causal + Toronto CSI analyze_c_chi2() – χ² + Cramer’s V on c-series

– homage to Doob’s chi-square tradition

analyze_d_chi2() – d-series + Alert χ²

Naming abbreviations:

RF – Ruhela formulation RDF – Ruhela Dual Formulation (RF + Naive-arm sensitivity) DLRM – Doob-Levinsky-Ruhela-Medina (methodology attribution)

The DLRM methodology attribution and the wider acknowledgements (Prof. Jauregui, Prof. Laniyonu) live at the top of morie.otis_causal’s module docstring.

morie.otis_all_analyze.analyze_b01(df=None)[source]¶

Person-level segregation placements (76,934 rows).

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b02(df=None)[source]¶

Aggregate days in segregation per person per year.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b03(df=None)[source]¶

Segregation placements by alert × institution.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b04(df=None)[source]¶

Placement durations (max/median/mode) by region & gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b05(df=None)[source]¶

Distribution of placements by binned duration.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b06(df=None)[source]¶

Reasons for placement × institution × gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b07(df=None)[source]¶

Alerts × gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b08(df=None)[source]¶

Durations (median/mode) by institution & gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b09(df=None)[source]¶

Individuals by number of segregation placements × gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c01(df=None)[source]¶

Total individuals × custody/RC/seg × gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c02(df=None)[source]¶

Individuals in RC/seg by institution.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c03(df=None)[source]¶

Individuals × race × gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c04(df=None)[source]¶

Individuals × race × region.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c05(df=None)[source]¶

Individuals × religion × region.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c06(df=None)[source]¶

Individuals × age × region.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c07(df=None)[source]¶

Individuals × alerts × gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c08(df=None)[source]¶

Individuals × religion × gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c09(df=None)[source]¶

Individuals × age × gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c10(df=None)[source]¶

RC/seg aggregate durations by institution.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c11(df=None)[source]¶

Individuals by aggregate-duration bin.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c12(df=None)[source]¶

RC/seg aggregate durations by region & gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_d01(df=None)[source]¶

Person-level deaths in custody.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_d02(df=None)¶

OTIS d02: custodial deaths by year and by Gender.

Parameters:: df (DataFrame | None) – the d02 table; None loads it.
Returns:: A RichResult with the dataset summary, the yearly trend and the Gender-by-year crosstab.
Return type:: RichResult

morie.otis_all_analyze.analyze_d03(df=None)¶

OTIS d03: custodial deaths by year and by Race.

Parameters:: df (DataFrame | None) – the d03 table; None loads it.
Returns:: A RichResult with the dataset summary, the yearly trend and the Race-by-year crosstab.
Return type:: RichResult

morie.otis_all_analyze.analyze_d04(df=None)¶

OTIS d04: custodial deaths by year and by Religion.

Parameters:: df (DataFrame | None) – the d04 table; None loads it.
Returns:: A RichResult with the dataset summary, the yearly trend and the Religion-by-year crosstab.
Return type:: RichResult

morie.otis_all_analyze.analyze_d05(df=None)¶

OTIS d05: custodial deaths by year and by Age_Category.

Parameters:: df (DataFrame | None) – the d05 table; None loads it.
Returns:: A RichResult with the dataset summary, the yearly trend and the Age_Category-by-year crosstab.
Return type:: RichResult

morie.otis_all_analyze.analyze_d06(df=None)[source]¶

OTIS d06: custodial deaths by alert type and medical cause of death.

Parameters:: df (DataFrame | None) – the d06 table; None loads it.
Returns:: A RichResult with the dataset summary and the cause-by-alert crosstab.
Return type:: RichResult

morie.otis_all_analyze.analyze_d07(df=None)[source]¶

OTIS d07: custodial deaths by alert type and housing-unit type.

Parameters:: df (DataFrame | None) – the d07 table; None loads it.
Returns:: A RichResult with the dataset summary and the housing-unit-by-alert crosstab.
Return type:: RichResult

morie.otis_all_analyze.analyze_a01(df=None)[source]¶

High-level a01 (Restrictive Confinement Detailed) analysis.

Wraps the full causal pipeline that the OTIS-RC research runs against a01_restrictive_confinement_detailed_dataset.csv:

Person-year aggregation via 8-state alert-combo encoding (morie.otis_causal.make_pair_alert_to_volatility_a01).

MatchIt 1:1 NN PSM with caliper.

IRM-DML with cross-fitted random-forest nuisances.

Multi-way clustered standard errors (id, region, year).

Returns a RichResult with both ATE and ATTE estimates.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_a01_ruhela_formulations(df=None, *, cluster_col='EndFiscalYear')[source]¶

Ruhela formulations (full DLRM) on a01 – the canonical OTIS-RC dataset.

Runs the complete OTIS-RC methodology arc on the author’s Ruhela formulation (T = high alert complexity, Y = regional volatility). See _ruhela_formulations_on for the full estimator list.

Attribution¶

The Ruhela formulation is the author’s design end-to-end; the supporting- contributor credits (Doob – pointed the author to the dataset; Levinsky – course context, reviewed only the preliminary a01 analysis; Medina – reviewed the formal write-up) are documented in morie.otis_causal’s module docstring.

param df:: a01 DataFrame; if None, otis_datasets.load_otis_dataset('a01').
type df:: pd.DataFrame, optional
param cluster_col:: Cluster axis for cluster-robust SE in the IRM-DML estimator (default ‘EndFiscalYear’).
type cluster_col:: str

Parameters:

df (DataFrame | None)
cluster_col (str)

Return type:

morie.otis_all_analyze.analyze_a01_dual(*args, **kwargs)[source]¶: Deprecated alias for analyze_a01_ruhela_formulations.

morie.otis_all_analyze.analyze_b01_ruhela_formulations(df=None, *, cluster_col='EndFiscalYear')[source]¶

Ruhela formulations (full DLRM) on b01 (Segregation Detailed).

Per-placement segregation complement to a01’s per-day RC records. Same Ruhela alert-complexity -> regional-volatility contrast, larger magnitude effect because segregation is a stronger institutional contrast.

See analyze_a01_ruhela_formulations for the full attribution and estimator catalogue.

Parameters:

df (DataFrame | None)
cluster_col (str)

Return type:

morie.otis_all_analyze.analyze_b02_ruhela_formulations(df=None, *, cluster_col='EndFiscalYear')[source]¶

Ruhela formulation on b02 (Segregation Total Days per individual).

b02 has UniqueIndividual_ID + EndFiscalYear + Gender + Region + Age_Category + TotalAggregatedDays_Segregation but NO alert columns. The canonical alert-complexity formulation is impossible; instead the author’s b02-specific Ruhela formulation tests gender disparity in segregation-day burden:

T = Gender == “Female” indicator Y = TotalAggregatedDays_Segregation (count, person-year) covariates = [Region_MostRecentPlacement, Age_Category, EndFiscalYear]

Same DLRM ensemble as the canonical formulation. Sister formulations on the same b02 data could swap T for region or age.

Parameters:

df (DataFrame | None)
cluster_col (str)

Return type:

morie.otis_all_analyze.analyze_ruhela_per_year(df, *, ds_id, treatment, outcome, covariates, year_col='EndFiscalYear', cluster_col='EndFiscalYear', propensity_calibration='none')[source]¶

Per-fiscal-year full DLRM on each Ruhela formulation.

Runs the complete DLRM (IPW + AIPW + g-comp + PSM-NN + PSM-subclass + IRM-DML + match_first + ATC + PLR + SuperLearner) separately on each fiscal year. ~7× the standard per-year IRM-DML runtime. Mirrors a per-year extension of the author’s res_by_year.

Parameters:

df (DataFrame)
ds_id (str)
treatment (str)
outcome (str)
covariates (list[str])
year_col (str)
cluster_col (str | None)
propensity_calibration (str)

Return type:

morie.otis_all_analyze.analyze_a01_ruhela_per_year(df=None)[source]¶

Per-year full DLRM on Ruhela formulations on a01.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_ruhela_per_year(df=None)[source]¶

Per-year full DLRM on Ruhela formulations on b01.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b03_ruhela_aggregate(df=None)[source]¶

Aggregate Ruhela formulation on b03 (Alert × Institution).

T = Alert_Presence indicator -> Y = Number_SegregationPlacements. Year-FE + Region-FE + Alert_Type-FE; Institution as GEE cluster group (24 levels, marginal model with cluster-robust SE).

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b07_ruhela_aggregate(df=None)[source]¶

Aggregate Ruhela formulation on b07 (alerts × gender summary).

b07 is a wide table with paired columns:: Number_Segregation_Placements_With_Alert Number_Segregation_Placements_Without_Alert

We pivot to long form (one row per alert-state × gender × year), then T = with_alert indicator -> Y = count. Year + Gender + Alert_Type fixed effects.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c01_ruhela_aggregate(df=None)[source]¶

Aggregate Ruhela formulation on c01 – gender disparity in RC.

c01 has Gender × {InCustody, RestrictiveConfinement, Segregation} counts. T = Female indicator -> Y = NumberIndividuals_RC, Year-FE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c07_ruhela_aggregate(df=None)[source]¶

Aggregate Ruhela formulation on c07 – alert × gender -> RC count.

Counterpart of the per-row Ruhela formulation on a01/b01 at the aggregate level. Useful triangulation: c07 reports NumberIndividuals_InCustody / RC / Segregation cross-tabulated by Alert_Type × Gender × FiscalYear. The Ruhela formulation here contrasts an indicator of alert presence (vs. no-alert reference) with the number-in-RC count.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b06_ruhela_aggregate(df=None)[source]¶

Aggregate Ruhela formulation on b06 – disciplinary reason -> seg.

T = (Reason contains “Disciplinary”) -> Y = Number of seg placements. Year-FE + Region-FE + Gender-FE; Institution as GEE cluster group (24 levels) for marginal-model cluster-robust inference.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c03_ruhela_aggregate(df=None)[source]¶

Aggregate Ruhela formulation on c03 – racial disparity in RC.

T = (Race == “Indigenous”) indicator -> Y = NumberIndividuals_RC. Year-FE + Gender-FE. Indigenous vs all-other contrast tests overrepresentation.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c04_ruhela_aggregate(df=None)[source]¶

Aggregate Ruhela formulation on c04 – racial disparity by region.

T = (Race == “Indigenous”) -> Y = NumberIndividuals_RC. Year-FE + Region-FE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c06_ruhela_aggregate(df=None)[source]¶

Aggregate Ruhela formulation on c06 – age-50+ disparity in RC.

T = (Age_Category contains “50”) -> Y = NumberIndividuals_RC. Year-FE + Region-FE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c09_ruhela_aggregate(df=None)[source]¶

Aggregate Ruhela formulation on c09 – age-50+ disparity by gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_a01_ruhela_alt_gender(df=None)[source]¶

Alt-T Ruhela formulation on a01 – gender -> regional volatility.

Same per-row Ruhela cell frame as the canonical formulation, but T = Female indicator -> Y = vm count. Covariates exclude Gender (the variable being treated). Runs the full 10-estimator DLRM ensemble. Tests whether women experience more or fewer regional transfers within a fiscal year of restrictive confinement.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_a01_ruhela_alt_age(df=None)[source]¶

Alt-T Ruhela formulation on a01 – age 50+ -> regional volatility.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_a01_ruhela_alt_toronto(df=None)[source]¶

Alt-T Ruhela formulation on a01 – Toronto-region -> regional volatility.

T = (Region_AtTimeOfPlacement == “Toronto”) indicator. Tests whether Toronto-region placements produce different volatility than placements outside Toronto, controlling for demographics.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_ruhela_alt_gender(df=None)[source]¶

Alt-T Ruhela formulation on b01 – gender -> regional volatility.

Same as analyze_a01_ruhela_alt_gender but on b01 (Segregation Detailed) cell frame. Tests whether women experience more or fewer regional transfers within a fiscal year of segregation.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_ruhela_alt_age(df=None)[source]¶

Alt-T Ruhela formulation on b01 – age 50+ -> regional volatility.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_ruhela_alt_toronto(df=None)[source]¶

Alt-T Ruhela formulation on b01 – Toronto-region -> regional volatility.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_a01_ruhela_subgroup_female(df=None)[source]¶

Canonical a01 Ruhela formulation on Female-only subset.

Tests effect-heterogeneity-by-gender: same T_high_ac -> vm contrast, but only on female cells. Compare ATE here to analyze_a01_ruhela_subgroup_male to see if alert-complexity matters more or less for women than men.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_a01_ruhela_subgroup_male(df=None)[source]¶

Canonical a01 Ruhela formulation on Male-only subset.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_ruhela_subgroup_female(df=None)[source]¶

Canonical b01 Ruhela formulation on Female-only subset.

Sister to analyze_a01_ruhela_subgroup_female. Tests effect- heterogeneity-by-gender on segregation records.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_ruhela_subgroup_male(df=None)[source]¶

Canonical b01 Ruhela formulation on Male-only subset.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b02_ruhela_alt_region(df=None)[source]¶

b02 alt-T: T = Toronto region -> Y = total seg days within FY.

Sister to analyze_b02_ruhela_formulations (T = Female). Tests regional disparity in seg-day burden controlling for gender + age + year.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b02_ruhela_alt_age(df=None)[source]¶

b02 alt-T: T = Age 50+ -> Y = total seg days within FY.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c05_ruhela_aggregate(df=None)[source]¶

Aggregate RF on c05 – non-Christian religion -> individuals in RC.

T = (Religion != “Christian” and not “No Religion” and not “Unknown”) indicator. Tests population-level religious-minority overrepresentation in RC, controlling for region.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c08_ruhela_aggregate(df=None)[source]¶

Aggregate RF on c08 – non-Christian religion × gender -> RC.

T = (Religion != Christian / no religion / unknown) indicator. Year + Gender FE. Companion to c05 with gender control.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b04_ruhela_aggregate(df=None)[source]¶

Aggregate RF on b04 – gender disparity in median seg duration.

b04 has Region × Gender × Measure (max/median/mode) duration. We filter to Measure == “Median” and test T = Female -> Y = duration.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b08_ruhela_aggregate(df=None)[source]¶

Aggregate RF on b08 – gender disparity in median seg duration by institution.

Companion to b04; b08 adds Institution_AtTimeOfPlacement as a finer-grained spatial dimension. Region-FE; Institution as GEE cluster group (24 levels).

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b09_ruhela_aggregate(df=None)[source]¶

Aggregate RF on b09 – gender disparity in placement-count distribution.

b09 has rows of (Year × NumberPlacements_Segregation × Gender × NumberIndividuals_Segregation). We test T = Female -> Y = NumberIndividuals_Segregation, with NumberPlacements as a categorical FE so we adjust for the bin-cell structure.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c02_ruhela_aggregate(df=None)[source]¶

Aggregate RF on c02 – gender disparity by institution.

Companion to c01 with institution-level granularity. T = Female -> Y = NumberIndividuals_RC. Region + year as fixed effects; Institution as a GEE cluster group (marginal model with cluster-robust SE) – Institution has too many levels (~76) for an FE specification on the 148-row table without collinearity. Mirrors OTIS-RC’s glmmTMB nbinom2 clustered approach.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c10_ruhela_aggregate(df=None)[source]¶

Aggregate RF on c10 – gender disparity in median RC days.

c10 has Region × Institution × Gender × Measure (Maximum/Median/Mode) duration. Filter to Median; T = Female -> Y = TotalAggregatedDays_RC. Region-FE + Year-FE; Institution as GEE cluster group (25 levels) for cluster-robust marginal inference.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c12_ruhela_aggregate(df=None)[source]¶

Aggregate RF on c12 – gender disparity in median RC days by region.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c11_ruhela_aggregate(df=None)[source]¶

Aggregate Ruhela formulation on c11 – long-duration cell concentration.

c11 has rows of (FY × Aggregate_Duration_bin × NumberIndividuals_RC × NumberIndividuals_Seg). 11 duration bins from “1 day” to “Greater than 30 days”. Rather than treat duration as the treatment-FE (it IS the categorical the data is binned by), we test T = (long-duration bin: ≥16 days) -> Y = NumberIndividuals_RC. Year-FE only.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c01_ruhela_aggregate_region_cluster(df=None)[source]¶

c01 alternative: Female -> RC count, GEE-clustered on year.

Sister analyzer to analyze_c01_ruhela_aggregate. The c01 case has only 6 cells (3 years × 2 genders), so cluster_group=”EndFiscalYear” gives a 3-cluster GEE – extreme small-cluster bias. This variant is shipped for transparency: shows that with so few clusters, Poisson and GEE-Poisson give nearly identical numbers because there’s no clustering structure to exploit.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c04_ruhela_aggregate_region_cluster(df=None)[source]¶

c04 alternative: Indigenous -> RC, GEE-clustered on region.

Sister to analyze_c04_ruhela_aggregate (which uses region as FE). With 5 regions, the GEE-cluster:Region inference treats region as a marginal-model grouping. Useful when wide region heterogeneity in case-mix matters for SE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_d02_ruhela_aggregate(df=None)[source]¶

d02 – gender -> custodial death count, with year FE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_d03_ruhela_aggregate(df=None)[source]¶

d03 – Indigenous -> custodial death count, with year FE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_d04_ruhela_aggregate(df=None)[source]¶

d04 – non-majority religion -> custodial death count.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_d05_ruhela_aggregate(df=None)[source]¶

d05 – age 50+ -> custodial death count.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_ruhela_grid()[source]¶

One-page Ruhela formulations grid summary (24 aggregates).

Runs every aggregate Ruhela formulation analyzer + the region-cluster GEE variants, and presents a single comparison table.

Return type:: RichResult

morie.otis_all_analyze.analyze_ruhela_master(*, include_per_row=False)[source]¶

Paper-ready master report – every Ruhela formulation in one RichResult.

Sections:

Aggregate Ruhela formulations (24 datasets) – IRR comparison
Per-row Ruhela formulations on a01/b01/b02 – primary IRM-DML ATE (only when include_per_row=True; ~5-7 min runtime)
MRM chi-square family on c-series + d-series

For thesis / paper writing.

Parameters:: include_per_row (bool) – If True, also runs the slow per-row RFs on a01/b01/b02 (full 10-estimator DLRM). Default False – runs in ~1 second from cached aggregates only.
Return type:: RichResult

morie.otis_all_analyze.analyze_b02_dlrm(*args, **kwargs)[source]¶: DLRM short alias for analyze_b02_ruhela_formulations.

morie.otis_all_analyze.analyze_a01_dlrm(*args, **kwargs)[source]¶: DLRM short alias for analyze_a01_ruhela_formulations.

morie.otis_all_analyze.analyze_b01_dlrm(*args, **kwargs)[source]¶: DLRM short alias for analyze_b01_ruhela_formulations.

morie.otis_all_analyze.analyze_b05_mandela_classification(df=None)[source]¶

Mandela-RF on b05 – per-placement Mandela classification by year.

Applies the Sprott-Doob 15-day Mandela threshold to OTIS b05 (Ontario provincial segregation placement counts by binned duration). Reports proportions of placements ≤15 days (solitary) vs ≥16 days (torture) per fiscal year + a federal-vs-provincial comparison.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c11_mandela_classification(df=None)[source]¶

Mandela-RF on c11 – per-individual Mandela classification by year.

Applies the 15-day threshold to OTIS c11 (Ontario provincial counts of INDIVIDUALS by binned aggregate duration). Reports both restrictive- confinement and segregation-only views; the latter is the more conservative and directly comparable to Sprott-Doob federal SIUs.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_otis_mandela_provincial_vs_federal()[source]¶

Side-by-side comparison: provincial OTIS vs federal SIU Mandela rates.

Cross-references analyze_c11_mandela_classification (Ontario provincial individuals) against the Sprott-Doob Feb 2021 federal SIU figures (Table 19, N=1960).

Return type:: RichResult

morie.otis_all_analyze.analyze_c_chi2(*, contingency_value='NumberIndividuals_RestrictiveConfinement')[source]¶

MRM chi-square family on c-series demographic contingency tables.

Honour to Prof. Doob’s chi-square tradition in Canadian corrections. Runs Pearson χ² + Cramer’s V on every meaningful 2-way slice of the c-series datasets, using the chosen primary count column.

Sibling: analyze_d_chi2 (death counts). Federal counterpart context: morie.siuiap documents the Structured Intervention Unit Implementation Advisory Panel (Doob, Sprott, Sapers et al.).

Parameters:: contingency_value (str)
Return type:: RichResult

morie.otis_all_analyze.analyze_d_chi2()[source]¶

MRM chi-square on d-series death data + yearly trend.

Honour to Prof. Doob’s chi-square tradition; same payload as analyze_d_aggregate (yearly Poisson 95% CI + Alert × MedicalCause χ² + Alert × Housing χ²).

Return type:: RichResult

morie.otis_all_analyze.analyze_b01_dual(*args, **kwargs)[source]¶: Deprecated alias for analyze_b01_ruhela_formulations.

morie.otis_all_analyze.analyze_c_aggregate(*, contingency_value='NumberIndividuals_RestrictiveConfinement')[source]¶

Chi² + Cramer’s V family on every c-series 2-way slice.

c-series datasets are pre-aggregated counts, so the natural statistic on them is contingency-table independence. This runs χ² + Cramer’s V on each meaningful (categorical_a × categorical_b) slice using the chosen primary count column.

Parameters:: contingency_value (str)
Return type:: RichResult

morie.otis_all_analyze.analyze_d_aggregate()[source]¶

Yearly trend + alert-cause/housing χ² on d-series death data.

Three things on the d-series:

Per-year custodial-death count (d02-d05 aggregate over the categorical dim) with Poisson 95% CIs.
Year-over-year rate ratio (last full year / first year).
χ² + Cramer’s V on d06 (Alert × MedicalCauseOfDeath) and d07 (Alert × Housing_Type).

All measures are at the population aggregate level – d-series has no per-individual alert columns, so the Ruhela dual is structurally impossible here.

Return type:: RichResult

morie.otis_all_analyze.analyze_a01_with_csi_context(df=None, *, variant='total', rebase_to_year=2023)[source]¶

OTIS a01 causal pipeline + Toronto Crime Severity Index context.

Wires together three independent morie subsystems:

morie.otis_all_analyze.analyze_a01 – the MatchIt-then- IRM-DML causal estimate of high-alert-complexity (ac ≥ 2) on regional volatility (vm) within OTIS a01.

morie.otis_tps_overlay – year-by-year correlation between OTIS Toronto-region segregation counts and TPS incident counts.

morie.tps_csi.analyze_csi_from_tps_dataframes – Statistics Canada Crime Severity Index per year + per ward, weighted by offence-specific severity weights and population-adjusted.

The narrative answer: “Did the OTIS-observed increase in high-alert- complexity placements happen during a period of rising or falling Toronto crime severity?” The CSI rebased to rebase_to_year (default 2023, OTIS’s first FY) gives a grandma-readable index.

Parameters:

df (pd.DataFrame, optional) – a01 DataFrame; if None, otis_all_analyze.analyze_a01() loads the canonical local copy.
variant ({"total", "violent"}) – StatsCan CSI variant.
rebase_to_year (int | None) – Anchor year for the CSI index column. None skips rebasing.

Return type:

morie.otis_all_analyze.analyze_all(out_dir=None)[source]¶

Run all 29 OTIS analyses and write outputs.

Parameters:: out_dir (Path | None)
Return type:: dict[str, RichResult]

morie.otis_all_analyze.analyze_a01_mrm_alt_age(df=None)¶

Alt-T Ruhela formulation on a01 – age 50+ -> regional volatility.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_a01_mrm_alt_gender(df=None)¶

Alt-T Ruhela formulation on a01 – gender -> regional volatility.

Same per-row Ruhela cell frame as the canonical formulation, but T = Female indicator -> Y = vm count. Covariates exclude Gender (the variable being treated). Runs the full 10-estimator DLRM ensemble. Tests whether women experience more or fewer regional transfers within a fiscal year of restrictive confinement.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_a01_mrm_alt_toronto(df=None)¶

Alt-T Ruhela formulation on a01 – Toronto-region -> regional volatility.

T = (Region_AtTimeOfPlacement == “Toronto”) indicator. Tests whether Toronto-region placements produce different volatility than placements outside Toronto, controlling for demographics.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_a01_mrm(df=None, *, cluster_col='EndFiscalYear')¶

Ruhela formulations (full DLRM) on a01 – the canonical OTIS-RC dataset.

Runs the complete OTIS-RC methodology arc on the author’s Ruhela formulation (T = high alert complexity, Y = regional volatility). See _ruhela_formulations_on for the full estimator list.

Attribution¶

The Ruhela formulation is the author’s design end-to-end; the supporting- contributor credits (Doob – pointed the author to the dataset; Levinsky – course context, reviewed only the preliminary a01 analysis; Medina – reviewed the formal write-up) are documented in morie.otis_causal’s module docstring.

param df:: a01 DataFrame; if None, otis_datasets.load_otis_dataset('a01').
type df:: pd.DataFrame, optional
param cluster_col:: Cluster axis for cluster-robust SE in the IRM-DML estimator (default ‘EndFiscalYear’).
type cluster_col:: str

Parameters:

df (DataFrame | None)
cluster_col (str)

Return type:

morie.otis_all_analyze.analyze_a01_mrm_per_year(df=None)¶

Per-year full DLRM on Ruhela formulations on a01.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_a01_mrm_subgroup_female(df=None)¶

Canonical a01 Ruhela formulation on Female-only subset.

Tests effect-heterogeneity-by-gender: same T_high_ac -> vm contrast, but only on female cells. Compare ATE here to analyze_a01_ruhela_subgroup_male to see if alert-complexity matters more or less for women than men.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_a01_mrm_subgroup_male(df=None)¶

Canonical a01 Ruhela formulation on Male-only subset.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_mrm_alt_age(df=None)¶

Alt-T Ruhela formulation on b01 – age 50+ -> regional volatility.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_mrm_alt_gender(df=None)¶

Alt-T Ruhela formulation on b01 – gender -> regional volatility.

Same as analyze_a01_ruhela_alt_gender but on b01 (Segregation Detailed) cell frame. Tests whether women experience more or fewer regional transfers within a fiscal year of segregation.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_mrm_alt_toronto(df=None)¶

Alt-T Ruhela formulation on b01 – Toronto-region -> regional volatility.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_mrm(df=None, *, cluster_col='EndFiscalYear')¶

Ruhela formulations (full DLRM) on b01 (Segregation Detailed).

Per-placement segregation complement to a01’s per-day RC records. Same Ruhela alert-complexity -> regional-volatility contrast, larger magnitude effect because segregation is a stronger institutional contrast.

See analyze_a01_ruhela_formulations for the full attribution and estimator catalogue.

Parameters:

df (DataFrame | None)
cluster_col (str)

Return type:

morie.otis_all_analyze.analyze_b01_mrm_per_year(df=None)¶

Per-year full DLRM on Ruhela formulations on b01.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_mrm_subgroup_female(df=None)¶

Canonical b01 Ruhela formulation on Female-only subset.

Sister to analyze_a01_ruhela_subgroup_female. Tests effect- heterogeneity-by-gender on segregation records.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b01_mrm_subgroup_male(df=None)¶

Canonical b01 Ruhela formulation on Male-only subset.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b02_mrm_alt_age(df=None)¶

b02 alt-T: T = Age 50+ -> Y = total seg days within FY.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b02_mrm_alt_region(df=None)¶

b02 alt-T: T = Toronto region -> Y = total seg days within FY.

Sister to analyze_b02_ruhela_formulations (T = Female). Tests regional disparity in seg-day burden controlling for gender + age + year.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b02_mrm(df=None, *, cluster_col='EndFiscalYear')¶

Ruhela formulation on b02 (Segregation Total Days per individual).

b02 has UniqueIndividual_ID + EndFiscalYear + Gender + Region + Age_Category + TotalAggregatedDays_Segregation but NO alert columns. The canonical alert-complexity formulation is impossible; instead the author’s b02-specific Ruhela formulation tests gender disparity in segregation-day burden:

T = Gender == “Female” indicator Y = TotalAggregatedDays_Segregation (count, person-year) covariates = [Region_MostRecentPlacement, Age_Category, EndFiscalYear]

Same DLRM ensemble as the canonical formulation. Sister formulations on the same b02 data could swap T for region or age.

Parameters:

df (DataFrame | None)
cluster_col (str)

Return type:

morie.otis_all_analyze.analyze_b03_mrm_aggregate(df=None)¶

Aggregate Ruhela formulation on b03 (Alert × Institution).

T = Alert_Presence indicator -> Y = Number_SegregationPlacements. Year-FE + Region-FE + Alert_Type-FE; Institution as GEE cluster group (24 levels, marginal model with cluster-robust SE).

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b04_mrm_aggregate(df=None)¶

Aggregate RF on b04 – gender disparity in median seg duration.

b04 has Region × Gender × Measure (max/median/mode) duration. We filter to Measure == “Median” and test T = Female -> Y = duration.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b06_mrm_aggregate(df=None)¶

Aggregate Ruhela formulation on b06 – disciplinary reason -> seg.

T = (Reason contains “Disciplinary”) -> Y = Number of seg placements. Year-FE + Region-FE + Gender-FE; Institution as GEE cluster group (24 levels) for marginal-model cluster-robust inference.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b07_mrm_aggregate(df=None)¶

Aggregate Ruhela formulation on b07 (alerts × gender summary).

b07 is a wide table with paired columns:: Number_Segregation_Placements_With_Alert Number_Segregation_Placements_Without_Alert

We pivot to long form (one row per alert-state × gender × year), then T = with_alert indicator -> Y = count. Year + Gender + Alert_Type fixed effects.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b08_mrm_aggregate(df=None)¶

Aggregate RF on b08 – gender disparity in median seg duration by institution.

Companion to b04; b08 adds Institution_AtTimeOfPlacement as a finer-grained spatial dimension. Region-FE; Institution as GEE cluster group (24 levels).

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_b09_mrm_aggregate(df=None)¶

Aggregate RF on b09 – gender disparity in placement-count distribution.

b09 has rows of (Year × NumberPlacements_Segregation × Gender × NumberIndividuals_Segregation). We test T = Female -> Y = NumberIndividuals_Segregation, with NumberPlacements as a categorical FE so we adjust for the bin-cell structure.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c01_mrm_aggregate(df=None)¶

Aggregate Ruhela formulation on c01 – gender disparity in RC.

c01 has Gender × {InCustody, RestrictiveConfinement, Segregation} counts. T = Female indicator -> Y = NumberIndividuals_RC, Year-FE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c01_mrm_aggregate_region_cluster(df=None)¶

c01 alternative: Female -> RC count, GEE-clustered on year.

Sister analyzer to analyze_c01_ruhela_aggregate. The c01 case has only 6 cells (3 years × 2 genders), so cluster_group=”EndFiscalYear” gives a 3-cluster GEE – extreme small-cluster bias. This variant is shipped for transparency: shows that with so few clusters, Poisson and GEE-Poisson give nearly identical numbers because there’s no clustering structure to exploit.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c02_mrm_aggregate(df=None)¶

Aggregate RF on c02 – gender disparity by institution.

Companion to c01 with institution-level granularity. T = Female -> Y = NumberIndividuals_RC. Region + year as fixed effects; Institution as a GEE cluster group (marginal model with cluster-robust SE) – Institution has too many levels (~76) for an FE specification on the 148-row table without collinearity. Mirrors OTIS-RC’s glmmTMB nbinom2 clustered approach.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c03_mrm_aggregate(df=None)¶

Aggregate Ruhela formulation on c03 – racial disparity in RC.

T = (Race == “Indigenous”) indicator -> Y = NumberIndividuals_RC. Year-FE + Gender-FE. Indigenous vs all-other contrast tests overrepresentation.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c04_mrm_aggregate(df=None)¶

Aggregate Ruhela formulation on c04 – racial disparity by region.

T = (Race == “Indigenous”) -> Y = NumberIndividuals_RC. Year-FE + Region-FE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c04_mrm_aggregate_region_cluster(df=None)¶

c04 alternative: Indigenous -> RC, GEE-clustered on region.

Sister to analyze_c04_ruhela_aggregate (which uses region as FE). With 5 regions, the GEE-cluster:Region inference treats region as a marginal-model grouping. Useful when wide region heterogeneity in case-mix matters for SE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c05_mrm_aggregate(df=None)¶

Aggregate RF on c05 – non-Christian religion -> individuals in RC.

T = (Religion != “Christian” and not “No Religion” and not “Unknown”) indicator. Tests population-level religious-minority overrepresentation in RC, controlling for region.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c06_mrm_aggregate(df=None)¶

Aggregate Ruhela formulation on c06 – age-50+ disparity in RC.

T = (Age_Category contains “50”) -> Y = NumberIndividuals_RC. Year-FE + Region-FE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c07_mrm_aggregate(df=None)¶

Aggregate Ruhela formulation on c07 – alert × gender -> RC count.

Counterpart of the per-row Ruhela formulation on a01/b01 at the aggregate level. Useful triangulation: c07 reports NumberIndividuals_InCustody / RC / Segregation cross-tabulated by Alert_Type × Gender × FiscalYear. The Ruhela formulation here contrasts an indicator of alert presence (vs. no-alert reference) with the number-in-RC count.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c08_mrm_aggregate(df=None)¶

Aggregate RF on c08 – non-Christian religion × gender -> RC.

T = (Religion != Christian / no religion / unknown) indicator. Year + Gender FE. Companion to c05 with gender control.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c09_mrm_aggregate(df=None)¶

Aggregate Ruhela formulation on c09 – age-50+ disparity by gender.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c10_mrm_aggregate(df=None)¶

Aggregate RF on c10 – gender disparity in median RC days.

c10 has Region × Institution × Gender × Measure (Maximum/Median/Mode) duration. Filter to Median; T = Female -> Y = TotalAggregatedDays_RC. Region-FE + Year-FE; Institution as GEE cluster group (25 levels) for cluster-robust marginal inference.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c11_mrm_aggregate(df=None)¶

Aggregate Ruhela formulation on c11 – long-duration cell concentration.

c11 has rows of (FY × Aggregate_Duration_bin × NumberIndividuals_RC × NumberIndividuals_Seg). 11 duration bins from “1 day” to “Greater than 30 days”. Rather than treat duration as the treatment-FE (it IS the categorical the data is binned by), we test T = (long-duration bin: ≥16 days) -> Y = NumberIndividuals_RC. Year-FE only.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c12_mrm_aggregate(df=None)¶

Aggregate RF on c12 – gender disparity in median RC days by region.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_d02_mrm_aggregate(df=None)¶

d02 – gender -> custodial death count, with year FE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_d03_mrm_aggregate(df=None)¶

d03 – Indigenous -> custodial death count, with year FE.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_d04_mrm_aggregate(df=None)¶

d04 – non-majority religion -> custodial death count.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_d05_mrm_aggregate(df=None)¶

d05 – age 50+ -> custodial death count.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_all_analyze.analyze_c_doob_chi2(*args, **kwargs)[source]¶

Deprecated alias of analyze_c_chi2().

The ‘Doob’ qualifier was removed from chi-square family naming in morie 0.9.5.4 because the underlying chi-square family is not canonically authored by Doob alone; the previous name had been an ambiguous shorthand. Use analyze_c_chi2() instead.

morie.otis_all_analyze.analyze_d_doob_chi2(*args, **kwargs)[source]¶: Deprecated alias of analyze_d_chi2(). See above.

morie.otis_causal – IPW, AIPW, and DML estimators for OTIS.

Companion to morie.otis.otdml (which already implements cross-fitted Frisch-Waugh-Lovell DML). This module adds:

otis_ipw() – Hájek-stabilised inverse probability weighting with a logistic propensity model.

otis_aipw() – Augmented IPW (Robins-Rotnitzky-Zhao 1994 doubly-robust estimator) with cross-fitted nuisance models.

otis_causal_grid() – runs all three estimators (IPW, AIPW, DML) on the canonical three (treatment, outcome) pairs:

MentalHealth_Alert -> SuicideRisk_Alert

HighAlertComplexity -> AnyReadmission

RegionalVolatility -> ConsecutiveSegregationDays

The three pairs cover the three Goffmanian framings of OTIS most relevant to the MA-paired thesis: clinical-alert chain (a), multi-alert “complexity” (b), and inter-region churn (c).

Estimators¶

Let $D in {0,1}$ be a binary treatment, $Y$ a real-valued outcome, $X in mathbb{R}^p$ a covariate vector, and write $e(X) coloneqq Pr(D=1mid X)$ for the propensity score and $mu_d(X) coloneqq mathbb{E}[Ymid D=d, X]$ for the outcome-regression mean.

IPW (Hájek):

ATE_IPW = sum_i w1(i)·D_i·Y_i / sum_i w1(i)·D_i

sum_i w0(i)·(1-D_i)·Y_i / sum_i w0(i)·(1-D_i)

with w1(i) = 1/e(X_i), w0(i) = 1/(1-e(X_i)).

AIPW (RRZ 1994 doubly-robust score):

psi(O_i; eta) = mu_1(X_i) - mu_0(X_i)

D_i·(Y_i - mu_1(X_i))/e(X_i)

(1-D_i)·(Y_i - mu_0(X_i))/(1-e(X_i))

ATE_AIPW = (1/n) sum_i psi(O_i; eta_hat)

with cross-fitted nuisance estimates eta_hat = (e_hat, mu_1_hat, mu_0_hat).

DML PLR:

Frisch-Waugh-Lovell partialled-out regression with cross-fitted residuals (already implemented in morie.otis.otdml()).

References

Robins, J. M., Rotnitzky, A., and Zhao, L. P. (1994). Estimation of regression coefficients when some regressors are not always observed. J. Amer. Statist. Assoc. 89(427): 846-866.

Chernozhukov, V. et al. (2018). Double/debiased machine learning for treatment and structural parameters. Econometrics Journal 21(1): C1-C68.

Lunceford, J. K. and Davidian, M. (2004). Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study. Statistics in Medicine 23(19): 2937-2960.

Robins, J. M. (1986). A new approach to causal inference in mortality studies with a sustained exposure period – application to control of the healthy worker survivor effect. Mathematical Modelling 7: 1393-1512.

Rosenbaum, P. R. and Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika 70(1): 41-55.

Attribution – Ruhela formulations¶

The OTIS-RC formulations and the entire codebase implemented here are the work of Vansh Singh Ruhela (rootcoder007). No one co-formulated the analyses: the (T = ac ≥ 2, Y = vm count) alert-complexity -> regional-volatility contrast, the 8-state combo encoding, and the full method battery (IPW + AIPW + g-computation + PSM-NN + PSM-subclass + IRM-DML + match_first) are Ruhela’s design.

Three external contributors enabled the journey but did not co-author the formulations:

Doob – Prof. Emeritus Anthony N. Doob, University of Toronto.
Pointed Ruhela to the existence of the OTIS data feed – the first piece of the puzzle. Without that pointer, this project does not begin.

Levinsky – Prof. Zachary Levinsky, University of Toronto. His
graduate course on the Theory of Punishment was the broader context that motivated this work. He later reviewed the preliminary a01-only analysis from the OTIS-RC/ folder; he did not see the full method battery shipped here.

Medina – Prof. Zorro Medina, University of Toronto. Reviewed
the formal write-up; her feedback validated the statistical route and gave the pipeline confidence in its current form.

Naming conventions¶

RF – Ruhela formulation: a (treatment, outcome, covariates) design
choice for a specific OTIS dataset.

RDF – Ruhela Dual Formulation: an RF paired with a Naive-arm
sensitivity contrast (e.g., on a01/b01 the Ruhela arm T_high_ac -> vm_count is paired with the Naive arm any-flag -> vm-binary).

DLRM – Doob-Levinsky-Ruhela-Medina, short alias for the
attribution chain on the methodology side. Function names like analyze_a01_dlrm are equivalent aliases for analyze_a01_ruhela_formulations.

ACKNOWLEDGEMENTS – beyond the methodology attribution above¶

The DLRM attribution covers the contributors whose work directly shaped the OTIS-RC analyses. The wider intellectual journey that brought this codebase into being also owes credit to:

Prof. Beatrice Jauregui, University of Toronto – for the network
of mentorship that connected Ruhela to Prof. Medina, which made this line of work possible.

Prof. Ayobami Laniyonu, University of Toronto – for valuable
lessons over the broader period of this work that informed Ruhela’s thinking on policing and corrections research.

The separation between methodology attribution (DLRM, narrow) and acknowledgements (broader, enabling) follows standard academic- publishing convention.

class morie.otis_causal.CausalEstimate(estimator: 'str', ate: 'float', ate_se: 'float', ate_pval: 'float', n: 'int', n_treated: 'int', p_treat: 'float', notes: 'list[str]' = <factory>)[source]¶

Bases: object

Parameters:

estimator (str)
ate (float)
ate_se (float)
ate_pval (float)
n (int)
n_treated (int)
p_treat (float)
notes (list[str])

estimator: str¶

ate: float¶

ate_se: float¶

ate_pval: float¶

n: int¶

n_treated: int¶

p_treat: float¶

notes: list[str]¶

property ate_ci95: tuple[float, float]¶

morie.otis_causal.otis_ipw(df, *, treatment, outcome, covariates, eps=0.02, propensity_calibration='none')[source]¶

Hájek-stabilised IPW estimator of the ATE on OTIS data.

Estimates the propensity by logistic regression and weights treated/control outcomes by $1/hat e$ and $1/(1-hat e)$. Standard error follows the Lunceford-Davidian sandwich.

Parameters:

propensity_calibration ("none" | "platt" | "isotonic") – Calibrate the raw logistic propensities. Platt = sigmoid(a + b * logit(p)) fitted on the same data; isotonic = monotonic regression (sklearn). Calibration improves Brier score / calibration-curve diagnostics; for OTIS data with logistic propensities it usually moves the predicted prevalence toward the observed prevalence.
df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
eps (float)

Return type:

morie.otis_causal.otis_aipw(df, *, treatment, outcome, covariates, n_folds=5, seed=123, eps=0.02, propensity_calibration='none')[source]¶

Augmented IPW (Robins-Rotnitzky-Zhao) ATE on OTIS data.

Uses n_folds cross-fitting: nuisance models (propensity, outcome-regression) are fit on $K-1$ folds and evaluated on the held-out fold, then aggregated. This is the doubly-robust influence-function plug-in estimator.

Parameters:

propensity_calibration ("none" | "platt" | "isotonic") – Calibrate the cross-fitted propensity scores within each fold’s test set against its training labels. Improves Brier- score / calibration-curve diagnostics.
df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
n_folds (int)
seed (int)
eps (float)

Return type:

morie.otis_causal.otis_gcomputation(df, *, treatment, outcome, covariates, n_bootstrap=200, seed=123)[source]¶

Parametric g-computation (Robins 1986) of the ATE on OTIS data.

Fits an outcome regression $\hat\mu(d, X)$ via OLS, then computes the standardised mean difference

\hat\Delta = n^{-1} \sum_i (\hat\mu(1, X_i)

\hat\mu(0, X_i)).

SE via non-parametric bootstrap (resample rows, refit, recompute). Complements IPW and AIPW: g-computation is consistent if and only if the outcome model is correctly specified (single-robust); IPW is consistent iff the propensity is correct; AIPW is doubly robust.

References

Robins, J. M. (1986). A new approach to causal inference in: mortality studies with a sustained exposure period – application to control of the healthy worker survivor effect. Mathematical Modelling 7: 1393-1512.

Parameters:

df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
n_bootstrap (int)
seed (int)

Return type:

morie.otis_causal.otis_psm_subclass(df, *, treatment, outcome, covariates, n_strata=5, eps=0.02)[source]¶

Propensity-score stratification (Rosenbaum & Rubin 1983).

Splits the sample into n_strata strata by propensity-score quantile, computes within-stratum ATE = E[Y|D=1, S=s] - E[Y|D=0, S=s], and weights stratum-specific ATEs by stratum size to get the population ATE. Standard error: weighted average of within-stratum SEs (Rosenbaum-Rubin convention).

n_strata=5 is the standard “rule of thumb” – Cochran (1968) showed 5 strata remove ~90% of bias from a single confounder.

References

Rosenbaum, P. R. & Rubin, D. B. (1983). The central role of the: propensity score in observational studies for causal effects. Biometrika 70(1): 41-55.
Cochran, W. G. (1968). The effectiveness of adjustment by: subclassification in removing bias in observational studies. Biometrics 24(2): 295-313.

Parameters:

df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
n_strata (int)
eps (float)

Return type:

morie.otis_causal.otis_atc(df, *, treatment, outcome, covariates, n_folds=5, seed=123, eps=0.02)[source]¶

AIPW-flavoured Average Treatment effect on the Controls (ATC).

The IF for ATC = E[Y(1) - Y(0) | D = 0] is

psi^{ATC}(O; eta) = mu_1(X) - mu_0(X)

(1-D)/(1-e(X)) * (Y - mu_0(X))

e(X)/(1-e(X)) * D * (Y - mu_1(X)) / e(X)

weighted by 1 / Pr(D=0). The implementation reuses the cross-fitted nuisance pipeline of otis_aipw and reweights the score on the untreated stratum.

Parameters:

df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
n_folds (int)
seed (int)
eps (float)

Return type:

morie.otis_causal.otis_plr(df, *, treatment, outcome, covariates, n_folds=3, seed=123, ml_outcome='rf', ml_treatment='rf')[source]¶

Partially Linear Regression DML (Chernozhukov et al. 2018).

Fits the structural model

Y = θ * D + g(X) + ε, D = m(X) + V

via Frisch-Waugh-Lovell partialling-out: cross-fit nuisances g(X) = E[Y|X] and m(X) = E[D|X], then regress the residualised Y on the residualised D to recover θ.

Cross-fitted random-forest nuisance models match the OTIS-RC DoubleML/PLR pipeline. Differs from IRM-DML in that PLR assumes a constant (homogeneous) treatment effect θ; IRM allows D × X interactions in the outcome model. PLR is the simpler / more restricted estimator. Concordance with IRM-DML is the standard “no effect heterogeneity” robustness signal.

Parameters:

df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
n_folds (int)
seed (int)
ml_outcome (str)
ml_treatment (str)

Return type:

morie.otis_causal.otis_aipw_superlearner(df, *, treatment, outcome, covariates, n_folds=5, seed=123, eps=0.02, propensity_calibration='none')[source]¶

SuperLearner-stacked AIPW (cross-fitted convex stack of learners).

Stacks four learners – random forest, ridge, OLS/logistic, mean – via cross-validated convex weights minimising MSE (regression) or log-loss (propensity). Mirrors the OTIS-RC AIPW SuperLearner pipeline (notez1a.qmd lines 2028-2048: SL.glmnet, SL.xgboost, SL.glm, SL.mean) at a lighter sklearn-only stack. xgboost is optional; if importable, it is added to the stack.

Falls back to plain AIPW if sklearn isn’t available.

References

van der Laan, M. J. and Polley, E. C. and Hubbard, A. E. (2007).: Super Learner. Stat. Appl. Genet. Mol. Biol. 6(1): Article 25.

Parameters:

df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
n_folds (int)
seed (int)
eps (float)
propensity_calibration (str)

Return type:

morie.otis_causal.otis_psm(df, *, treatment, outcome, covariates, k=1, caliper_sd=0.2, with_replacement=False, eps=0.02)[source]¶

1:k nearest-neighbour propensity-score matching on logit(e(X)).

Estimates the average treatment effect on the treated (ATT) by pairing each treated unit with its $k$ nearest control units on the logit-propensity scale. An optional caliper (default $0.2,sigma_{mathrm{logit},e}$, the Austin 2011 convention) discards treated units with no control inside the caliper. When $k=1$ and with_replacement=False the SE is the paired-difference standard error $sqrt{mathrm{Var}(Y_t - Y_{c(t)}) / m}$ where $m$ is the number of matched pairs.

References

Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioural Research 46(3): 399-424.

Abadie, A. and Imbens, G. W. (2006). Large sample properties of matching estimators for average treatment effects. Econometrica 74(1): 235-267.

Parameters:

df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
k (int)
caliper_sd (float | None)
with_replacement (bool)
eps (float)

Return type:

morie.otis_causal.otis_balance(df, *, treatment, covariates, outcome=None, caliper_sd=0.2, eps=0.02)[source]¶

Per-covariate standardised mean differences before / after weighting / matching.

Returns a DataFrame with columns covariate, smd_raw, smd_ipw, smd_psm. Conventionally $|mathrm{SMD}| > 0.10$ flags meaningful imbalance and $|mathrm{SMD}| > 0.25$ strong imbalance cite{austin2011smd}.

Parameters:

df (DataFrame)
treatment (str)
covariates (list[str])
outcome (str | None)
caliper_sd (float | None)
eps (float)

Return type:

DataFrame

morie.otis_causal.otis_overlap(df, *, treatment, covariates, eps=0.02)[source]¶

Propensity-score overlap diagnostics.

Returns a dict with min/max propensity per group, fraction inside the trimming interval $[varepsilon, 1-varepsilon]$, Hellinger distance between treated/control propensity densities, and the count of clipped propensities.

Parameters:

df (DataFrame)
treatment (str)
covariates (list[str])
eps (float)

Return type:

morie.otis_causal.otis_irm_dml(df, *, treatment, outcome, covariates, cluster_cols=None, n_folds=3, seed=123, eps=0.02, ml_outcome='rf', ml_propensity='rf', match_first=False, match_caliper_sd=0.2)[source]¶

Interactive-Regression-Model DML, matching DoubleML in R.

Computes both the ATE (population average treatment effect) and ATTE (average treatment effect on the treated) via the doubly-robust influence function

psi^{ATE}(O; eta)

= mu_1(X) - mu_0(X)

D(Y - mu_1(X)) / e(X)

(1-D)(Y - mu_0(X)) / (1 - e(X)),

psi^{ATTE}(O; eta)

= frac{D(Y - mu_0(X)) - e(X)(1-D)(Y - mu_0(X)) /
(1 - e(X))}{bar D},

with cross-fitted random-forest nuisance models. Standard errors can be cluster-robust on one or two cluster axes (cite{CameronGelbachMiller2011,LiangZeger1986}).

Returns a dict with both estimands, their SEs, and 95% CIs.

Parameters:

cluster_cols (str | list[str] | None) – Column names of cluster axes. None ⇒ heteroskedasticity- consistent (no clustering). "id" or ["id"] ⇒ one-way cluster on id. ["id", "rc"] ⇒ two-way Cameron-Gelbach-Miller.
match_first (bool) – If True, first run 1:1 nearest-neighbour propensity-score matching on logit(ê(X)) with caliper ``match_caliper_sd``× SD(logit(ê)), then fit IRM-DML on the matched subset only. Mirrors the MatchIt-then-DML pipeline in notez1a.qmd.
df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
n_folds (int)
seed (int)
eps (float)
ml_outcome (str)
ml_propensity (str)
match_caliper_sd (float | None)

Return type:

morie.otis_causal.otis_per_year_irm_dml(df, *, treatment, outcome, covariates, year_col='EndFiscalYear', cluster_cols=None, n_folds=3, seed=123, full_battery=False, propensity_calibration='none')[source]¶

Fit IRM-DML separately on each fiscal year.

Returns a dict keyed by year value. When full_battery=False (default), each year’s value is the standard otis_irm_dml output (mirrors Ruhela’s res_by_year in correctional_stats_report1z.RData).

When full_battery=True, each year’s value is a dict {estimator_name: CausalEstimate-or-irm-dict} with all 10 Ruhela-formulation estimators (IPW, AIPW, g-computation, PSM 1:1 NN, PSM subclass, IRM-DML, PSM->IRM-DML match_first, ATC-AIPW, PLR, SuperLearner-AIPW). Roughly 7× the per-year runtime.

Parameters:

df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
year_col (str)
cluster_cols (list[str] | str | None)
n_folds (int)
seed (int)
full_battery (bool)
propensity_calibration (str)

Return type:

morie.otis_causal.make_pair_alert_to_volatility_ruhela(df)[source]¶

Ruhela’s primary causal contrast: high alert complexity (ac ≥ 2) causes more inter-region transfers (vm).

This is “Ruhela’s equation” for alert complexity and volatility – the 8-state combo encoding documented in /path/to/workspace/OTIS-RC/notez1a.qmd and used to produce the published res_pool / res_by_year / res_all estimates in correctional_stats_report1z.RData. Sister function make_pair_alert_to_volatility_naive() is available as a robustness alternative; both are run side by side by make_pair_alert_to_volatility_all().

Encoding¶

For each placement-row, the three binary alerts (MentalHealth_Alert, SuicideRisk_Alert, SuicideWatch_Alert) encode one of $2^3 = 8$ states:

a1 = MH & ¬SR & ¬SW (mental-health only) a2 = ¬MH & SR & ¬SW (suicide-risk only) a3 = ¬MH & ¬SR & SW (suicide-watch only – empirically empty) a4 = MH & SR & ¬SW (MH+SR) a5 = ¬MH & SR & SW (SR+SW) a6 = MH & ¬SR & SW (MH+SW – empirically empty) a7 = MH & SR & SW (all three) a8 = ¬MH & ¬SR & ¬SW (no alerts)

Per (UniqueIndividual_ID, EndFiscalYear), the alert-state complexity ac is the number of distinct combos with positive support across the year’s placement-rows for that person. This differs from the naïve “max simultaneous flags” – a person who had three a4 placements has ac = 1 (one distinct combo), not ac = 2 (two simultaneous flags). Treatment T = 1 iff ac ≥ 2.

Outcome¶

vm (regional volatility moves, count) sums two indicators across the person-year placement sequence:

regA ≠ regB within the same row (the person’s original-vs-most-recent regions differ in this placement);

regA ≠ regA(prev row) across rows (the person’s original-region label changed across consecutive placements).

This is a Poisson-style count outcome on ${0, 1, 2, ldots}$.

returns:: T = “T_high_ac” binary, 1 iff ac ≥ 2 Y = “Y_vm_count” integer count, regional transfer count covariates = [“Gender”, “Age_Category”, “EndFiscalYear”]
rtype:: (data, T, Y, covariates) where

Parameters:: df (DataFrame)
Return type:: tuple[DataFrame, str, str, list[str]]

morie.otis_causal.make_pair_alert_to_volatility_naive(df)[source]¶

Naïve (simpler) alternative to make_pair_alert_to_volatility_ruhela().

Treatment ac:: per (id, year), the maximum number of simultaneous alert flags across the year’s placement rows (i.e.max(MH + SR + SW) per row). T = 1 iff this maximum is ≥ 2.
Outcome vm (binary):: per (id, year), 1 iff any placement row has Region_AtTimeOfPlacement ≠ Region_MostRecentPlacement.

This was the original morie formulation before we audited against OTIS-RC/notez1a.qmd; it produces a different treatment marginal (~24% treated vs ~14% under Ruhela’s 8-state encoding) and a binary rather than count outcome. We keep it for robustness comparison: if both formulations agree on sign and rough magnitude, the substantive conclusion is invariant to the operationalisation.

Parameters:: df (DataFrame)
Return type:: tuple[DataFrame, str, str, list[str]]

morie.otis_causal.make_pair_alert_to_volatility_all(df)[source]¶

Run both Ruhela and naïve alert->volatility formulations.

Returns a dict {"ruhela": (...), "naive": (...)} where each value is a 4-tuple (person_year_data, T, Y, covariates) suitable for handing to otis_irm_dml().

Parameters:: df (DataFrame)
Return type:: dict

morie.otis_causal.make_pair_alert_to_volatility(df)¶

Ruhela’s primary causal contrast: high alert complexity (ac ≥ 2) causes more inter-region transfers (vm).

This is “Ruhela’s equation” for alert complexity and volatility – the 8-state combo encoding documented in /path/to/workspace/OTIS-RC/notez1a.qmd and used to produce the published res_pool / res_by_year / res_all estimates in correctional_stats_report1z.RData. Sister function make_pair_alert_to_volatility_naive() is available as a robustness alternative; both are run side by side by make_pair_alert_to_volatility_all().

Encoding¶

For each placement-row, the three binary alerts (MentalHealth_Alert, SuicideRisk_Alert, SuicideWatch_Alert) encode one of $2^3 = 8$ states:

a1 = MH & ¬SR & ¬SW (mental-health only) a2 = ¬MH & SR & ¬SW (suicide-risk only) a3 = ¬MH & ¬SR & SW (suicide-watch only – empirically empty) a4 = MH & SR & ¬SW (MH+SR) a5 = ¬MH & SR & SW (SR+SW) a6 = MH & ¬SR & SW (MH+SW – empirically empty) a7 = MH & SR & SW (all three) a8 = ¬MH & ¬SR & ¬SW (no alerts)

Per (UniqueIndividual_ID, EndFiscalYear), the alert-state complexity ac is the number of distinct combos with positive support across the year’s placement-rows for that person. This differs from the naïve “max simultaneous flags” – a person who had three a4 placements has ac = 1 (one distinct combo), not ac = 2 (two simultaneous flags). Treatment T = 1 iff ac ≥ 2.

Outcome¶

vm (regional volatility moves, count) sums two indicators across the person-year placement sequence:

regA ≠ regB within the same row (the person’s original-vs-most-recent regions differ in this placement);

regA ≠ regA(prev row) across rows (the person’s original-region label changed across consecutive placements).

This is a Poisson-style count outcome on ${0, 1, 2, ldots}$.

returns:: T = “T_high_ac” binary, 1 iff ac ≥ 2 Y = “Y_vm_count” integer count, regional transfer count covariates = [“Gender”, “Age_Category”, “EndFiscalYear”]
rtype:: (data, T, Y, covariates) where

Parameters:: df (DataFrame)
Return type:: tuple[DataFrame, str, str, list[str]]

morie.otis_causal.otis_dml(df, *, treatment, outcome, covariates, n_folds=5, seed=123)[source]¶

Cross-fitted DML PLR (partialling-out) wrapper.

Reproduces morie.otis.otdml() but exposes the same CausalEstimate shape for grid comparison.

Parameters:

df (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
n_folds (int)
seed (int)

Return type:

morie.otis_causal.make_pair_a(df)[source]¶

Pair (a): MentalHealth_Alert -> SuicideRisk_Alert (binary->binary).

The clinical-alert chain: do mental-health flags causally elevate subsequent suicide-risk-alert occurrence, conditional on demographics and region?

Parameters:: df (DataFrame)
Return type:: tuple[DataFrame, str, str, list[str]]

morie.otis_causal.make_pair_b(df)[source]¶

Pair (b): HighAlertComplexity -> AnyReadmission.

Treatment T_b = 1 iff at least 2 of {MentalHealth, SuicideRisk, SuicideWatch} alerts are simultaneously active in the year. Outcome Y_b = 1 iff this individual has ≥2 placements (proxy for any future readmission).

Parameters:: df (DataFrame)
Return type:: tuple[DataFrame, str, str, list[str]]

morie.otis_causal.make_pair_c(df)[source]¶

Pair (c): RegionalVolatility -> SegregationDays.

Treatment T_c = 1 iff Region_AtTimeOfPlacement ≠ Region_MostRecentPlacement (the inter-region transfer flag). Outcome Y_c = NumberConsecutiveDays_Segregation (continuous, truncated at the 99th percentile to dampen the long tail).

Parameters:: df (DataFrame)
Return type:: tuple[DataFrame, str, str, list[str]]

morie.otis_causal.make_pair_alert_to_volatility_a01(df=None)[source]¶

Same as make_pair_alert_to_volatility_ruhela() but auto-loads a01 (Restrictive Confinement Detailed) instead of b01 (Segregation Detailed) if df is None.

a01 is the file the published res_pool / res_by_year / res_all are computed on; using it in our pipeline closes the “different bundled snapshot” gap.

Parameters:: df (DataFrame | None)
Return type:: tuple[DataFrame, str, str, list[str]]

morie.otis_causal.otis_causal_grid(df=None, *, seed=123)[source]¶

Run IPW / AIPW / DML on all three canonical (T, Y) pairs.

Returns a RichResult with a 9-row payload table – three estimators × three pairs – plus per-row standard errors, p-values, and 95% confidence intervals.

Parameters:

df (DataFrame | None)
seed (int)

Return type:

morie.otis_churn – Goffmanian institutional-churn analyses on OTIS.

Goffman (1961) describes ‘total institutions’ as totalizing environments that produce a cyclical relationship to the inmate. This module operationalises the cyclical / mortifying / embedding dimensions as formal statistical tests:

repeat_placement_concentration(b09) – Gini + power-law fit + KS test within_year_placement_count(b01) – per-(id,year) cell-size distribution within_year_region_diversity(b01) – intra-year region cycling mortification_cooccurrence(b01) – joint alert chi² + Cramer’s V disciplinary_medical_overlap(b01) – disc × med-protect co-occurrence embedding_distribution(b02) – total-days survival (lognormal vs Pareto AIC) intra_year_transition_matrix(a01) – Markov regA->regA’ within FY path_complexity_gini(b01) – Gini split by (year × region) region_alert_state_richness(b01) – distinct (region × combo) per cell regC_demog_contingency(b01) – multi-region × Gender × Age χ² irr_glmm_vm(b01) – Poisson + NB IRR for vm

Each emits a RichResult.

NOTE: OTIS UniqueIndividual_ID is anonymised as YYYY-XXXXX-AA, where AA is the dataset acronym (RC for a01, SG for b01/b02, DC for d01). The YYYY prefix locks each ID to one fiscal year, AND IDs are re-randomized per dataset file even within the same year. So longitudinal individual-level linkage and cross-dataset linkage are both not possible by design. All metrics above are either aggregate (b09 counts), within-year individual (b01), or within-year total-days (b02). See docs/source/methods/otis_linkage.md.

morie.otis_churn.gini(x)[source]¶

Gini coefficient of non-negative counts; 0 for an empty or all-zero input.

Computed as (n + 1 - 2 * sum(cumsum(sorted x)) / sum(x)) / n.

Examples

>>> from morie.fn import _array_core as np
>>> gini(np.array([1, 1, 1, 1])), gini(np.array([0, 0, 0, 1]))
(0.0, 0.75)

Parameters:: x (marr)
Return type:: float

morie.otis_churn.repeat_placement_concentration(df=None)[source]¶

Goffman’s ‘cyclical inmate’ – heavy concentration of placements among a small fraction of individuals.

Uses OTIS b09 (NumberPlacements × NumberIndividuals).

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_churn.within_year_placement_count(df=None)[source]¶

Distribution of segregation placements per (individual × fiscal year) cell.

Because OTIS UniqueIndividual_ID is anonymised per fiscal year (YYYY-XXXXX-AA), each (id, year) cell is one anonymous person-year. This metric describes how concentrated the placements are within a fiscal year – Goffman’s intra-year cycling. Cross-year readmission is not measurable in OTIS by design. See OTIS_LINKAGE.md.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_churn.within_year_region_diversity(df=None)[source]¶

Distribution of distinct regions per individual within one fiscal year.

Because OTIS UniqueIndividual_ID is anonymised per fiscal year (YYYY-XXXXX-AA), this metric measures intra-year region cycling only – not multi-year mobility. See OTIS_LINKAGE.md.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_churn.mortification_cooccurrence(df=None)[source]¶

Goffman’s ‘mortification of self’ is operationalised here as concurrent stigmatising-alert flags on a single placement.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_churn.disciplinary_medical_overlap(df=None)[source]¶

Goffman’s ‘tinkering trades’ tension: the same institution classifies the same person via medical AND disciplinary rationales. We look at co-occurrence on individual records.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_churn.embedding_distribution(df=None)[source]¶

TotalAggregatedDays_Segregation distribution per individual per fiscal year. Compare lognormal vs Pareto fit via AIC.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_churn.intra_year_transition_matrix(df=None)[source]¶

Intra-year region-to-region transition matrix from OTIS placements.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_churn.path_complexity_gini(df=None)[source]¶

Gini of placements-per-(id × year) cells, split by (year, region).

Extends within_year_placement_count by reporting per-region and per-year Gini coefficients, exposing where Goffmanian cycling is most concentrated.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_churn.region_alert_state_richness(df=None)[source]¶

Distinct (region × alert-combo) states occupied per (id × year).

Goffman’s mortification × institutional-mobility hybrid: how many distinct role-zones (region paired with alert presentation) does a person traverse within one fiscal year?

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_churn.regC_demog_contingency(df=None)[source]¶

Multi-region path indicator (regC = number of distinct regions visited within FY) × Gender × Age contingency, with chi² + Cramer’s V.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_churn.irr_glmm_vm(df=None)[source]¶

Poisson + Negative-Binomial GLM IRR table for vm ~ T_high_ac plus demographic covariates, on the (UniqueIndividual_ID, FY) cell.

A simplified implementation of the OTIS-RC glmmTMB pipeline: no random effect (statsmodels can’t easily do mixed NB on this scale). Provides Poisson and NB2 IRR side by side as a robustness pair.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.otis_churn.analyze_all(out_dir=None)[source]¶

Run every OTIS churn analysis and write each result as text and JSON.

An analysis that fails is recorded as a result carrying the error as a warning, so one failure does not stop the rest.

Parameters:: out_dir (Path | None) – output directory (created if needed); None uses the default.
Returns:: Results keyed by analysis name.
Return type:: dict[str, RichResult]

morie.otis_datasets – registry + loader for all 29 OTIS CSVs.

The Ontario Tracking Information System (OTIS) is the open-data feed for Ontario’s adult correctional system, published by the Ministry of the Solicitor General’s Service Management and Oversight Branch under the Jahn settlement framework. The 29 datasets in the official A01RCDD release fall into 4 series:

a-series (1 dataset) – Restrictive confinement detailed
(the canonical per-day record file of the OTIS-RC analyses)

b-series (9 datasets) – Segregation placement records c-series (12 datasets) – Individuals in segregation & restrictive

confinement

d-series (7 datasets) – Deaths in custody

CSV path convention:: data/datasets/OTIS/<id>_<short_label>.csv

Where <id> is the canonical dataset name from the official Ontario data dictionary (a01, b01, c03, d05, …).

Public API:

DATASET_REGISTRY – dict mapping id -> metadata load_otis_dataset(id) – returns DataFrame list_otis_datasets() – list of all (id, description, n_cols) download_otis_dataset(id, target_dir=None)

– fetch a fresh copy from data.ontario.ca
via the CKAN action API

download_all_otis(target_dir=None): – fetch every file in the package

OTIS_DATA_DIR – Path OTIS_CKAN_PACKAGE – package id on data.ontario.ca

References

Ministry of the Solicitor General. Data on Inmates in Ontario.: https://data.ontario.ca/dataset/data-on-inmates-in-ontario
Ministry of the Solicitor General. Restrictive Confinement,: Segregation and Deaths in Custody Data Dictionary, 2025-11-03. https://data.ontario.ca/dataset/data-on-inmates-in-ontario/resource/d83fe893-9634-4794-a0c1-c17bf619a95a

class morie.otis_datasets.OtisDataset(id, series, csv_filename, description, columns, panel, primary_metric)[source]¶

Bases: object

One OTIS dataset’s metadata.

Parameters:

id (str)
series (str)
csv_filename (str)
description (str)
columns (tuple[str, ...])
panel (bool)
primary_metric (str)

id: str¶

series: str¶

csv_filename: str¶

description: str¶

columns: tuple[str, ...]¶

panel: bool¶

primary_metric: str¶

property csv_path: Path¶

morie.otis_datasets.load_otis_dataset(dataset_id, data_dir=None, *, download=True)[source]¶

Load one OTIS dataset by id (b01, c03, d05, …).

Returns a pandas DataFrame with the columns documented in DATASET_REGISTRY[dataset_id].columns.

The OTIS CSVs are Ontario open data. If the file is not present locally and download is true (default), it is fetched once from the Ontario CKAN portal into the per-user cache and reused thereafter. Set download=False or $MORIE_OTIS_NO_DOWNLOAD to require a local copy instead.

Parameters:

dataset_id (str)
data_dir (Path | str | None)
download (bool)

Return type:

DataFrame

morie.otis_datasets.list_otis_datasets()[source]¶

List all 29 OTIS datasets as (id, description, n_columns).

Return type:: list[tuple[str, str, int]]

morie.otis_datasets.download_otis_dataset(dataset_id, target_dir=None, *, overwrite=False)[source]¶

Fetch one OTIS CSV from CKAN and write to target_dir.

Returns the local path to the downloaded file. No-op if the file already exists and overwrite=False.

Parameters:

dataset_id (str)
target_dir (str | Path | None)
overwrite (bool)

Return type:

morie.otis_datasets.download_all_otis(target_dir=None, *, overwrite=False, skip_missing=True)[source]¶

Fetch every CSV in the OTIS registry from CKAN.

Returns {dataset_id: local_path} for successfully downloaded files. When skip_missing=True (default), files that aren’t found on CKAN are silently skipped; otherwise the function raises.

Parameters:

target_dir (str | Path | None)
overwrite (bool)
skip_missing (bool)

Return type:

dict[str, Path]

morie.otis_datasets.otis_describe(dataset_id, language='en')[source]¶

Describe an OTIS dataset by combining registry + parsed dictionary.

Looks up dataset_id in DATASET_REGISTRY, then attempts to attach the matching parsed schema from morie.dataset_dictionary.load_otis_dictionary() so users get both the canonical column list AND the rich English / French descriptions from the published data dictionary.

Parameters:

dataset_id (str) – e.g. "b01", "c03".
language (str, default "en") – "en" or "fr" for the localised description text.

Returns:

payload keys: dataset_id, series, csv_filename, columns_registry, schema_columns (from the DatasetSchema if found, else None), description, language, schema_present, n.

Return type:

morie.otis_datasets.otis_validate_against_dictionary(dataset_id=None)[source]¶

Cross-validate OTIS CSVs against the parsed data dictionary.

For each dataset in DATASET_REGISTRY (or just the one named by dataset_id), loads the CSV header, looks up the matching parsed schema, and reports columns present in the CSV but absent in the dictionary, and vice versa.

Surfaces — but does NOT attempt to repair — the small set of known upstream annotation drifts in the Ontario release:

c05 dictionary filename says ..._race_by_region but the on-disk CSV is ..._religion_by_region (variable list confirms religion, not race; the dict filename is a typo)
_id appears as the first variable in every dictionary entry but is a CKAN-visualiser-only synthetic column — never present in the published CSVs
The a01_<Year>_… row in the dictionary is a per-year split note about MentalHealth_Alert only, not a separate dataset

Returns:: payload keys: per_dataset (list of dicts with dataset_id, missing_in_csv, extra_in_csv, schema_present), known_drifts (the three above), n_datasets, n_with_drift, n.
Return type:: RichResult
Parameters:: dataset_id (str | None)

morie.otis_tps_overlay – cross-link OTIS (Ontario corrections) with TPS (Toronto police) data.

Both feeds touch Toronto, so meaningful overlays:

Year-over-year correlation – does annual segregation/RC use in the OTIS “Toronto” region track annual TPS incident counts?
Per-OTIS-region rollups – total seg/RC counts per region, with the Toronto-region row joinable to TPS aggregates.
Crime × confinement composite – combine TPS composite-risk index (per-neighbourhood) with OTIS seg/RC totals (Toronto-region only) to surface high-correlation years.

All emit RichResult.

morie.otis_tps_overlay.yoy_correlation(tps_categories=None, *, sample_rows=50000)[source]¶

Year-by-year correlation between OTIS Toronto-region segregation placements and TPS crime incident counts (per category).

Parameters:

tps_categories (list[str] | None)
sample_rows (int | None)

Return type:

morie.otis_tps_overlay.per_region_rollup(*, tps_total_by_year=None)[source]¶

OTIS seg/RC totals per region × year, with the Toronto row flagged for TPS-overlay use. Also splits by gender.

Parameters:: tps_total_by_year (Series | None)
Return type:: RichResult

morie.otis_tps_overlay.composite_overlay(*, sample_rows=30000)[source]¶

For each TPS category, compute year-aligned correlation with OTIS Toronto-region segregation. Same idea as yoy_correlation but uses ALL fiscal years jointly via concat.

Parameters:: sample_rows (int | None)
Return type:: RichResult

morie.otis_tps_overlay.analyze_all(out_dir=None, *, sample_rows=30000)[source]¶

Run the OTIS x TPS overlay analyses (regional roll-up, year-over-year correlation) and write each as text and JSON.

Parameters:

out_dir (Path | None) – output directory (created if needed); None uses the default.
sample_rows (int | None) – TPS rows sampled for the correlation; None uses all.

Returns:

Results keyed by analysis name; a failed analysis carries its error as a warning.

Return type:

TPS — Toronto Police Service¶

morie.tps_io – multi-format readers for TPS open-data exports.

Each TPS category at data/datasets/TPS/<Category>/ has 9 sibling format exports of the same incident records:

CSV – pandas.read_csv (canonical fast path) Excel – pandas.read_excel via openpyxl GeoJSON – pure JSON, geometry as Python list-of-coords FeatureCollection – ESRI JSON variant of GeoJSON KML / KMZ – zipfile + ElementTree, geometry as coord list GeoPackage – sqlite3, geometry stored as WKB blobs SQLiteGeodatabase – sqlite3, ESRI variant of GeoPackage Shapefile – needs pyshp; graceful degrade if not installed FileGeoDatabase – needs fiona / gdal; graceful degrade

This module returns a pandas.DataFrame for every format, with the geometry (when available) attached as a geometry column whose values are simple Python lists/tuples – no shapely / geopandas dependency.

For NeighbourhoodCrimeRates the geometry is Polygon; for incident datasets the geometry is Point. Either way, the geometry column contains the (longitude, latitude) tuples in WGS84.

Public API:: load_tps(name, format=’csv’, nrows=None) -> pd.DataFrame list_tps_formats(name) -> dict[str, Path] available_formats() -> list of supported format names

morie.tps_io.load_tps(name, format='csv', nrows=None)[source]¶

Load TPS dataset name in the given format.

Parameters:

name ("Assault", "Homicides", … (case-insensitive))
format (one of csv | excel | geojson | featurecollection |) – kml | geopackage | sqlitegeodatabase | shapefile. “filegeodatabase” requires fiona/gdal – not supported on this build.
nrows (sample size cap; None = full dataset.)

Return type:

DataFrame

morie.tps_io.list_tps_formats(name)[source]¶

Map format name -> path of the file that would be loaded. For formats not present on disk, the path is omitted from the dict.

Parameters:: name (str)
Return type:: dict[str, Path]

morie.tps_io.available_formats()[source]¶

List of formats this build can actually load.

Return type:: list[str]

morie.tps_datasets – registry + loader for Toronto Police Service public crime datasets.

13 categories at data/datasets/TPS/<Category>/CSV/:: Assault, AutoTheft, BicycleTheft, BreakandEnter, CommunitySafetyIndicators, HateCrimes, Homicides, IntimatePartnerAndFamilyViolence, NeighbourhoodCrimeRates, Robbery, ShootingAndFirearmDiscarges, TheftFromMovingVehicle, TheftOver

Each category also has Excel/Shapefile/GeoJSON/etc; this module reads the CSV by default but can be pointed at any sibling format.

Schema (per-incident, varies slightly by category):: OBJECTID, EVENT_UNIQUE_ID, REPORT_DATE, OCC_DATE, REPORT_YEAR, REPORT_MONTH, REPORT_DAY, REPORT_DOY, REPORT_DOW, REPORT_HOUR, OCC_YEAR, OCC_MONTH, OCC_DAY, OCC_DOY, OCC_DOW, OCC_HOUR, DIVISION, LOCATION_TYPE, PREMISES_TYPE, UCR_CODE, UCR_EXT, OFFENCE, CSI_CATEGORY, HOOD_158, NEIGHBOURHOOD_158, HOOD_140, NEIGHBOURHOOD_140, LONG_WGS84, LAT_WGS84, x, y
Public API:: TPS_REGISTRY – dict: name -> TpsDataset load_tps_dataset(name) -> pd.DataFrame list_tps_datasets() -> list of (name, n_rows_est, …)

class morie.tps_datasets.TpsDataset(name, description, primary_date, has_geometry)[source]¶

Bases: object

One TPS open-data category.

Parameters:

name (str)
description (str)
primary_date (str)
has_geometry (bool)

name: str¶

description: str¶

primary_date: str¶

has_geometry: bool¶

property csv_dir: Path¶

csv_path(fname=None)[source]¶

Parameters:: fname (str | None)
Return type:: Path

morie.tps_datasets.load_tps_dataset(name, csv_filename=None, nrows=None, *, format=None)[source]¶

Load one TPS dataset by category name.

name is case-insensitive. Pass nrows=N for a quick sample while developing on the largest tables (Assault is 254k rows).

format (optional) – load from a non-CSV format. Valid: csv, excel, geojson, featurecollection, kml, geopackage, sqlitegeodatabase, shapefile. Default csv. Geometric formats attach a geometry column (Python coord lists). See morie.tps_io.load_tps for details.

Parameters:

name (str)
csv_filename (str | None)
nrows (int | None)
format (str | None)

Return type:

DataFrame

morie.tps_datasets.list_tps_datasets()[source]¶

List all TPS datasets as (name, description, primary_date_col).

Return type:: list[tuple[str, str, str]]

morie.tps_crime – cross-category crime analyses for TPS datasets.

Builds on top of morie.tps_datasets + tps_io to compare incidents ACROSS the 13 TPS categories. Useful for: - Comparing year-over-year trends side-by-side - Identifying neighbourhoods that are ‘hot’ across many crime types - Computing bivariate spatial correlation (Moran’s BV) between two

crime categories on the same neighbourhoods

Building a composite crime-risk index per neighbourhood

All emit RichResult.

morie.tps_crime.yoy_panel(categories=None, *, sample_rows=50000)[source]¶

Side-by-side year-over-year panel across TPS categories.

Parameters:

categories (list[str] | None)
sample_rows (int | None)

Return type:

morie.tps_crime.composite_index(categories=None, *, sample_rows=50000, weights=None, top_n=25)[source]¶

Build a composite per-neighbourhood crime-risk index.

For each category, count incidents per HOOD_158, z-standardise across neighbourhoods, then sum (or weight-and-sum) to get a single composite. Higher index = neighbourhood with elevated incidence across many crime types.

weights lets you tune (e.g. weights={‘Homicides’: 5.0, ‘BicycleTheft’: 0.5}). Default = unit weights for all loaded cats.

Parameters:

categories (list[str] | None)
sample_rows (int | None)
weights (dict[str, float] | None)
top_n (int)

Return type:

morie.tps_crime.bivariate_morans_i(cat_a, cat_b, *, sample_rows=50000, k_neighbours=5)[source]¶

Bivariate Moran’s I – does category A’s count in a hood correlate with category B’s count in NEIGHBOURING hoods?

Parameters:

cat_a (str)
cat_b (str)
sample_rows (int | None)
k_neighbours (int)

Return type:

morie.tps_crime.category_correlation_matrix(*, sample_rows=50000)[source]¶

Pearson correlation of per-hood incident counts across all 13 TPS categories.

Parameters:: sample_rows (int | None)
Return type:: RichResult

morie.tps_csi – Statistics Canada Crime Severity Index (CSI) weights for the 9 Toronto Police Service open-data categories.

The Crime Severity Index, introduced in Wallace et al.(2009) cite{Wallace2009CSI} and maintained by the Canadian Centre for Justice and Community Safety Statistics, weights each emph{Criminal Code} offence by:

weight(offence) = (avg sentence in days)
× (proportion of offenders incarcerated)

so that violent offences with high incarceration rates and long sentences contribute disproportionately to a city’s per-capita CSI score, while minor property offences contribute less. This module exposes the weights used for the 9 TPS open-data categories (Assault, Auto Theft, Bicycle Theft, Break and Enter, Homicide, Robbery, Shooting and Firearm Discharges, Theft from Motor Vehicle, Theft Over) and provides per-year + per-neighbourhood CSI aggregates.

Important caveats¶

TPS open-data categories aggregate over multiple Criminal Code sub-offences (e.g.``Assault’’ covers Assault Level 1, Level 2 with a weapon, Level 3 aggravated, plus a small share of family and assault-of-peace-officer offences). The weights here are emph{representative blends} reflecting the typical distribution of sub-offences within each TPS category for FY2023; for an exact reproduction of Statistics Canada’s CSI for the City of Toronto, one must work directly from the CCJS UCR microdata (which is not in TPS open data).
The weights below are pinned to the last published Statistics Canada methodology update (Reweighting the Crime Severity Index'', Catalogue 85-004-X) and the Toronto-specific override tables in CCJS Annual Statistics 2023. Newer weight revisions (StatsCan revises every 5 years) may shift values by 5--15\% without changing relative ordering. Override via the ``weights keyword argument when calling functions in this module.
Statistics Canada itself reports two CSI variants: Total CSI'' and ``Violent CSI''. Functions here default to the Total CSI weights but accept a ``variant=`'violent`' argument to use violent-only weights, where non-violent categories (BBE, theft) are zeroed.

References

Wallace, M., Turner, J., Babyak, C., & Matarazzo, A. (2009).: Measuring Crime in Canada: Introducing the Crime Severity Index and Improvements to the Uniform Crime Reporting Survey. Statistics Canada Catalogue 85-004-X.
Statistics Canada (2024). Crime Severity Index, Census: Metropolitan Areas, 2023. Catalogue 35-10-0190-01.

morie.tps_csi.csi_weight(category, *, variant='total', weights=None)[source]¶

Return the CSI weight for a TPS open-data category.

Parameters:

category (str) – TPS category name (e.g. “Assault”, “Homicides”).
variant ({"total", "violent"}) – Which CSI variant to use.
weights (dict[str, float] | None) – Optional override. When provided, takes precedence.

Return type:

morie.tps_csi.csi_per_year(counts_per_year, *, variant='total', weights=None, population=None, per_capita_unit=100000, rebase_to_year=None, rebase_to_value=100.0)[source]¶

Compute Toronto’s CSI per fiscal year from per-category counts.

Returns a DataFrame indexed by year with columns:: raw_weighted_sum – Σ w_c · n_{c,year} total_count – Σ n_{c,year} population – Toronto population that year csi_per_capita – raw_weighted_sum / population × per_capita_unit simple_count_rate – total_count / population × per_capita_unit

(for comparison)

counts_per_year may be either a dict-of-dicts or a long-format DataFrame with columns year, category, count.

Parameters:

counts_per_year (dict[int, dict[str, int]] | DataFrame)
variant (Literal['total', 'violent'])
weights (dict[str, float] | None)
population (dict[int, int] | None)
per_capita_unit (int)
rebase_to_year (int | None)
rebase_to_value (float)

Return type:

DataFrame

morie.tps_csi.csi_per_neighbourhood(counts_per_hood, *, variant='total', weights=None)[source]¶

CSI per ward (HOOD_158).

Mirrors csi_per_year() but groups by neighbourhood rather than fiscal year. Population is not divided here because TPS open data does not ship a per-ward population table; users are expected to merge in the City of Toronto Open Data NeighbourhoodCrimeRates per-ward population for per-capita rates. Returns the un-normalised weighted sum + total count per ward.

Parameters:

counts_per_hood (dict[int, dict[str, int]] | DataFrame)
variant (Literal['total', 'violent'])
weights (dict[str, float] | None)

Return type:

DataFrame

morie.tps_csi.analyze_csi_from_tps_dataframes(dfs, *, year_col='OCC_YEAR', hood_col='HOOD_158', variant='total')[source]¶

High-level orchestration: take {category: tps_df} of the 9 TPS open-data feeds and return per-year + per-ward CSI.

Parameters:

dfs (dict[str, DataFrame]) – Keys must be a subset of CSI_CATEGORIES. Values are the TPS open-data CSV-loaded DataFrames (one row per incident).
year_col (str)
hood_col (str)
variant (Literal['total', 'violent'])

Return type:

morie.tps_temporal – temporal analyses for TPS crime data.

year_over_year_trend: linear regression on yearly counts
seasonal_pattern: monthly / DOW / hour-of-day cyclic stats
arima_forecast: simple ARIMA(1,1,1) on monthly counts
changepoint_detection: PELT-style changepoint on yearly counts

All emit RichResult.

morie.tps_temporal.year_over_year_trend(df, *, year_col='OCC_YEAR', ds_name='?')[source]¶

Linear regression of incident counts vs year.

Parameters:

df (DataFrame)
year_col (str)
ds_name (str)

Return type:

morie.tps_temporal.seasonal_pattern(df, *, ds_name='?')[source]¶

Month / day-of-week / hour-of-day cyclic patterns + chi-square test of uniformity.

Parameters:

df (DataFrame)
ds_name (str)

Return type:

morie.tps_temporal.changepoint_detection(df, *, year_col='OCC_YEAR', ds_name='?')[source]¶

Simple Pettitt-style change-point on yearly counts.

Parameters:

df (DataFrame)
year_col (str)
ds_name (str)

Return type:

morie.tps_temporal.arima_forecast(df, *, h=12, ds_name='?')[source]¶

ARIMA(1,1,1) on monthly counts; forecast h periods ahead.

Parameters:

df (DataFrame)
h (int)
ds_name (str)

Return type:

morie.tps_spatial – spatial analyses for TPS crime data.

Wires the existing morie.fn spatial primitives to TPS data: - Moran’s I (global) on neighbourhood-level counts - LISA (Local Moran’s I) for hot/cold spots - Ripley’s K for point-pattern clustering - Kernel density estimation - Gi*/Getis-Ord hotspot detection

All emit RichResult.

morie.tps_spatial.morans_i_neighbourhood(df, *, hood_col='HOOD_158', ds_name='?', k_neighbours=5)[source]¶

Compute global Moran’s I on neighbourhood-level incident counts.

Builds a k-NN spatial weights matrix from neighbourhood centroids (mean LAT/LONG of incidents in each hood as a proxy centroid), then runs Moran’s I on the count vector.

Parameters:

df (DataFrame)
hood_col (str)
ds_name (str)
k_neighbours (int)

Return type:

morie.tps_spatial.local_morans_i(df, *, hood_col='HOOD_158', ds_name='?', k_neighbours=5, top_n=20)[source]¶

LISA – local Moran’s Ii per neighbourhood, with hot/cold spot classification.

Parameters:

df (DataFrame)
hood_col (str)
ds_name (str)
k_neighbours (int)
top_n (int)

Return type:

morie.tps_spatial.kde_density(df, *, bandwidth=0.005, ds_name='?')[source]¶

Kernel density estimate of incident lat/long.

Returns summary stats; not the full grid (too big for a RichResult).

Parameters:

df (DataFrame)
bandwidth (float)
ds_name (str)

Return type:

morie.tps_spatial_advanced – heavyweight spatial statistics for TPS.

Builds on morie.tps_spatial (Moran’s I global, LISA, KDE) with:

ripley_k point-pattern clustering at multiple radii
getis_ord_g_star local hot/cold-spot z-scores (Gi*)
dbscan_clusters density-based clusters on lat/lon
kernel_intensity_at_centroids smoothed counts at neighbourhood centroids
polygon_morans_i queen-contiguity Moran’s I from polygons

Polygon contiguity uses the actual GeoJSON polygons from morie.tps_io.load_tps(NeighbourhoodCrimeRates, format=’geojson’), not the centroid-only k-NN approximation in tps_spatial.

morie.tps_spatial_advanced.ripley_k(df, *, ds_name='?', radii_km=None, max_n=5000)[source]¶

Ripley’s K function – for each radius r, expected number of other points within r of a typical point, normalised by intensity.

For a homogeneous Poisson process K(r) = π r²; values above this indicate clustering, below indicate regularity.

Parameters:

df (DataFrame)
ds_name (str)
radii_km (list[float] | None)
max_n (int)

Return type:

morie.tps_spatial_advanced.getis_ord_g_star(df, *, ds_name='?', hood_col='HOOD_158', k_neighbours=5, top_n=20)[source]¶

Local Getis-Ord Gi* statistic per neighbourhood.

z-score interpretation:

Gi* > 1.96 = significant hotspot at α=0.05 (high values: surrounded by other high values)

Gi* < -1.96 = significant cold spot

Parameters:

df (DataFrame)
ds_name (str)
hood_col (str)
k_neighbours (int)
top_n (int)

Return type:

morie.tps_spatial_advanced.dbscan_clusters(df, *, ds_name='?', eps_km=0.25, min_samples=30, max_n=30000)[source]¶

DBSCAN on point coordinates. Returns cluster summary.

Parameters:

df (DataFrame)
ds_name (str)
eps_km (float)
min_samples (int)
max_n (int)

Return type:

morie.tps_spatial_advanced.polygon_morans_i(*, ds_name='NeighbourhoodCrimeRates', value_col_prefix='ASSAULT_RATE', year=2024, k_neighbours=5)[source]¶

Polygon-aware Moran’s I using actual neighbourhood polygons from the GeoJSON export of NeighbourhoodCrimeRates, with a per-year crime-rate value column.

Parameters:

ds_name (str)
value_col_prefix (str)
year (int)
k_neighbours (int)

Return type:

morie.tps_spatial_advanced.bivariate_moran(*, ds_name='NeighbourhoodCrimeRates', x_col_prefix='ASSAULT_RATE', y_col_prefix='HOMICIDE_RATE', year=2024, k_neighbours=5)[source]¶

Bivariate Moran’s I between two crime-rate columns.

Generalises polygon_morans_i to two attributes: measures the cross-correlation between attribute X at location i and attribute Y at neighbouring locations j. Polygon centroids + k-nearest-neighbour weights, same construction as polygon_morans_i.

\[I_{xy} = \frac{n}{S_0}\, \frac{\sum_i \sum_j w_{ij}\, z^x_i\, z^y_j} {\sqrt{\sum_i (z^x_i)^2 \cdot \sum_i (z^y_i)^2}}\]

where $z^x$ and $z^y$ are z-scored attributes.

Parameters:

ds_name (str)
x_col_prefix (str)
y_col_prefix (str)
year (int)
k_neighbours (int)

Return type:

morie.tps_spatial_advanced.moran_sweep_heatmap(*, ds_name='NeighbourhoodCrimeRates', category_prefixes=None, years=None, k_neighbours=5)[source]¶

Run univariate Moran’s I across (category x year) and return the resulting matrix as a RichResult payload.

Parameters:

category_prefixes (tuple of str, optional) – Crime-rate column prefixes to sweep. Defaults to the 9 published TPS categories.
years (tuple of int, optional) – Years to sweep. Defaults to (2014, 2015, …, 2024).
ds_name (str)
k_neighbours (int)

Return type:

morie.tps_stochastic – stochastic-physics-of-crime analyses on TPS.

Modelled on Frenkel-style crime-physics (Hawkes self-exciting + Fokker-Planck density evolution + Langevin SDE), with practical forecasting baselines (SARIMA, XGBoost-when-available) for hold-out comparison.

Functions - hawkes_temporal_fit(df) – fit μ, κ, ω of self-exciting kernel,

compute branching ratio + AIC/BIC

sarima_forecast(df, h=12) – seasonal ARIMA, train/test MAPE
prophet_forecast(df, h=12) – Prophet trend+seasonality if installed
langevin_simulate(…) – Euler-Maruyama Ornstein-Uhlenbeck path
fokker_planck_grid(df) – finite-difference density evolution
model_compare(df) – AIC across baselines + SDE

All emit RichResult; figure_path payload entry written when matplotlib is available (saved as PNG under data/manifest/outputs/figures/tps_stochastic/).

References (cited in description blocks): - Mohler et al. 2011, “Self-exciting point process modeling of crime.” - Pitcher 2010, “Adding integrated nested Laplace approximations to

the Bayesian forecasting toolkit.”

Stochastic physics of crime literature (Short, D’Orsogna, Bertozzi 2010).

morie.tps_stochastic.hawkes_temporal_fit(df, *, ds_name='?', max_n=5000)[source]¶

Fit a temporal-only exponential Hawkes process to incident times.

Returns μ (background rate), κ (branching ratio), ω (decay), AIC/BIC.

Parameters:

df (DataFrame)
ds_name (str)
max_n (int)

Return type:

morie.tps_stochastic.sarima_forecast(df, *, ds_name='?', h=12, order=(1, 1, 1), seasonal=(0, 1, 1, 12))[source]¶

Seasonal ARIMA forecast on monthly incident counts.

Parameters:

df (DataFrame)
ds_name (str)
h (int)
order (tuple[int, int, int])
seasonal (tuple[int, int, int, int])

Return type:

morie.tps_stochastic.langevin_simulate(df, *, ds_name='?', n_paths=100, T_days=365, dt=1.0, seed=42)[source]¶

Euler-Maruyama simulation of an Ornstein-Uhlenbeck process fitted to incident counts:

dX_t = θ(μ − X_t) dt + σ dW_t

Fits θ, μ, σ via OLS on first-differences, then generates n_paths forward simulations of length T_days.

Parameters:

df (DataFrame)
ds_name (str)
n_paths (int)
T_days (int)
dt (float)
seed (int)

Return type:

morie.tps_stochastic.fokker_planck_grid(df, *, ds_name='?', n_grid=64, n_steps=200)[source]¶

1-D Fokker-Planck density evolution of daily incident counts with the OU drift+diffusion fitted in langevin_simulate.

Uses Crank-Nicolson finite-difference scheme with reflective boundaries on a grid spanning [0, max(observed)*1.5].

Parameters:

df (DataFrame)
ds_name (str)
n_grid (int)
n_steps (int)

Return type:

morie.tps_stochastic.analyze(name='Assault', *, sample_rows=50000)[source]¶

Run the four stochastic-physics analyses on one TPS category.

Parameters:

name (str)
sample_rows (int | None)

Return type:

morie.tps_hawkes_advanced – non-stationary Hawkes with non-exponential kernels.

Implements the Kwan-Chen-Dunsmuir (2024, arXiv:2408.09710v1) methodology for Hawkes process likelihood inference when the baseline intensity is time-varying and the excitation kernel is non-exponential (so the intensity process is non-Markovian).

Companion to morie.tps_stochastic – that module contains the classical exponential-kernel constant-baseline (Markovian / Mohler 2011) fit; this module adds:

Gamma, Weibull, and power-law (Lomax) excitation kernels.

Sinusoidal-with-trend time-varying baseline ν(t).

Time-rescaling residuals (Brown et al. 2002) for goodness-of-fit via Kolmogorov-Smirnov + Q-Q plots.

Eight-way model comparison via AIC and an explicit Markovian-vs-non-Markovian likelihood-ratio surface.

The complete intensity function is

λ(t) = ν(t) + ∫_0^{t-} g(t - s) dN_s

with kernel decomposition g(u) = η · g̃(u) where η ∈ (0, 1) is the branching ratio (mean offspring per event) and g̃ is a probability density on [0, ∞). Stationarity requires η < 1.

morie.tps_hawkes_advanced.fit_hawkes_general(t, T, kernel_kind='exponential', baseline_kind='constant', *, method='auto', eps=1e-09)[source]¶

MLE of a non-stationary Hawkes process.

method chooses how the likelihood is evaluated (all with the analytic gradient):

"exact": Ozaki’s O(n) recursion for the exponential kernel; for Weibull and gamma the double sum stops where the kernel underflows to exactly 0 (the same value as the full sum); Lomax the full O(n^2) sum.
"soe": the completely monotone kernels (Lomax; gamma with shape < 1) as a sum of exponentials (Beylkin & Monzon 2010), relative error eps per intensity, O(n K); a gamma kernel with shape >= 1 is truncated at eps.
"truncate": each event excites only lags with kernel tail mass above eps (light tails).
"em": the EM algorithm (Veen & Schoenberg 2008): the same MLE, a different route.
"inar": Kirchner’s (2017) INAR(p) least-squares estimator on binned counts, constant baseline only: a different (fast, approximate) estimator, also used for starting values.
"auto" (default): "exact" for the exponential kernel, "truncate" for Weibull, "soe" for Lomax and gamma ("exact" without the compiled core).

Returns a dict with theta, nll, aic, bic, branching_ratio, baseline_params, kernel_params, and the time-rescaling KS statistic.

Parameters:

t (marr)
T (float)
kernel_kind (Literal['exponential', 'gamma', 'weibull', 'lomax'])
baseline_kind (Literal['constant', 'sinusoidal'])
method (str)
eps (float)

Return type:

morie.tps_hawkes_advanced.splitmix_uniforms(n, seed)[source]¶

n uniforms on [0, 1) from splitmix64 (Steele, Lea & Flood 2014): the same numbers as rmoriebricklayer::cunif(n, seed), so the R and Python arms draw identical values where their results must agree (the within-day jitter of tied dates, a subsample).

Examples

>>> [round(u, 12) for u in splitmix_uniforms(2, 42)]
[0.741564878772, 0.159910392877]

Parameters:

n (int)
seed (int)

Return type:

list[float]

morie.tps_hawkes_advanced.hawkes_advanced_fit(df, *, kernel='gamma', baseline='sinusoidal', ds_name='?', max_n=None, method='auto', eps=1e-09)[source]¶

Fit a single (kernel, baseline) combination with figures.

A sibling to morie.tps_stochastic.hawkes_temporal_fit which is locked to the (exponential, constant) Markovian special case.

Parameters:

df (DataFrame)
kernel (Literal['exponential', 'gamma', 'weibull', 'lomax'])
baseline (Literal['constant', 'sinusoidal'])
ds_name (str)
max_n (int | None)
method (str)
eps (float)

Return type:

morie.tps_hawkes_advanced.compare_hawkes_kernels(df, *, ds_name='?', max_n=None, baselines=('constant', 'sinusoidal'), kernels=('exponential', 'gamma', 'weibull', 'lomax'), method='auto', eps=1e-09)[source]¶

Fit every (kernel, baseline) combination and rank by AIC.

Mirrors Section 5 (numerical examples) of Kwan-Chen-Dunsmuir 2024: the Markovian classical Hawkes is the (exponential, constant) row; the non-Markovian non-stationary models are everything else.

Parameters:

df (DataFrame)
ds_name (str)
max_n (int | None)
baselines (tuple[Literal['constant', 'sinusoidal'], ...])
kernels (tuple[Literal['exponential', 'gamma', 'weibull', 'lomax'], ...])
method (str)
eps (float)

Return type:

morie.tps_hawkes_advanced.hawkes_markovian_vs_nonmarkovian(df, *, ds_name='?', max_n=None, method='auto', eps=1e-09)[source]¶

Focused 2-way comparison: classical exp/const vs gamma/sinusoidal.

The two endpoints of the Kwan-Chen-Dunsmuir framework – quickest to run on the dashboard. max_n=None keeps every event; method and eps as in fit_hawkes_general().

Parameters:

df (DataFrame)
ds_name (str)
max_n (int | None)
method (str)
eps (float)

Return type:

morie.tps_statphysics – Statistical physics of crime applied to TPS.

Implements the 4 canonical methods reviewed by D’Orsogna & Perc, “Statistical physics of crime: A review”, Phys. Life Rev. 12 (2015) 1-21 (arxiv:1411.1743). Each function consumes one TPS category and returns a RichResult with a paragraph-level interpretation.

1. ``sdb_reaction_diffusion(category, …)`` -- Short-D'Orsogna-Brantingham

2008 hot-spot PDE on a Toronto grid:

∂A/∂t = η ∇²A − ω A + θ ρ ∂ρ/∂t = ∇·(D∇ρ − 2 ρ ∇log A) − ρ A + γ

A is the attractiveness field; ρ the criminal density. Crime incidents are absorbed into A, ρ flows up the gradient of A. Stable spots emerge when (η, ω, θ, D, γ) place the system in the localised-spike regime. We fit the steady-state spike count to the observed cluster count from DBSCAN.

2. ``levy_flight_alpha(category)`` -- Brockmann-Hufnagel-Geisel 2006: Lévy-flight diagnostic. We compute step lengths between chronologically-consecutive incidents and fit a power-law tail p(ℓ) ∝ ℓ^{−α} via Hill-MLE on the upper-tail.

3. ``urban_scaling_beta(category)`` -- Bettencourt et al. 2007 / 2010: urban scaling: log(crime_i) = log(Y₀) + β · log(pop_i) + ε for the 158 Toronto wards. β > 1 = super-linear (crime grows faster than population), β = 1 = linear, β < 1 = sub-linear.

4. ``lotka_volterra_police_crime(category)`` -- Lotka-Volterra

predator-prey on a coarse-grained yearly time series:

dx/dt = α x − β x y (crime: prey) dy/dt = δ x y − γ y (police effort: predator)

Fit (α, β, γ, δ) by least-squares against TPS counts (proxy x) and normalised mass-stop / arrest counts (proxy y if available, else one-quarter-of-incidents as a placeholder).

References

D’Orsogna & Perc (2015) §2.1, §3.2, §4.1
Short, D’Orsogna, Brantingham et al. (2008) “Statistical model of criminal behavior”, M3AS 18.
Brockmann, Hufnagel, Geisel (2006) “Scaling laws of human travel”, Nature 439.
Bettencourt, Lobo, Helbing, Kühnert, West (2007) “Growth, innovation, scaling, and the pace of life in cities”, PNAS 104.

morie.tps_statphysics.sdb_reaction_diffusion(category='Assault', *, sample_rows=30000, eta=0.05, omega=0.3, theta=1.5, D=0.1, gamma=0.05, n_steps=800, dt=0.04, nx=90, ny=60, save_fig=True)[source]¶

Solve the Short-D’Orsogna-Brantingham 2008 reaction-diffusion crime hot-spot PDE on a Toronto grid, seeded by observed incidents.

Returns a RichResult with the steady-state hot-spot count, mean attractiveness, mean criminal density, and a comparison to DBSCAN.

Parameters:

category (str)
sample_rows (int | None)
eta (float)
omega (float)
theta (float)
D (float)
gamma (float)
n_steps (int)
dt (float)
nx (int)
ny (int)
save_fig (bool)

Return type:

morie.tps_statphysics.levy_flight_alpha(category='Assault', *, sample_rows=30000, lmin_km=0.5, save_fig=True)[source]¶

Hill-MLE Lévy exponent on step lengths between consecutive incidents in chronological order (proxy for offender mobility).

Parameters:

category (str)
sample_rows (int | None)
lmin_km (float)
save_fig (bool)

Return type:

morie.tps_statphysics.urban_scaling_beta(category='Assault', *, year=2024, save_fig=True)[source]¶

Bettencourt 2007 super-linear urban scaling on the 158 wards.

Regresses log(crime per ward) on log(population per ward) -> β.

Parameters:

category (str)
year (int)
save_fig (bool)

Return type:

morie.tps_statphysics.lotka_volterra_police_crime(category='Assault', save_fig=True)[source]¶

Yearly Lotka-Volterra fit: crime as prey, mass-stop / arrest as predator.

With only 1 series we use the empirical x(t) and back out (α, β) from the LV equilibrium x* = γ/δ, y* = α/β under stationarity. For the predator series we use the cumulative count of incidents -> a crude effort-density proxy; the absolute scale is unidentified but the (α, γ) ratio and the cycle period are.

Parameters:

category (str)
save_fig (bool)

Return type:

morie.tps_statphysics.sdb_turing_demo(*, eta=0.2, omega=0.033, theta=0.56, D=30.0, gamma=0.019, n_steps=6000, dt=0.005, n=80, save_fig=True)[source]¶

Canonical SDB Turing-instability demo on a clean periodic grid.

Reproduces the localized hot-spot lattice from Short, D’Orsogna & Brantingham (2008) Fig. 4 / D’Orsogna & Perc (2015) Fig. 5 – which is what the author’s reference image #3 / #4 shows. Periodic boundaries, homogeneous initial state + Gaussian noise, runs to steady state. Output: 1×3 panel (early / mid / late).

Parameters:

eta (float)
omega (float)
theta (float)
D (float)
gamma (float)
n_steps (int)
dt (float)
n (int)
save_fig (bool)

Return type:

morie.tps_statphysics.inspection_game_phase(n_temptations=20, n_costs=20, n_steps=600, save_fig=True)[source]¶

Helbing-Szolnoki-Perc (2010) cooperator-defector-inspector replicator dynamics; produces the (Temptation T, Inspection cost γ) crime-rate phase diagram from D’Orsogna & Perc 2015 §5 / Image #5.

Three strategies (pure populations only):: C – cooperator P – defector / ‘predator’ O – punisher / inspector

Replicator dynamics with payoff matrix that includes punishment of P by O at cost γ. Steady-state defector fraction -> ‘crime rate’.

Parameters:

n_temptations (int)
n_costs (int)
n_steps (int)
save_fig (bool)

Return type:

morie.tps_statphysics.criminal_network_graph(category='Assault', *, sample_rows=30000, top_n_premises=20, save_fig=True)[source]¶

Co-occurrence network: nodes = top-N premise types, edges = sharing a common neighbourhood, weighted by joint incident count.

Replaces the criminal-role graph in D’Orsogna & Perc Fig. 9 / Diviák et al. (2019) – for our public TPS data we don’t have co-offender records, so we build the closest-meaning network from PREMISES_TYPE × HOOD_158 co-incidence.

Parameters:

category (str)
sample_rows (int | None)
top_n_premises (int)
save_fig (bool)

Return type:

morie.tps_statphysics.analyze_all(categories=None, save_fig=True)[source]¶

Run all 4 stat-phys analyses on each category.

Parameters:

categories (list[str] | None)
save_fig (bool)

Return type:

dict[str, dict[str, RichResult]]

morie.tps_render – clean Toronto map rendering.

Two design rules per the author, 2026-05-07:

NO floating neighbourhood text labels on the map (no “Downtown Yonge-East”, “Yonge-Bay Corridor”, “Mimico Queensway”, …). Hot-spot identification is delivered via the RichResult tables, not via on-canvas text.
Map is rotated ~17° CLOCKWISE in projected space so Lake Ontario’s shoreline sits level horizontally – matching the Sigar Li 2022 “Hotspot Policing for the City of Toronto” poster aesthetic and the Hohl 2024 ALMI homicide-cluster map.

Rotation works in cos-latitude-corrected metres (not raw degrees), so shapes stay metric-true regardless of where in Toronto’s bbox they sit.

Public API¶

project_xy(lat, lon) – degrees -> rotated planar metres (km) render_choropleth(rate_col, year, ...) – polygon choropleth render_dbscan(category, ...) – point-pattern + clusters render_yearly_grid(prefix, years, ...) – small-multiples

morie.tps_render.pretty_label(s)[source]¶

ASSAULT_RATE_2024 -> Assault rate · 2024. Strips underscores and casing so titles & legend labels look like prose.

Parameters:: s (str)
Return type:: str

morie.tps_render.project_xy(lat, lon, *, rot_deg_cw=17.5, lat_c=43.7, lon_c=-79.4)[source]¶

Project (lat, lon) -> (x_km, y_km) with a clockwise rotation.

Equirectangular projection centred at (lat_c, lon_c), then rotated rot_deg_cw degrees clockwise. Returns kilometres east-of-centre (after rotation) and kilometres north-of-centre (after rotation).

Clockwise convention: positive rot_deg_cw rotates the map so a line that previously sloped up-right slopes less (or down-right).

Parameters:

lat (marr | float)
lon (marr | float)
rot_deg_cw (float)
lat_c (float)
lon_c (float)

Return type:

tuple[marr, marr]

morie.tps_render.render_choropleth(*, rate_col='ASSAULT_RATE_2024', title=None, cmap=None, outfile=None, fig_h=7.0, fig_w=12.0, show_ids=True, border_color='#1a1a1a', border_lw=0.7)[source]¶

Polygon choropleth using NeighbourhoodCrimeRates GeoJSON polys.

Style matches the Hohl 2024 ALMI / Sigar Li 2022 Toronto poster: - thick black polygon borders (border_color/border_lw) - sequential cmap chosen per category (default Reds family) - small numeric polygon-ID labels (show_ids=True) – IDs only,

never neighbourhood names per the author 2026-05-07

title and colour-bar legend with proper spaces (no underscores)

Parameters:

rate_col (str)
title (str | None)
cmap (str | None)
outfile (str | Path | None)
fig_h (float)
fig_w (float)
show_ids (bool)
border_color (str)
border_lw (float)

Return type:

morie.tps_render.render_quad(category='Homicides', *, year=2024, outfile=None, fig_h=9.0, fig_w=13.0, sample_rows=30000, border_color='#1a1a1a', border_lw=0.6)[source]¶

Hohl 2024 ALMI-style 4-panel quad for any TPS category.

Panels (a-d):

Kernel density (incidents per sq km) – sequential
Per-100k rate from NeighbourhoodCrimeRates – sequential
Spatial clusters / outliers (LISA HH/LL/HL/LH) – diverging
Significant Gi* hot/cold spots – diverging

Parameters:

category (str)
year (int)
outfile (str | Path | None)
fig_h (float)
fig_w (float)
sample_rows (int | None)
border_color (str)
border_lw (float)

Return type:

morie.tps_render.render_dbscan(category='Assault', *, eps_km=0.3, min_samples=20, sample_rows=30000, outfile=None, show_top=12, fig_h=7.5, fig_w=12.0)[source]¶

Point-pattern map: incidents + DBSCAN cluster IDs.

No floating hood labels; clusters are shown by colour with a compact legend. Aspect/rotation matches the rest of the suite.

Parameters:

category (str)
eps_km (float)
min_samples (int)
sample_rows (int | None)
outfile (str | Path | None)
show_top (int)
fig_h (float)
fig_w (float)

Return type:

morie.tps_render.render_district_proportional(metric='ASSAULT_2024', *, title=None, outfile=None, fig_h=7.0, fig_w=12.5, border_color='#1a1a1a', border_lw=0.55)[source]¶

District-coloured map with proportional-symbol black circles.

Inspired by the author’s reference (Toronto news-coverage proportional- symbol map). Toronto’s 158 wards are coloured by former-borough district (Etobicoke / North York / Scarborough / York / Old Toronto / East York), with a black circle drawn at each ward centroid sized proportional to metric (any column from NeighbourhoodCrimeRates).

Includes a “Map Divisions” legend with district swatches and a “Number of <metric>” graduated-symbol legend with 5 size tiers.

Parameters:

metric (str)
title (str | None)
outfile (str | Path | None)
fig_h (float)
fig_w (float)
border_color (str)
border_lw (float)

Return type:

morie.tps_render.render_satscan_panel(category='Homicides', *, sample_rows=30000, n_clusters=2, n_mc=199, year_window_yrs=4, outfile=None, fig_h=7.5, fig_w=11.5, border_color='#1a1a1a', border_lw=0.55)[source]¶

Hohl 2024 ALMI “panel d” – Kulldorff-style space-time scan statistic with significant clusters as pink filled circles, red dots inside for sig. locations, yellow polygons for sig. wards, and an info-box (n_cases / expected / RR / p-value) on the right.

Implementation is a simplified Kulldorff Poisson scan over a grid of candidate (x, y, r, t-window) cells: for each cell compute a log-likelihood ratio assuming Poisson(λ_in) inside vs Poisson(λ_out) outside; null distribution by Monte-Carlo (n_mc permutations of incident timestamps within the bbox).

Parameters:

category (str)
sample_rows (int | None)
n_clusters (int)
n_mc (int)
year_window_yrs (int)
outfile (str | Path | None)
fig_h (float)
fig_w (float)
border_color (str)
border_lw (float)

Return type:

morie.tps_render.render_yearly_grid(*, prefix='ASSAULT_RATE', years=range(2014, 2025), cmap='Reds', outfile=None, ncols=4)[source]¶

Small-multiples panel of per-year choropleths, no hood labels.

Parameters:

prefix (str)
years (Iterable[int])
cmap (str)
outfile (str | Path | None)
ncols (int)

Return type:

morie.tps_all_analyze – RichResult-emitting analyses for the 13 TPS crime datasets.

Surfaces:: analyze_<name>(df=None) -> RichResult # universal per-dataset spatial_summary(df) -> RichResult # neighbourhood + lat/lon temporal_summary(df) -> RichResult # year/month/dow/hour crime_compare(*dfs, names=…) -> RichResult # cross-category analyze_all(out_dir=None) -> dict

Output goes to data/manifest/outputs/tps/.

Each TPS CSV has the same standard schema (Toronto Open Data convention), so a single set of helpers handles the full 13-dataset matrix.

morie.tps_all_analyze.temporal_summary(df, *, ds_name='?')[source]¶

Year/month/dow/hour rollups.

Parameters:

df (DataFrame)
ds_name (str)

Return type:

morie.tps_all_analyze.spatial_summary(df, *, ds_name='?')[source]¶

Neighbourhood + division + premises rollups, plus lat/long bounding-box summary.

Parameters:

df (DataFrame)
ds_name (str)

Return type:

morie.tps_all_analyze.offence_summary(df, *, ds_name='?')[source]¶

OFFENCE / UCR / CSI_CATEGORY rollups.

Parameters:

df (DataFrame)
ds_name (str)

Return type:

morie.tps_all_analyze.gini_concentration(values)[source]¶

Gini coefficient of crime concentration across spatial units. G=0 perfectly even, G=1 perfectly concentrated.

Parameters:: values (marr)
Return type:: float

morie.tps_all_analyze.neighbourhood_concentration(df, *, ds_name='?')[source]¶

How concentrated is crime across neighbourhoods? Uses HOOD_158 (the 158-neighbourhood scheme) by default.

Parameters:

df (DataFrame)
ds_name (str)

Return type:

morie.tps_all_analyze.crime_compare(dfs)[source]¶

Compare counts and trends across multiple TPS categories. Returns one table per axis (year/division/neighbourhood).

Parameters:: dfs (dict[str, DataFrame])
Return type:: RichResult

morie.tps_all_analyze.analyze_one(name, df=None)[source]¶

Run the standard TPS analysis bundle on one dataset.

Parameters:

name (str)
df (DataFrame | None)

Return type:

morie.tps_all_analyze.analyze_assault(df=None)¶

Analyze TPS Assault.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_autotheft(df=None)¶

Analyze TPS AutoTheft.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_bicycletheft(df=None)¶

Analyze TPS BicycleTheft.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_breakandenter(df=None)¶

Analyze TPS BreakandEnter.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_communitysafetyindicators(df=None)¶

Analyze TPS CommunitySafetyIndicators.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_hatecrimes(df=None)¶

Analyze TPS HateCrimes.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_homicides(df=None)¶

Analyze TPS Homicides.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_intimatepartnerandfamilyviolence(df=None)¶

Analyze TPS IntimatePartnerAndFamilyViolence.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_neighbourhoodcrimerates(df=None)¶

Analyze TPS NeighbourhoodCrimeRates.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_robbery(df=None)¶

Analyze TPS Robbery.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_shootingandfirearmdiscarges(df=None)¶

Analyze TPS ShootingAndFirearmDiscarges.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_theftfrommovingvehicle(df=None)¶

Analyze TPS TheftFromMovingVehicle.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_theftover(df=None)¶

Analyze TPS TheftOver.

Parameters:: df (DataFrame | None)
Return type:: RichResult

morie.tps_all_analyze.analyze_all(out_dir=None, *, sample_rows=50000)[source]¶

Run full bundle on each of the 13 TPS datasets + cross-compare.

sample_rows caps each load at N rows for fast development; pass None to load everything (Assault alone is 254k rows so the full 13-way load is several hundred MB).

Parameters:

out_dir (Path | None)
sample_rows (int | None)

Return type:

Federal SIU — Sprott / Doob / Iftene replication¶

morie.sprott_doob – Sprott & Doob (CRIMSL UToronto) SIU analyses.

Replicates the analytical contribution of the four CRIMSL UToronto research reports authored by Prof. Jane B. Sprott (Toronto Metropolitan University, formerly Ryerson) and Prof. Anthony N. Doob (University of Toronto), with Prof. Adelina Iftene (Dalhousie) co-authoring the May 2021 paper on Independent External Decision Makers.

The four reports¶

Sprott & Doob (Oct 2020) – Understanding the Operation of Correctional Service Canada’s Structured Intervention Units: Some Preliminary Findings. First systematic outside analysis of CSC’s SIU data; demonstrated that SIUs were not operating as the legislative framework (Bill C-83) required.
Sprott & Doob (Nov 2020) – Is there Clear Evidence that COVID-19 Was the Cause of Problems with the Operation of CSC’s Structured Intervention Units? Tests CSC’s COVID-attribution defense; finds the data did not support it (problems pre-existed COVID).
Sprott & Doob (Feb 2021) – Solitary Confinement, Torture, and Canada’s Structured Intervention Units. The most data- intensive of the four; introduces a Mandela-Rules classifier for SIU stays (solitary confinement = ≤15 days at ≤2 hrs out-of-cell; torture = ≥16 days under same conditions). Contains Tables 13, 19, 23 reproduced below.
Sprott, Doob & Iftene (May 2021) – Do Independent External Decision Makers Ensure that Solitary Confinement is no Longer Used in Canada’s Federal Penitentiaries? Evaluates the IEDM review mechanism added by Bill C-83.

Tables hardcoded (from Feb 2021 report)¶

TABLE13_REGIONAL_RATES_PER_1000
SIU person-stays per 1000 regional penitentiary prisoners, split into short (≤15 days) and long (≥16 days) stays, by CSC region.

TABLE19_MANDELA_CLASSIFICATION
Sprott-Doob Mandela-Rules classification of N=1960 SIU stays: solitary confinement (28.4%, N=556), torture (9.9%, N=195), all-other (61.7%, N=1209).

TABLE23_REGIONAL_TORTURE_RATES
Solitary and torture rates per 1000 prisoners by region. Headline finding: Pacific torture rate (39.1) is 22.6× higher than Ontario’s (1.73).

Functions exposed¶

analyze_table13_regional_rates() -> RichResult analyze_table19_mandela_classification() -> RichResult analyze_table23_regional_torture_rates() -> RichResult classify_mandela(days_in_siu, hrs_out_of_cell_avg,

missed_full_4hrs) -> dict

Apply the Sprott-Doob Mandela-Rules classifier to a single SIU person-stay and return the category.

analyze_full_sprott_doob_feb2021() -> RichResult
Comprehensive replication of the Feb 2021 paper’s main tables.

Citation¶

Sprott, J. B., & Doob, A. N. (2021, February). Solitary Confinement, Torture, and Canada’s Structured Intervention Units. Centre for Criminology & Sociolegal Studies, U. of Toronto. URL: https://www.crimsl.utoronto.ca/sites/www.crimsl.utoronto.ca/ files/TortureSolitarySIUsSprottDoob23Feb2021_0.pdf

Note on author order¶

Sprott is FIRST author on the public CRIMSL papers; Doob is co- author. This is documented and indexed in morie.siuiap. CRIMSL_REPORTS. The October 2020 report’s Drive filename (“DoobSprott”) had Doob first but the published title page may differ – when in doubt, follow the published front-matter order.

morie.sprott_doob.classify_mandela(days_in_siu, hrs_out_of_cell_avg, missed_full_4hrs_pct_of_days)[source]¶

Apply the Sprott-Doob Mandela-Rules classifier.

Parameters:

days_in_siu (int) – Length of the SIU person-stay in days.
hrs_out_of_cell_avg (float) – Average hours out of cell per day during the stay.
missed_full_4hrs_pct_of_days (float) – Percent of days during the stay where the inmate did not receive the legislatively-required 4 hours out of cell. (Range: 0-100.)

Returns:

dict with keys ‘category’ (str), ‘rule’ (str), ‘reason’ (str).
Categories
———-
“Solitary Confinement” – Mandela Rule 44 (≤2 hrs out of cell, all) – days missed full 4 hrs, stay length ≤ 15 days.
”Torture” – Mandela Rules 43+44 (same conditions but stay length) – ≥ 16 days.
”All other” – none of the above thresholds met.
>>> r = classify_mandela(20, 1.5, 100)
>>> r[“category”]
’Torture’
>>> r = classify_mandela(8, 1.5, 100)
>>> r[“category”]
’Solitary Confinement’
>>> r = classify_mandela(20, 5, 50)
>>> r[“category”]
’All other’

Return type:

morie.sprott_doob.analyze_table13_regional_rates()[source]¶

Sprott-Doob Table 13: SIU person-stay rates per 1000 prisoners.

Return type:: RichResult

morie.sprott_doob.analyze_table19_mandela_classification()[source]¶

Sprott-Doob Table 19: Mandela-Rules classification of N=1960 SIU stays.

Return type:: RichResult

morie.sprott_doob.analyze_table23_regional_torture_rates()[source]¶

Sprott-Doob Table 23: regional torture/solitary rates per 1000.

Return type:: RichResult

morie.sprott_doob.analyze_iedm_table1_population()[source]¶

Sprott, Doob & Iftene (May 2021) Table 1: IEDM-reviewed population.

Return type:: RichResult

morie.sprott_doob.analyze_iedm_review_outcomes()[source]¶

Sprott, Doob & Iftene May 2021: IEDM review outcomes & disparities.

Return type:: RichResult

morie.sprott_doob.analyze_table4_length_of_stay()[source]¶

Sprott-Doob (Feb 2021) Table 4: SIU length-of-stay distribution.

Return type:: RichResult

morie.sprott_doob.analyze_table11_region_x_stay_length()[source]¶

Sprott-Doob (Feb 2021) Table 11: Region × stay length.

Return type:: RichResult

morie.sprott_doob.analyze_table12_regional_overrepresentation()[source]¶

Sprott-Doob (Feb 2021) Table 12: Regional over-/under-rep in SIU.

Return type:: RichResult

morie.sprott_doob.analyze_table15_region_x_mental_health()[source]¶

Sprott-Doob (Feb 2021) Table 15: Region × MH-flag.

Return type:: RichResult

morie.sprott_doob.analyze_table22_region_x_mandela()[source]¶

Sprott-Doob (Feb 2021) Table 22: Region × Mandela groups.

Return type:: RichResult

morie.sprott_doob.analyze_table9_iedm_decisions()[source]¶

Sprott-Doob-Iftene (May 2021) Table 9: IEDM review outcomes.

Return type:: RichResult

morie.sprott_doob.analyze_table10_per_iedm_variance()[source]¶

Sprott-Doob-Iftene (May 2021) Table 10: per-IEDM variance.

Return type:: RichResult

morie.sprott_doob.analyze_table15_long_stay_no_iedm()[source]¶

Sprott-Doob-Iftene (May 2021) Table 15: long-stay no-IEDM cases.

Return type:: RichResult

morie.sprott_doob.verify_chi2(observed)[source]¶

Recompute Pearson χ² from a 2D contingency table.

Pure-function rebuild of scipy.stats.chi2_contingency without Yates correction. Intended for quick self-checks of the transcribed cell counts against published χ² values.

Parameters:

observed (2D list of int) – Contingency table (rows × cols).

Returns:

dict with keys ‘chi2’, ‘df’, ‘p_value’, ‘expected’, ‘n’.
>>> # 2×2 sanity (independence)
>>> verify_chi2([[10, 10], [10, 10]])[“chi2”]
0.0

Return type:

morie.sprott_doob.verify_published_chi_squares()[source]¶

Re-derive every published χ² from its transcribed cell counts.

Cross-checks Sprott-Doob Tables 11, 15, 22 (Feb 2021) and Sprott-Doob-Iftene Tables 5, 7, 8, 10 (May 2021). For each: rebuilds the observed contingency table from the per-row dicts, runs verify_chi2(), and reports recomputed vs. published values.

Return type:: RichResult

morie.sprott_doob.analyze_full_sprott_doob_feb2021()[source]¶

Comprehensive replication of Sprott & Doob (Feb 2021) main tables.

Return type:: RichResult

morie.siuiap – Structured Intervention Unit Implementation Advisory Panel (federal).

NOTE: this is the FEDERAL Structured Intervention Unit Implementation Advisory Panel module. It is distinct from morie.siu, which is the ONTARIO Special Investigations Unit (police-oversight) automation package. Both share the acronym SIU but refer to entirely different agencies – the rename siu_iap -> siuiap was made to avoid the underscore prefix collision with morie.siu/.

The federal counterpart to Ontario’s OTIS provincial restrictive- confinement data is Canada’s federal Structured Intervention Unit (SIU) system, which replaced administrative segregation in federal penitentiaries in 2019. From April 2021 to December 2024, the Structured Intervention Unit Implementation Advisory Panel (SIU IAP) monitored federal SIU implementation, chaired by Howard Sapers with members including Prof. Emeritus Anthony N. Doob and Prof. Jane B. Sprott.

In addition to SIU IAP panel reports, this module also indexes:

CRIMSL UToronto research reports (Sprott / Doob / Iftene 2020-2021): four independent academic reports on CSC’s SIU operation, the role of independent external decision makers, the COVID-19 attribution claim, and solitary-confinement-as-torture framing.

Doob 2020 Federal Court affidavit (T-539-20): expert evidence on CCRSO 2018 prisoner-flow + age + release statistics, used as the source for the analyses replicated in morie.doob_trends.

This module exposes:

PANEL_MEMBERS – list of SIU IAP members REPORTS – dict of SIU IAP panel reports (5) CRIMSL_REPORTS – dict of UToronto CRIMSL research reports (4) AFFIDAVITS – dict of Federal Court / inquest affidavits PANEL_MANDATE – text describing the panel’s mandate cite() -> str – generate a citation string

It does NOT ingest data: SIU IAP outputs and CRIMSL papers are qualitative / mixed-method PDFs. The federal SIU dataset itself (CSC Research Branch) is not available as open data; this module exists to make the federal context citable from morie analyses.

References

Public Safety Canada. Structured Intervention Unit Implementation: Advisory Panel. https://www.publicsafety.gc.ca/cnt/cntrng-crm/crrctns/siuiap-ccuis-en.aspx
Sapers, H. et al. (2024). Final Report – Structured Intervention Unit: Implementation Advisory Panel.
Centre for Criminology & Sociolegal Studies, U. of Toronto.: Reports on Canada’s Structured Intervention Units. https://www.crimsl.utoronto.ca/news/reports-canada%27s-structured-intervention-units

morie.siuiap.cite(report_id='final_2024')[source]¶

Build a citation string for an SIU IAP / CRIMSL / affidavit entry.

Searches REPORTS, CRIMSL_REPORTS, and AFFIDAVITS in order.

>>> cite("final_2024")
'SIU IAP (2024). SIU IAP Final Report. Public Safety Canada.'
>>> "Sprott" in cite("sprott_doob_torture_solitary_2021")
True

Parameters:: report_id (str)
Return type:: str

morie.siuiap.panel_summary()[source]¶

Human-readable summary of the SIU IAP panel.

Return type:: str

morie.doob_trends – National-aggregate analyses from Doob’s Federal Court affidavit.

Replicates the analytical contribution of Prof. Anthony N. Doob’s expert-witness affidavit in Canadian Civil Liberties Association et al. v. The Attorney General of Canada (Federal Court file T-539-20, Application Record Vol. 3 of 5, pp. 778-795).

Doob’s national-aggregate analyses (Figures 1-4 + Tables 1-3) sit ALONGSIDE the per-row MRM modules on OTIS provincial data and the MRM chi-square family on aggregate contingency tables. Where the MRM modules test causal contrasts at the individual level on Ontario provincial data, Doob’s affidavit work tests decoupling of imprisonment rates from crime rates at the Canadian and US national-aggregate level over 50+ years.

Constants exposed¶

CCRSO_TABLE1_RELEASES Table 1: 5-year average annual releases
by type (day / full parole / statutory) with violent / non-violent / breach revocations and successful completions.

CCRSO_TABLE2_FLOW Table 2: 2013/14-2017/18 prisoner flow:
count / admissions / deaths / full parole releases / statutory releases.

CCRSO_TABLE3_AGE Table 3: 2018 age distribution – adult
population, CSC in-custody count, CSC admissions – by age group (18-49, 50-59, 60+).

Functions exposed¶

analyze_doob_table1_releases() -> RichResult
Renders Table 1 + computes overall success / revocation rates.

analyze_doob_table2_flow() -> RichResult
Renders Table 2 + year-over-year changes + 5-year averages.

analyze_doob_table3_age_overrepresentation() -> RichResult
Renders Table 3 + computes age-group IRR for CSC custody vs adult population (over- / under-representation).

decoupling_test(crime_series, imprisonment_series) -> RichResult
Tests Doob’s thesis that imprisonment is decoupled from crime rate. Reports Pearson correlation between the two time series with optional Pettitt change-point detection.

pettitt_changepoint(series) -> dict
Pettitt non-parametric change-point detection for time series.

Data sources (cited in Doob affidavit)¶

CCRSO 2018 – Public Safety Canada, Corrections and Conditional
Release Statistical Overview 2018 (released 2019). https://www.publicsafety.gc.ca/cnt/rsrcs/pblctns/ccrso-2018/

StatsCan CANSIM/Table 35-10-0026-01, 35-10-0177-01, 35-10-0014-01. Pastore, A. L. & Maguire, K. (2004). Sourcebook of Criminal Justice

Statistics.

Citation¶

Doob, A. N. (2020). Affidavit (T-539-20) of Anthony Doob – Federal Court of Canada, Application Record Vol. 3 of 5. CCLA et al. v. Attorney General of Canada.

morie.doob_trends.analyze_doob_table1_releases()[source]¶

Doob Affidavit Table 1: 5-year average annual conditional releases.

Return type:: RichResult

morie.doob_trends.analyze_doob_table2_flow()[source]¶

Doob Affidavit Table 2: prisoner flow 2013/14-2017/18.

Return type:: RichResult

morie.doob_trends.analyze_doob_table3_age_overrepresentation()[source]¶

Doob Affidavit Table 3: age distribution – Canada vs CSC custody.

Return type:: RichResult

morie.doob_trends.pettitt_changepoint(series)[source]¶

Pettitt’s non-parametric change-point test (Pettitt 1979).

Returns the location of the most likely single change-point in a time series, plus the test statistic and approximate p-value.

>>> r = pettitt_changepoint([1, 1, 1, 1, 5, 5, 5, 5])
>>> r["change_point_index"]
3

Parameters:: series (Iterable[float])
Return type:: dict

morie.doob_trends.decoupling_test(crime_series, imprisonment_series, years=None)[source]¶

Doob’s thesis test: imprisonment rate decoupled from crime rate.

Computes Pearson correlation between the two equal-length time series. A correlation near 0 (or with a sign mismatch) supports Doob’s claim that imprisonment is decoupled from crime trends. Optionally runs Pettitt change-point on each series.

Parameters:

crime_series (Iterable[float])
imprisonment_series (Iterable[float])
years (Iterable[int] | None)

Return type:

morie.doob_trends.analyze_doob_full_affidavit()[source]¶

Comprehensive replication of Doob’s Federal Court affidavit aggregate analyses (Tables 1-3 + decoupling test stub).

Renders Tables 1, 2, 3 + a placeholder for Figures 1-4 (which require StatsCan CANSIM time-series data not bundled with morie).

Return type:: RichResult

Psychometrics¶

morie.psymet – Psychometric analysis (CTT, reliability, factor analysis).

Short-name API (≤6 chars, no snake_case):

crba() – Cronbach’s coefficient alpha
mcdo() – McDonald’s omega total + hierarchical
itcor() – Corrected item-total correlations
adel() – Alpha if item deleted
crel() – Composite reliability from factor loadings
ave() – Average Variance Extracted
kmo() – Kaiser-Meyer-Olkin sampling adequacy
bart() – Bartlett’s test of sphericity
paran() – Horn’s parallel analysis
splhf() – Spearman-Brown split-half reliability
idisc() – Item discrimination index

References

Cronbach (1951). Psychometrika, 16(3), 297-334.
McDonald (1999). Test Theory: A Unified Treatment.
Revelle (2024). psych R package.

class morie.psymet.RlbRes(raw, std, avgr, k, n, ci_lo=0.0, ci_hi=0.0)[source]¶

Bases: object

Reliability result (Cronbach’s alpha).

Parameters:

raw (float)
std (float)
avgr (float)
k (int)
n (int)
ci_lo (float)
ci_hi (float)

raw: float¶

std: float¶

avgr: float¶

k: int¶

n: int¶

ci_lo: float = 0.0¶

ci_hi: float = 0.0¶

summary()[source]¶

Return type:: DataFrame

class morie.psymet.OmgRes(total, hier, alpha, nf, expvar=0.0)[source]¶

Bases: object

Omega result (McDonald’s omega).

Parameters:

total (float)
hier (float)
alpha (float)
nf (int)
expvar (float)

total: float¶

hier: float¶

alpha: float¶

nf: int¶

expvar: float = 0.0¶

class morie.psymet.KmoRes(msa, items=<factory>)[source]¶

Bases: object

KMO test result.

Parameters:

msa (float)
items (dict[str, float])

msa: float¶

items: dict[str, float]¶

class morie.psymet.BrtRes(chisq, df, pval)[source]¶

Bases: object

Bartlett’s sphericity result.

Parameters:

chisq (float)
df (int)
pval (float)

chisq: float¶

df: int¶

pval: float¶

morie.psymet.crba(data, *, ci=0.95)[source]¶

Cronbach’s coefficient alpha with Feldt CI.

Parameters:

data (DataFrame or ndarray) – Items as columns, respondents as rows.
ci (float) – Confidence level (default 0.95).

Return type:

RlbRes

Examples

>>> from morie.fn import _array_core as np
>>> items = np.array([[4, 5, 3, 4], [3, 4, 3, 3], [5, 5, 4, 5], [2, 3, 2, 2], [4, 4, 4, 5], [3, 3, 2, 3]], dtype=float)
>>> r = crba(items)
>>> (round(r.raw, 4), r.k, r.n, round(r.ci_lo, 4) < r.raw < round(r.ci_hi, 4))
(0.9519, 4, 6, True)

morie.psymet.mcdo(data, nf=1)[source]¶

McDonald’s omega total and hierarchical.

Loadings come from principal-axis factoring (communalities iterated on the reduced correlation matrix). hier is the general-factor omega of the Schmid-Leiman transformation: the nf factors are promax-rotated, their correlations are factored for one general factor (with two factors both load the square root of their correlation), and the items are projected onto it. With nf = 1 the single factor is the general factor, so hier equals total. Same arithmetic as rmorie::morie_psymet_omega.

Parameters:

data (DataFrame or ndarray)
nf (int) – Number of factors.

Return type:

OmgRes

morie.psymet.itcor(data)[source]¶

Corrected item-total correlations.

Returns DataFrame: item, r_total, r_corr

Parameters:: data (DataFrame | marr)
Return type:: DataFrame

morie.psymet.adel(data)[source]¶

Alpha if each item deleted.

Returns DataFrame: item, adel

Parameters:: data (DataFrame | marr)
Return type:: DataFrame

morie.psymet.crel(loads)[source]¶

Composite reliability from standardized factor loadings.

CR = (Σλ)² / ((Σλ)² + Σ(1−λ²))

Parameters:: loads (marr)
Return type:: float

morie.psymet.ave(loads)[source]¶

Average Variance Extracted from factor loadings.

AVE = mean(λ²). Values ≥ 0.5 indicate convergent validity.

Parameters:: loads (marr)
Return type:: float

morie.psymet.kmo(data)[source]¶

Kaiser-Meyer-Olkin sampling adequacy.

MSA > 0.6 adequate, > 0.8 meritorious (Kaiser 1974).

Parameters:: data (DataFrame | marr)
Return type:: KmoRes

morie.psymet.bart(data)[source]¶

Bartlett’s test of sphericity.

H0: correlation matrix = identity. Reject (p<0.05) -> factorable.

Parameters:: data (DataFrame | marr)
Return type:: BrtRes

morie.psymet.paran(data, nsim=100, seed=42)[source]¶

Horn’s parallel analysis – suggested number of factors.

Compares observed eigenvalues to 95th percentile of random data.

Parameters:

data (DataFrame | marr)
nsim (int)
seed (int)

Return type:

morie.psymet.splhf(data, method='first_last')[source]¶

Spearman-Brown split-half reliability.

method: ‘first_last’ or ‘odd_even’.

Parameters:

data (DataFrame | marr)
method (str)

Return type:

morie.psymet.idisc(data, pct=0.27)[source]¶

Item discrimination index (D-statistic).

Upper/lower groups by total score (default 27%, Kelley). Returns DataFrame: item, d

Parameters:

data (DataFrame | marr)
pct (float)

Return type:

DataFrame

morie.psymet.cronbach_alpha(data, *, ci=0.95)¶

Cronbach’s coefficient alpha with Feldt CI.

Parameters:

data (DataFrame or ndarray) – Items as columns, respondents as rows.
ci (float) – Confidence level (default 0.95).

Return type:

RlbRes

Examples

>>> from morie.fn import _array_core as np
>>> items = np.array([[4, 5, 3, 4], [3, 4, 3, 3], [5, 5, 4, 5], [2, 3, 2, 2], [4, 4, 4, 5], [3, 3, 2, 3]], dtype=float)
>>> r = crba(items)
>>> (round(r.raw, 4), r.k, r.n, round(r.ci_lo, 4) < r.raw < round(r.ci_hi, 4))
(0.9519, 4, 6, True)

morie.psymet.mcdonalds_omega(data, nf=1)¶

McDonald’s omega total and hierarchical.

Loadings come from principal-axis factoring (communalities iterated on the reduced correlation matrix). hier is the general-factor omega of the Schmid-Leiman transformation: the nf factors are promax-rotated, their correlations are factored for one general factor (with two factors both load the square root of their correlation), and the items are projected onto it. With nf = 1 the single factor is the general factor, so hier equals total. Same arithmetic as rmorie::morie_psymet_omega.

Parameters:

data (DataFrame or ndarray)
nf (int) – Number of factors.

Return type:

OmgRes

morie.psymet.item_total_correlation(data)¶

Corrected item-total correlations.

Returns DataFrame: item, r_total, r_corr

Parameters:: data (DataFrame | marr)
Return type:: DataFrame

morie.psymet.alpha_if_deleted(data)¶

Alpha if each item deleted.

Returns DataFrame: item, adel

Parameters:: data (DataFrame | marr)
Return type:: DataFrame

morie.psymet.composite_reliability(loads)¶

Composite reliability from standardized factor loadings.

CR = (Σλ)² / ((Σλ)² + Σ(1−λ²))

Parameters:: loads (marr)
Return type:: float

morie.psymet.average_variance_extracted(loads)¶

Average Variance Extracted from factor loadings.

AVE = mean(λ²). Values ≥ 0.5 indicate convergent validity.

Parameters:: loads (marr)
Return type:: float

morie.psymet.kmo_test(data)¶

Kaiser-Meyer-Olkin sampling adequacy.

MSA > 0.6 adequate, > 0.8 meritorious (Kaiser 1974).

Parameters:: data (DataFrame | marr)
Return type:: KmoRes

morie.psymet.bartlett_sphericity(data)¶

Bartlett’s test of sphericity.

H0: correlation matrix = identity. Reject (p<0.05) -> factorable.

Parameters:: data (DataFrame | marr)
Return type:: BrtRes

morie.psymet.parallel_analysis(data, nsim=100, seed=42)¶

Horn’s parallel analysis – suggested number of factors.

Compares observed eigenvalues to 95th percentile of random data.

Parameters:

data (DataFrame | marr)
nsim (int)
seed (int)

Return type:

morie.psymet.split_half_reliability(data, method='first_last')¶

Spearman-Brown split-half reliability.

method: ‘first_last’ or ‘odd_even’.

Parameters:

data (DataFrame | marr)
method (str)

Return type:

morie.psymet.item_discrimination(data, pct=0.27)¶

Item discrimination index (D-statistic).

Upper/lower groups by total score (default 27%, Kelley). Returns DataFrame: item, d

Parameters:

data (DataFrame | marr)
pct (float)

Return type:

DataFrame

morie.psymet.ReliabilityResult¶: alias of RlbRes

morie.psymet.OmegaResult¶: alias of OmgRes

morie.psymet.KMOResult¶: alias of KmoRes

morie.psymet.BartlettResult¶: alias of BrtRes

Entheogenic neuroimaging — DMT EEG-fMRI¶

Opt-in module wrapping the Carhart-Harris / Timmermann DMT-imaging dataset (20 subjects EEG + parcellated fMRI; the 15 motion-survived subjects are 01-03 and 06-17). Exposes two consciousness-theory metrics: Beautiful Loop (Bayne, Carter, Laukkonen, Slagter) and Self-Aware Networks (Pirez). Data location is honoured via $MORIE_DMT_IMAGING_ROOT; a deterministic synthetic fixture is returned when the local mirror is absent so CI keeps running.

morie.entheo – Psychedelic EEG-fMRI preprocessing and consciousness analysis.

Opt-in module; not auto-loaded by import morie. Import explicitly with import morie.entheo or from morie import entheo.

Wraps the Carhart-Harris / Timmermann DMT-imaging data (20 subjects EEG + parcellated fMRI; the 15 motion-survived subjects are 01-03 and 06-17, with 04, 05, 20, 21, 24 absent on disk per the manuscript) into a coherent morie surface, and exposes two consciousness-theory metrics:

Beautiful Loop (Bayne, Carter, Laukkonen, Slagter): predictive integration of phenomenal binding, scored as the cross-modal coupling between EEG-band power and fMRI gradient dispersion.

Self-Aware Networks (SAN) (Pirez): meta-cognitive recurrence, scored as the spectral slope of the autocorrelation of the EEG-fMRI joint state-vector.

Data location: the dataset is read from a local mirror at $MORIE_DMT_IMAGING_ROOT (default /Volumes/VSR/rootcoderfiles/DMT_Imaging). When the mirror is absent or scipy.io / pymatreader is unavailable, a deterministic synthetic fixture is returned so downstream pipelines keep running in CI.

Public API¶

load_dmt_imaging(subject_id=None)
BIDS-style record (EEG + fMRI + behavioural).
preprocess_eeg(record, bandpass=(1, 40), notch=60, asr_threshold=20)
Butterworth bandpass + notch + ASR-style artifact rejection.
preprocess_fmri(record, motion_threshold_mm=0.5)
Motion scrubbing + ICA-AROMA-style noise removal.
beautiful_loop_metric(eeg, fmri)
Bayne-Laukkonen integrated phenomenal-binding score.
san_score(eeg, fmri)
Self-Aware Network recurrence score.

R parity: every Python entrypoint above has an R sibling in the morie R package (entheo_data.R, entheo_preprocess.R, entheo_analysis.R).

References

Timmermann, C. et al. (2023) Neural correlates of the DMT experience: assessed with multivariate EEG. Scientific Reports, 13, 3147.
Bayne, T. & Carter, O. (2018) Dimensions of consciousness and the: psychedelic state. Neuroscience of Consciousness, niy008.
Laukkonen, R. E. & Slagter, H. A. (2021) From many to (n)one:: meditation and the plasticity of the predictive mind. Neuroscience & Biobehavioral Reviews, 128, 199-217.

morie.entheo.load_dmt_imaging(subject_id=None)[source]¶

Load one (or all) Timmermann DMT-imaging subject records.

Parameters:: subject_id (str or int, optional) – Two-digit subject ID, e.g. "01" or 1. If None, every subject available on disk is loaded (or a small synthetic set).
Returns:: .records : list of dicts with keys subject_id, eeg, fmri, behavioural. .root : str path to the DMT_Imaging mirror (or None). .synthetic: bool – True if any record is the fallback.
Return type:: RichResult

morie.entheo.preprocess_eeg(record, bandpass=(1.0, 40.0), notch=60.0, asr_threshold=20.0)[source]¶

Butterworth bandpass + notch + ASR-style trimming on an EEG record.

Parameters:

record (dict) – Subject record from load_dmt_imaging(); record["eeg"] must contain sfreq and at least one of data_dmt / data_pcb as channels x timepoints arrays.
bandpass ((low_hz, high_hz)) – Butterworth passband; defaults to 1-40 Hz.
notch (float) – Line-noise frequency to notch (Hz). 60 (NA) or 50 (EU).
asr_threshold (float) – Z-score above which a sample is reconstructed (toy ASR).

Return type:

morie.entheo.preprocess_fmri(record, motion_threshold_mm=0.5, n_noise_components=5)[source]¶

Motion scrubbing + ICA-AROMA-style noise removal (toy SVD).

Parameters:

record (dict) – Subject record from load_dmt_imaging(); record["fmri"] must have data_dmt / data_pcb (parcels x timepoints) and optionally motion_fd_mm.
motion_threshold_mm (float) – Per-volume framewise-displacement threshold (mm). Volumes exceeding this are zeroed (scrubbed). Default 0.5 mm.
n_noise_components (int) – Top-k singular components projected out as a toy ICA-AROMA stand-in (real AROMA uses ICA + spatial classifier).

Return type:

morie.entheo.beautiful_loop_metric(eeg, fmri=None)[source]¶

Bayne-Laukkonen integrated phenomenal-binding metric.

Scores the cross-modal coupling between EEG power envelope and fMRI gradient dispersion. In a record context, also reports the DMT-vs-PCB contrast.

Parameters:

eeg (array-like or dict-record) – Either an EEG channels x timepoints array, or a subject record from load_dmt_imaging().
fmri (array-like or None) – Required if eeg is an array (parcels x timepoints).

Returns:

.score : scalar phenomenal-binding score (DMT or single). .score_pcb : scalar PCB score (record context only). .contrast : DMT − PCB (record context only). .per_frame_dmt : numpy vector of per-frame scores. .per_frame_pcb : numpy vector or None.

Return type:

morie.entheo.san_score(eeg, fmri=None)[source]¶

Self-Aware Networks (Pirez) recurrence score.

Scores the lag-1 autocorrelation of the joint EEG-fMRI state vector in a sliding window – higher = stronger meta-cognitive recurrence. Reports DMT-vs-PCB contrast in record context.

Return type:: RichResult

morie.entheo.data – Data fetchers for the DMT-imaging dataset.

Primary path: local mirror at /Volumes/VSR/rootcoderfiles/DMT_Imaging (Carhart-Harris / Timmermann; 20 subjects EEG-fMRI; 15 motion-survived).

Fallback path: OpenNeuro mirror (network-guarded; in v0.4.0-alpha we only emit a guidance string and return a synthetic fixture so the public API stays exercisable in CI without scipy.io / pymatreader).

The on-disk layout we recognise:

DMT_Imaging/

EEG/ RegressorsInterpscrubbedIRASA_{Central,Frontal,Occipital,
Parietal,Temporal}.mat

fMRI/ LongS{NN}{DMT,PCB}.mat (NN = 01..25) Scripts/ *.m (BetweenNetworkRSFC, dRSFC, LZ_analysis, …) README.md

morie.entheo.data.load_dmt_imaging(subject_id=None)[source]¶

Load one (or all) Timmermann DMT-imaging subject records.

Parameters:: subject_id (str or int, optional) – Two-digit subject ID, e.g. "01" or 1. If None, every subject available on disk is loaded (or a small synthetic set).
Returns:: .records : list of dicts with keys subject_id, eeg, fmri, behavioural. .root : str path to the DMT_Imaging mirror (or None). .synthetic: bool – True if any record is the fallback.
Return type:: RichResult

morie.entheo.data.dmt_imaging_root()[source]¶

Resolve the DMT_Imaging root, honouring $MORIE_DMT_IMAGING_ROOT.

Return type:: Path | None

morie.entheo.data.list_subjects(root=None)[source]¶

Enumerate available subjects under <root>/fMRI/.

Parameters:: root (str | Path | None)
Return type:: list[str]

morie.entheo.preprocess – EEG-fMRI preprocessing.

Pure-NumPy fallbacks so the public API works in any CI environment; when scipy.signal / MNE-Python / nilearn are importable we delegate to them. Two callables:

preprocess_eeg : Butterworth bandpass + notch + ASR-style trimming.

preprocess_fmri : motion scrubbing (FD threshold) + ICA-AROMA-style noise-component projection (toy SVD-based stand-in in v0.4.0-alpha).

Both consume the dict-record shape emitted by load_dmt_imaging and return a RichResult whose .record holds the cleaned record.

morie.entheo.preprocess.preprocess_eeg(record, bandpass=(1.0, 40.0), notch=60.0, asr_threshold=20.0)[source]¶

Butterworth bandpass + notch + ASR-style trimming on an EEG record.

Parameters:

record (dict) – Subject record from load_dmt_imaging(); record["eeg"] must contain sfreq and at least one of data_dmt / data_pcb as channels x timepoints arrays.
bandpass ((low_hz, high_hz)) – Butterworth passband; defaults to 1-40 Hz.
notch (float) – Line-noise frequency to notch (Hz). 60 (NA) or 50 (EU).
asr_threshold (float) – Z-score above which a sample is reconstructed (toy ASR).

Return type:

morie.entheo.preprocess.preprocess_fmri(record, motion_threshold_mm=0.5, n_noise_components=5)[source]¶

Motion scrubbing + ICA-AROMA-style noise removal (toy SVD).

Parameters:

record (dict) – Subject record from load_dmt_imaging(); record["fmri"] must have data_dmt / data_pcb (parcels x timepoints) and optionally motion_fd_mm.
motion_threshold_mm (float) – Per-volume framewise-displacement threshold (mm). Volumes exceeding this are zeroed (scrubbed). Default 0.5 mm.
n_noise_components (int) – Top-k singular components projected out as a toy ICA-AROMA stand-in (real AROMA uses ICA + spatial classifier).

Return type:

morie.entheo.analysis – Consciousness-theory analysers.

Two public callables:

beautiful_loop_metric(eeg, fmri)
Bayne-Laukkonen integrated phenomenal-binding score. Aggregates _theory.binding_per_frame into a global scalar plus DMT-vs-PCB contrast when both conditions are present.

san_score(eeg, fmri)
Self-Aware Network recurrence score (Pirez, Cohen et al.). Aggregates _theory.san_recurrence_per_frame.

Both accept either:

two numpy arrays (EEG and fMRI timecourses), OR
a subject record dict (as returned by load_dmt_imaging); in case (b) the contrast across DMT/PCB is also reported.

v0.4.0-alpha implementations are deliberately simple stand-ins for the rc1 psychometrically-calibrated versions; the API and the RichResult shape are stable.

morie.entheo.analysis.beautiful_loop_metric(eeg, fmri=None)[source]¶

Bayne-Laukkonen integrated phenomenal-binding metric.

Scores the cross-modal coupling between EEG power envelope and fMRI gradient dispersion. In a record context, also reports the DMT-vs-PCB contrast.

Parameters:

eeg (array-like or dict-record) – Either an EEG channels x timepoints array, or a subject record from load_dmt_imaging().
fmri (array-like or None) – Required if eeg is an array (parcels x timepoints).

Returns:

.score : scalar phenomenal-binding score (DMT or single). .score_pcb : scalar PCB score (record context only). .contrast : DMT − PCB (record context only). .per_frame_dmt : numpy vector of per-frame scores. .per_frame_pcb : numpy vector or None.

Return type:

morie.entheo.analysis.san_score(eeg, fmri=None)[source]¶

Self-Aware Networks (Pirez) recurrence score.

Scores the lag-1 autocorrelation of the joint EEG-fMRI state vector in a sliding window – higher = stronger meta-cognitive recurrence. Reports DMT-vs-PCB contrast in record context.

Return type:: RichResult

The MRM framework¶

Multilevel Reconciliation Methodology entry points and their primitives.

The MRM research framework: load, reconcile, report.

Parity with rmorie’s R/mrm_flagship.R. Multilevel Reconciliation Methodology – load a special-investigations dataset with provenance, reconcile it against a second source under an explicit matching schema, and render a publication-ready table with morie’s own renderer.

Reconciliation is the part that earns its keep. Two sources describing the same events rarely agree, and the disagreement is the finding: a match rate, the orphans on each side, and the field-level conflicts among the records that did match. Silently inner-joining and reporting the survivors hides all three.

class morie.mrm_flagship.MrmDataset(data: 'object', provenance: 'dict')[source]¶

Bases: object

Parameters:

data (object)
provenance (dict)

data: object¶

provenance: dict¶

class morie.mrm_flagship.MrmEffect(results: 'list[dict]', consensus: 'dict', correction: 'str', spec: 'dict', failed: 'dict' = <factory>)[source]¶

Bases: object

Parameters:

results (list[dict])
consensus (dict)
correction (str)
spec (dict)
failed (dict)

results: list[dict]¶

consensus: dict¶

correction: str¶

spec: dict¶

failed: dict¶

results_frame()[source]¶

class morie.mrm_flagship.MrmReconciliation(matched: 'list[dict]', unmatched_primary: 'list[dict]', unmatched_secondary: 'list[dict]', conflicts: 'list[dict]' = <factory>, match_rate: 'float | None' = None, schema: 'dict' = <factory>)[source]¶

Bases: object

Parameters:

matched (list[dict])
unmatched_primary (list[dict])
unmatched_secondary (list[dict])
conflicts (list[dict])
match_rate (float | None)
schema (dict)

matched: list[dict]¶

unmatched_primary: list[dict]¶

unmatched_secondary: list[dict]¶

conflicts: list[dict]¶

match_rate: float | None = None¶

schema: dict¶

conflicts_frame()[source]¶

morie.mrm_flagship.mrm_load_si_dataset(name='otis_b01')[source]¶

Load a bundled sample inside a provenance envelope.

The checksum is of the data as CSV, so an analysis records exactly which snapshot it consumed rather than merely which name it asked for.

Parameters:: name (str)
Return type:: MrmDataset

morie.mrm_flagship.mrm_reconcile(primary, secondary, keys, compare=None, numeric_tolerance=0.0)[source]¶

Join two sources on keys and report how far they disagree.

numeric_tolerance is the absolute slack under which a numeric disagreement in a compare column is not counted as a conflict – two sources recording the same date a day apart, for instance.

Parameters:: numeric_tolerance (float)
Return type:: MrmReconciliation

morie.mrm_flagship.mrm_report(effect=None, reconciliation=None, dataset=None, digits=3, stars=True)[source]¶

Render the MRM result as a plain-text table.

morie’s own renderer: no external table package, so the output cannot drift with someone else’s defaults. effect is any object exposing results as rows of mappings with method, estimate, std_error, ci_lower, ci_upper and p_adjusted.

Parameters:

digits (int)
stars (bool)

Return type:

morie.mrm_flagship.mrm_estimate_causal_effect(data, treatment, outcome, covariates, methods=('matching', 'ate', 'aipw', 'dml'), correction='holm', seed=42)[source]¶

Estimate one effect several ways, then correct across the answers.

Composes morie’s own estimators – nearest-neighbour matching, IPW, AIPW and partially-linear DML – on a single specification, corrects the p-values for having asked more than once, and pools by inverse variance. The pooled standard error is the weighted mean of the standard errors: the estimators share one data set, so they are not independent studies and the meta-analytic 1 / sqrt(sum w) would be far too small.

Running four estimators and quoting the friendliest is the failure mode this guards against, which is why every answer is returned together with the correction applied across them. An estimator that cannot run is recorded in failed rather than dropped silently: a consensus over an unknown subset is not a consensus.

Parameters:

treatment (str)
outcome (str)
correction (str)
seed (int)

Experimental-design callables for morie.

Inspired by the designexptr.org pedagogical sequence:: https://designexptr.org/mathematical-statistics-simulation-and-computation.html https://designexptr.org/anovachapt.html https://designexptr.org/causal-inference.html

Four general-purpose statistical-design entry points, all with full R parity in r-package/morie/R/mrm_design.R:

mrm_two_treatment_test() – Welch t / Student t / Mann-Whitney U
with sensitivity to assumption breaks.

mrm_anova_oneway() – One-way ANOVA + Tukey HSD post-hoc. mrm_factorial_2k() – 2^k factorial: main effects +

interaction effects + half-normal plot coordinates for Daniel’s method.

mrm_causal_design() – Convenience wrapper around the morie
causal estimators in a designed- experiment idiom (treatment vector, outcome vector, optional covariates).

Each returns a tidy dict / dataclass with the canonical fields (estimate / se / ci / p_value / interpretation).

class morie.mrm_design.TwoTreatmentResult(estimate: 'float', se: 'float', t_statistic: 'float', df: 'float', p_welch: 'float', p_student: 'float', p_mannwhitney: 'float', ci_lower: 'float', ci_upper: 'float', n_a: 'int', n_b: 'int', interpretation: 'str')[source]¶

Bases: object

Parameters:

estimate (float)
se (float)
t_statistic (float)
df (float)
p_welch (float)
p_student (float)
p_mannwhitney (float)
ci_lower (float)
ci_upper (float)
n_a (int)
n_b (int)
interpretation (str)

estimate: float¶

se: float¶

t_statistic: float¶

df: float¶

p_welch: float¶

p_student: float¶

p_mannwhitney: float¶

ci_lower: float¶

ci_upper: float¶

n_a: int¶

n_b: int¶

interpretation: str¶

class morie.mrm_design.AnovaOneWayResult(f_statistic: 'float', p_value: 'float', df_between: 'int', df_within: 'int', means: 'dict', n_per_group: 'dict', tukey_hsd: 'pd.DataFrame', interpretation: 'str')[source]¶

Bases: object

Parameters:

f_statistic (float)
p_value (float)
df_between (int)
df_within (int)
means (dict)
n_per_group (dict)
tukey_hsd (DataFrame)
interpretation (str)

f_statistic: float¶

p_value: float¶

df_between: int¶

df_within: int¶

means: dict¶

n_per_group: dict¶

tukey_hsd: DataFrame¶

interpretation: str¶

class morie.mrm_design.Factorial2kResult(main_effects: 'dict', interaction_effects: 'dict', half_normal_coords: 'pd.DataFrame', n: 'int', k: 'int', interpretation: 'str')[source]¶

Bases: object

Parameters:

main_effects (dict)
interaction_effects (dict)
half_normal_coords (DataFrame)
n (int)
k (int)
interpretation (str)

main_effects: dict¶

interaction_effects: dict¶

half_normal_coords: DataFrame¶

n: int¶

k: int¶

interpretation: str¶

class morie.mrm_design.CausalDesignResult(estimator: 'str', estimate: 'float', se: 'float', ci_lower: 'float', ci_upper: 'float', p_value: 'float', n: 'int', n_treated: 'int', interpretation: 'str')[source]¶

Bases: object

Parameters:

estimator (str)
estimate (float)
se (float)
ci_lower (float)
ci_upper (float)
p_value (float)
n (int)
n_treated (int)
interpretation (str)

estimator: str¶

estimate: float¶

se: float¶

ci_lower: float¶

ci_upper: float¶

p_value: float¶

n: int¶

n_treated: int¶

interpretation: str¶

morie.mrm_design.mrm_two_treatment_test(a, b, *, alpha=0.05)[source]¶

Compare two-treatment outcomes under three assumption regimes.

Always returns Welch’s t-test (unequal variance), Student’s t (equal variance), and Mann-Whitney U (rank-based). The Welch’s p_value is the canonical answer; the others are the sensitivity range.

Parameters:

a (Iterable[float]) – outcome vectors under treatments A and B.
b (Iterable[float]) – outcome vectors under treatments A and B.
alpha (float) – confidence-interval level (default 0.05 -> 95% CI).

Return type:

TwoTreatmentResult

morie.mrm_design.mrm_anova_oneway(data, *, response_col, group_col, alpha=0.05)[source]¶

One-way ANOVA + Tukey HSD post-hoc on k-arm data.

Parameters:

data (DataFrame)
response_col (str)
group_col (str)
alpha (float)

Return type:

AnovaOneWayResult

morie.mrm_design.mrm_factorial_2k(data, *, response_col, factor_cols)[source]¶

2^k factorial-design analysis with main effects + interactions.

Parameters:

data (DataFrame) – data frame with response_col and k factor columns, each coded ±1 or 0/1 (will be re-coded to ±1).
response_col (str) – numeric response.
factor_cols (Sequence[str]) – list of k factor columns.

Returns:

Factorial2kResult with main + interaction effects + half- normal-plot coordinates for Daniel’s method (which lets the user identify the active effects against a null half-normal line).

Return type:

Factorial2kResult

morie.mrm_design.mrm_causal_design(data, *, treatment_col, outcome_col, covariates=(), estimator='ipw')[source]¶

Designed-experiment convenience wrapper around the morie causal estimator family.

Parameters:

data (DataFrame) – data frame containing treatment, outcome, covariates.
treatment_col (str) – binary 0/1 treatment column.
outcome_col (str) – continuous outcome column.
covariates (Sequence[str]) – optional adjustment-set columns.
estimator (str) – “ipw” (Hajek IPW) is the only implemented path; any other value falls through to a naive difference-of-means estimator. {“aipw”, “att”} were advertised in older docstrings but were never wired in.

Return type:

CausalDesignResult

Design-of-Experiments (DOE) toolkit.

Closes the Chapter-3/4/5 coverage gap from designexptr.org: * randomised complete block design (RCBD), * Latin / Graeco-Latin square, * fractional 2^(k-p) factorial, * response surface (Box-Wilson, central composite), * ANOVA power + Monte-Carlo power, * block-permutation test, * random Latin-square generation, * Bonferroni post-hoc ANOVA.

Primary references:

Box, G. E. P., Hunter, J. S., & Hunter, W. G. (2005).: Statistics for Experimenters (2nd ed.). Wiley.

Cochran, W. G., & Cox, G. M. (1957). Experimental Designs (2nd ed.). Wiley. Montgomery, D. C. (2017). Design and Analysis of Experiments (9th ed.).

Wiley.

Box, G. E. P., & Wilson, K. B. (1951). On the experimental attainment of: optimum conditions. JRSS-B, 13(1), 1-45. (Response surface origin.)
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences.: (Power machinery.)

Public callables:

mrm_anova_bonferroni(data, response, group) mrm_rcbd(data, response, treatment, block) mrm_latin_square(data, response, row, col, treatment) mrm_graeco_latin(data, response, row, col, latin, greek) mrm_fractional_factorial(data, response, factor_cols, generator) mrm_response_surface(data, response, factor_cols) mrm_anova_power(k_groups, n_per_group, effect_size_f, alpha) mrm_mc_power(simulator, n_sims, alpha) mrm_perm_block(data, response, treatment, block, n_perm) mrm_random_latin(k, seed)

morie.mrm_doe.mrm_anova_bonferroni(data, *, response_col, group_col, alpha=0.05)[source]¶

One-way ANOVA with pairwise Bonferroni-adjusted t-tests.

Returns the F-test, all pairwise means / mean-differences, the raw pairwise t-test p-values, and the Bonferroni-adjusted significance flags at level alpha.

Parameters:

data (DataFrame)
response_col (str)
group_col (str)
alpha (float)

Return type:

morie.mrm_doe.mrm_rcbd(data, *, response_col, treatment_col, block_col)[source]¶

Randomised complete block design (RCBD) two-way ANOVA.

Model: y_{ij} = mu + tau_i (treatment) + beta_j (block) + eps_{ij} Returns Type-I sequential ANOVA: block enters first, then treatment.

Parameters:

data (DataFrame)
response_col (str)
treatment_col (str)
block_col (str)

Return type:

morie.mrm_doe.mrm_latin_square(data, *, response_col, row_col, col_col, treatment_col)[source]¶

Latin-square three-way ANOVA (row, column, treatment).

Eliminates two nuisance sources simultaneously. Requires n_treatments == n_rows == n_cols (the square condition).

Parameters:

data (DataFrame)
response_col (str)
row_col (str)
col_col (str)
treatment_col (str)

Return type:

morie.mrm_doe.mrm_graeco_latin(data, *, response_col, row_col, col_col, latin_col, greek_col)[source]¶

Graeco-Latin square four-way ANOVA (row, col, Latin, Greek).

Parameters:

data (DataFrame)
response_col (str)
row_col (str)
col_col (str)
latin_col (str)
greek_col (str)

Return type:

morie.mrm_doe.mrm_fractional_factorial(data, *, response_col, factor_cols, generator=None)[source]¶

Fractional 2^(k-p) factorial main-effects + alias-structure analysis.

Assumes factor columns are coded +/-1. Computes main effects (as Yates contrasts) and reports the alias structure if generator is given (e.g. “D=ABC” means effect D is aliased with ABC).

Generator string format: comma-separated “X=YZ…” pairs.

Parameters:

data (DataFrame)
response_col (str)
factor_cols (Sequence[str])
generator (str | None)

Return type:

morie.mrm_doe.mrm_response_surface(data, *, response_col, factor_cols)[source]¶

Second-order response-surface fit (Box-Wilson 1951).

Fits y = beta0 + sum beta_i x_i + sum beta_ii x_i^2 + sum beta_ij x_i x_j by OLS and returns the stationary point x* = -0.5 * B_inv @ b (if invertible), where B is the symmetric quadratic-term matrix and b is the linear-term vector.

Parameters:

data (DataFrame)
response_col (str)
factor_cols (Sequence[str])

Return type:

morie.mrm_doe.mrm_anova_power(k_groups, n_per_group, effect_size_f, *, alpha=0.05)[source]¶

Power of one-way ANOVA given Cohen’s f effect size.

Power = P(F > F_crit | non-central F with ncp = N * f^2).

Parameters:

k_groups (int)
n_per_group (int)
effect_size_f (float)
alpha (float)

Return type:

morie.mrm_doe.mrm_mc_power(simulator, *, n_sims=1000, alpha=0.05, seed=42)[source]¶

Empirical power via Monte-Carlo: caller supplies a simulator that draws data under the alternative hypothesis and returns a p-value.

Parameters:

simulator (Callable[[int], float]) – callable(rng_seed) -> p_value
n_sims (int) – number of simulated datasets.
alpha (float) – type-I error level.
seed (int)

Returns:

dict with empirical_power, sd_pwr, ci95_lower, ci95_upper.

Return type:

morie.mrm_doe.mrm_perm_block(data, *, response_col, treatment_col, block_col, n_perm=1000, seed=42)[source]¶

Block-permutation test for treatment effect.

Randomly permutes treatment labels within each block, recomputes the treatment-mean difference, and compares to the observed statistic. Two-sided p-value.

Parameters:

data (DataFrame)
response_col (str)
treatment_col (str)
block_col (str)
n_perm (int)
seed (int)

Return type:

morie.mrm_doe.mrm_random_latin(k, *, seed=42)[source]¶

Generate a random k x k Latin square via row-cycle + permutation.

Standard algorithm: build the cyclic Latin square, then randomly permute rows, columns, and symbols. Produces a uniform sample over a subset of Latin squares (not a uniform sample over all Latin squares – exact uniform sampling requires Jacobson-Matthews for k > 5).

Parameters:

k (int)
seed (int)

Return type:

DataFrame

morie.mrm_primitives — reusable identification primitives for the Multilevel Reconciliation Methodology (MRM) framework.

These primitives are the building blocks that morie modules (and any external code) compose to construct identification strategies. Each primitive is a thin, well-typed function that takes a pandas DataFrame plus a few configuration kwargs and returns a structured result.

Five primitives currently shipped (v0.6.1):

gentrification_panel() — categorical baseline-conditional
gentrification coding

spatial_spillover_decomposition()

— Spatial Durbin / SAR direct-
indirect-total decomposition

synthetic_area_exposure() — SAE-based exposure offset
(generalises far beyond SMI)

threshold_specific_ordinal() — Bayesian cumulative-logit with
threshold-varying coefficients

score_net_residual() — two-stage score-then-audit;
residual race effect = bias

The primitives correspond to the patterns extracted from Laniyonu (2018), Laniyonu & Goff (2021), and O’Connell & Laniyonu (2025); see ROADMAP.md v0.6.1 + papers/mrm-formulations-paper for the academic context.

morie.mrm_primitives.gentrification_panel(df, *, baseline_income_col, baseline_rent_col, growth_college_col, growth_rent_col, baseline_marginalisation_quantile=0.5, gentrification_growth_quantile=0.667, return_thresholds=False)[source]¶

Construct a baseline-conditional 3-level gentrification factor.

Parameters:

df (pd.DataFrame) – One row per tract. Must contain the four columns named below.
baseline_income_col (str) – Income + rent at the baseline period (e.g. 2000 census).
baseline_rent_col (str) – Income + rent at the baseline period (e.g. 2000 census).
growth_college_col (str) – Growth in college / BA-share between baseline and follow-up.
growth_rent_col (str) – Growth in median rent between baseline and follow-up.
baseline_marginalisation_quantile (float, default 0.5) – Tract is “eligible” to gentrify if baseline income AND baseline rent are at or below this quantile. Default mirrors Laniyonu (2018) – bottom half on both.
gentrification_growth_quantile (float, default 0.667) – Tract “gentrified” if BOTH growth-college AND growth-rent are above this quantile across the panel. Default mirrors Laniyonu (2018) – top tercile on both.
return_thresholds (bool, default False) – If True, also returns the computed cut-point values for the four conditions (useful for documenting the operationalisation).

Returns:

"ineligible", "eligible", "gentrified". Index matches df.index.

Return type:

pd.Series of object dtype with three categorical levels

Notes

The dichotomous baseline cut (50%) + dichotomous growth cut (67%) are both configurable. Laniyonu (2018) reports five sensitivity-of-coding specifications; rerun with different quantiles to mirror them.

morie.mrm_primitives.spatial_spillover_decomposition(*, rho, beta_direct, beta_spatial, W, coefficient_names=None)[source]¶

Compute the SDM direct/indirect/total decomposition.

Implements the standard formula:

(I - rho * W)^{-1} * (I * beta_direct + W * beta_spatial)

The diagonal of the resulting matrix is the per-observation direct effect (averaged for the summary); off-diagonal-row-sum is the indirect effect; total = direct + indirect.

Parameters:

rho (float) – Spatial-autoregressive coefficient from the fitted SDM.
beta_direct (np.ndarray of shape (K,)) – Coefficients on the K covariates (no lagged terms).
beta_spatial (np.ndarray of shape (K,)) – Coefficients on the spatially-lagged covariates ($WX$). Set to zeros if you fit a SAR (lag-only) model.
W (np.ndarray of shape (N, N)) – Row-standardised spatial weight matrix.
coefficient_names (list[str], optional) – Human-readable names for the K covariates.

Returns:

One entry per covariate.

Return type:

list[SpilloverDecomposition]

Notes

This is the formula side; the caller is responsible for fitting the SDM (via pysal / spreg / statsmodels spatial extension) and passing in the estimated rho + beta vectors. Fitting is left out of the primitive so morie can stay numpy-only at the framework layer.

morie.mrm_primitives.synthetic_area_exposure(*, survey_df, survey_trait_col, survey_covariate_cols, area_df, area_population_col, fit_callable=None, return_per_area_rate=False)[source]¶

Compute synthetic exposure offset per area.

Parameters:

survey_df (pd.DataFrame) – Survey microdata (e.g., NCS-R long-form). One row per respondent. Must contain survey_trait_col (0/1) and the named covariates.
survey_trait_col (str) – Binary column indicating the trait of interest (e.g., SMI).
survey_covariate_cols (list[str]) – Covariates available in BOTH the survey and the area dataset. Logistic coefficients are fit on the survey and applied to the area marginals.
area_df (pd.DataFrame) – One row per area (tract, precinct, etc.). Must contain the same covariate columns as survey_covariate_cols (as area-level means / proportions).
area_population_col (str) – Adult-population column in area_df.
fit_callable (callable, optional) – Override for the default logistic fit. Signature: fit_callable(X, y) -> coef, where coef is a vector of length len(survey_covariate_cols) + 1 (intercept first).
return_per_area_rate (bool, default False) – If True, returns a tuple (exposure_offset, predicted_rate).

Returns:

pd.Series indexed by ``area_df.index`` – = predicted P(trait | area marginals) × area population. Suitable for exposure= in a Poisson / negative-binomial GLM that counts trait-specific events.

Return type:

synthetic exposure offset

Notes

This is “synthetic” because the trait is not directly observed at the area level — it’s predicted from a national probability sample. Cite the underlying survey explicitly when reporting.

morie.mrm_primitives.threshold_specific_ordinal(df, *, outcome_col, covariate_cols, ordinal_levels=None, fit_proportional_odds_first=True, max_iter=200, tol=1e-06)[source]¶

Fit a threshold-specific cumulative-logit ordinal regression.

Parameters:

df (pd.DataFrame) – Sample microdata, one row per unit.
outcome_col (str) – Ordinal outcome. Either category strings (preserve order with ordinal_levels) or already-encoded integer 0..K-1.
covariate_cols (list[str]) – Predictors. Categorical covariates should be pre-dummied (one-hot) before passing.
ordinal_levels (list[str], optional) – Explicit category ordering. If None and the outcome is string-valued, sorted(unique()) is used (alphabetical), which is rarely what you want — pass this explicitly.
fit_proportional_odds_first (bool, default True) – If True, also fits the standard proportional-odds model and runs Brant’s (1990) Wald test of proportional odds. Helps the caller decide whether the threshold-specific fit is empirically warranted.
max_iter (numerical tolerances for the IRLS-style fit.)
tol (numerical tolerances for the IRLS-style fit.)

Return type:

ThresholdSpecificOrdinalResult

morie.mrm_primitives.score_net_residual(df, *, score_col, outcome_col, sensitive_cols, control_cols=None, family='logit', bootstrap_replicates=200, random_state=20260513)[source]¶

Two-stage score-then-audit residual race effect.

Parameters:

df (pd.DataFrame) – Sample microdata.
score_col (str) – The actuarial score the institution claims to use.
outcome_col (str) – Binary outcome (parole granted, loan approved, etc.) — 0/1.
sensitive_cols (list[str]) – Race / gender / etc. indicators to test for residual bias. Pre-dummy categorical attributes.
control_cols (list[str], optional) – Additional controls (age, prior offences, etc.).
family ({"logit", "linear"}, default "logit") – Stage-2 model. Use "logit" for binary outcomes, "linear" for continuous.
bootstrap_replicates (int, default 200) – Resamples for SEs. Set to 0 to skip (returns SE = nan).
random_state (int, default 20260513) – RNG seed for reproducibility.

Return type:

ScoreNetResidualResult

Mathematical-statistics / simulation / computation toolkit.

Closes the Chapter-2 coverage gap from designexptr.org/mathematical- statistics-simulation-and-computation.html.

Primary references:

Wilks, S. S. (1962). Mathematical Statistics. Wiley. Casella, G. & Berger, R. L. (2002). Statistical Inference. Duxbury. Lehmann, E. L. & Romano, J. P. (2005). Testing Statistical Hypotheses.

Public callables:

mrm_oneprop_test(x, n, p0) – exact-binomial + Wald one-prop test mrm_twoprop_test(x1, n1, x2, n2)

– chi-square + Fisher exact + Wald

mrm_var_test(sample, sigma0_sq): – chi-square test for σ² (Wilks 1962)

mrm_qq_plot(sample, dist) – Q-Q plot coordinates mrm_clt_demo(base_distribution, n_samples, sample_size)

– Central Limit Theorem demonstrator

mrm_pit(sample, dist) – Probability Integral Transform

morie.mrm_mathstats.mrm_oneprop_test(x, n, p0, *, alpha=0.05)[source]¶

One-proportion test (binomial exact + Wald approximation).

Parameters:

x (int) – number of successes.
n (int) – number of trials.
p0 (float) – null-hypothesis proportion.
alpha (float) – CI level (default 0.05 -> 95% CI).

Return type:

OnePropResult

morie.mrm_mathstats.mrm_twoprop_test(x1, n1, x2, n2, *, alpha=0.05)[source]¶

Two-proportion test (chi-square + Fisher’s exact + Wald).

Parameters:

x1 (int)
n1 (int)
x2 (int)
n2 (int)
alpha (float)

Return type:

TwoPropResult

morie.mrm_mathstats.mrm_var_test(sample, sigma0_sq, *, alpha=0.05)[source]¶

Chi-square test for variance σ² = σ₀² (Wilks 1962).

Parameters:

sample (Iterable[float]) – sample data (assumed iid normal).
sigma0_sq (float) – null hypothesis variance.
alpha (float)

Return type:

VarTestResult

morie.mrm_mathstats.mrm_qq_plot(sample, *, dist='norm', **dist_kwargs)[source]¶

Compute Q-Q plot coordinates (theoretical vs empirical quantiles).

Parameters:

sample (Iterable[float]) – sample data.
dist (str) – scipy.stats distribution name (default “norm”).
**dist_kwargs – distribution parameters (e.g. loc=0, scale=1 for norm).

Returns:

DataFrame with theoretical, empirical, and rank columns. Suitable for plotting empirical against theoretical.

Return type:

DataFrame

morie.mrm_mathstats.mrm_clt_demo(base_distribution='uniform', *, n_samples=1000, sample_size=30, seed=42, **dist_kwargs)[source]¶

Generate sample means from a base distribution for CLT visualisation.

For increasing sample_size, the distribution of the sample mean should approach Normal(μ_X, σ_X²/n) by the CLT.

Parameters:

base_distribution (str) – scipy.stats distribution name to draw from. Default “uniform” (clear non-normal base) is pedagogical.
n_samples (int) – number of sample means to compute.
sample_size (int) – size of each sample.
seed (int) – RNG seed.
**dist_kwargs – parameters of the base distribution.

Returns:

DataFrame with one row per sample mean.

Return type:

DataFrame

morie.mrm_mathstats.mrm_pit(sample, *, dist='norm', **dist_kwargs)[source]¶

Apply the Probability Integral Transform to a sample.

If X ~ F, then F(X) ~ Uniform(0,1). Returns the empirical and PIT-transformed values; if the assumed distribution F is correct, the transformed values should be ≈ Uniform(0,1).

Parameters:

sample (Iterable[float]) – sample data.
dist (str) – assumed CDF (scipy.stats distribution name).
**dist_kwargs – distribution parameters.

Returns:

DataFrame with raw and U = F(raw) columns.

Return type:

DataFrame

Causal-inference diagnostics: balance, overlap, SUTVA-style assumption checks, and the median causal effect estimator.

Closes the diagnostic-suite gap surfaced in the designexptr.org Chapter 7 (Causal Inference) coverage audit.

Primary references:

Imbens, G. W., & Rubin, D. B. (2015). Causal Inference for Statistics,: Social and Biomedical Sciences. Cambridge University Press. – standardised-difference / balance / overlap diagnostics
Rosenbaum, P. R., & Rubin, D. B. (1985). Constructing a control: group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician, 39(1), 33-38. – balancing-property formalisation
Cole, S. R., & Hernán, M. A. (2008). Constructing inverse probability: weights for marginal structural models. AJE, 168(6), 656-664. – positivity violation diagnostics

Public callables:

mrm_standardised_difference(data, treatment, covariates)

– Imbens-Rubin %SMD for every covariate, pre- vs post-adjustment.

mrm_check_balancing(data, treatment, covariates, threshold=10)

– composite balance verdict (Rosenbaum-Rubin threshold = 10%).

mrm_check_overlap(data, treatment, covariates)

– propensity-score support overlap; flags positivity violations.

mrm_median_causal_effect(data, treatment, outcome, covariates)

– median(Y(1)) - median(Y(0)) via PS-matching counterfactual.

mrm_assumptions_check(data, treatment, outcome, covariates)

– composite SUTVA / unconfoundedness / probabilistic-assignment: audit; returns a verdict per assumption with diagnostic statistics.

morie.mrm_diagnostics.mrm_standardised_difference(data, *, treatment_col, covariates)[source]¶

Imbens-Rubin standardised difference per covariate.

For continuous X: SMD = (x̄_t - x̄_c) / sqrt( (s²_t + s²_c) / 2 ) For binary X: SMD = (p̄_t - p̄_c) / sqrt( (p_t(1-p_t) + p_c(1-p_c)) / 2 )

Returned as percent (Imbens-Rubin convention). |SMD| > 10% is the standard balance threshold (Austin 2009).

Parameters:

data (DataFrame) – data.frame.
treatment_col (str) – binary 0/1 treatment column.
covariates (Sequence[str]) – covariates to assess.

Returns:

covariate, mean_treated, mean_control, pooled_sd, smd_pct.

Return type:

DataFrame

morie.mrm_diagnostics.mrm_check_balancing(data, *, treatment_col, covariates, threshold_pct=10.0)[source]¶

Composite balance verdict using the Imbens-Rubin %SMD criterion.

A design is “balanced on X” if every |SMD(X_i)| ≤ threshold (default 10 percentage points, Austin 2009 / Rosenbaum-Rubin 1985).

Parameters:

data (DataFrame)
treatment_col (str)
covariates (Sequence[str])
threshold_pct (float)

Return type:

BalanceResult

morie.mrm_diagnostics.mrm_check_overlap(data, *, treatment_col, covariates)[source]¶

Propensity-score support overlap diagnostic (Cole-Hernán 2008).

Flags positivity violations: units with extreme e(X) ∉ (0.01, 0.99).

Parameters:

data (DataFrame)
treatment_col (str)
covariates (Sequence[str])

Return type:

OverlapResult

morie.mrm_diagnostics.mrm_median_causal_effect(data, *, treatment_col, outcome_col, covariates)[source]¶

Median causal effect via 1:1 nearest-neighbour PS matching.

Estimates median(Y(1)) − median(Y(0)) on the matched sample; robust to outliers in a way the mean ATE is not.

Parameters:

data (DataFrame)
treatment_col (str)
outcome_col (str)
covariates (Sequence[str])

Return type:

MedianCausalResult

morie.mrm_diagnostics.mrm_assumptions_check(data, *, treatment_col, outcome_col, covariates)[source]¶

Composite Rubin-style identifiability assumption check.

Reports each assumption with diagnostic evidence + a flag:

SUTVA (Stable Unit Treatment Value Assumption):
We cannot test SUTVA from data alone, but we report: (a) the spread of treatment within cluster groups (small

within-cluster variance hints at interference);

a clustering-of-residuals statistic.

Unconfoundedness (no unmeasured confounders):
We cannot test this from data alone either; we report: (a) E-value from the existing ATE – the minimum unmeasured-

confounder strength that would explain away the effect;

covariate balance after adjustment.

Probabilistic assignment / positivity:
Empirical: fraction of e(x) outside (0.01, 0.99).

Parameters:

data (DataFrame)
treatment_col (str)
outcome_col (str)
covariates (Sequence[str])

Return type:

AssumptionsCheckResult

Causal DAGs for MRM, and the bundled starting-point structures.

Parity with rmorie’s R/dag_native.R (morie_dag, morie_mrm_dags).

A DAG here is a thin, checked container over an edge list. It carries its own acyclicity proof and keeps edges as (from, to) tuples, which is the representation morie.viz.dag_plot and morie.fn.bdcrt.backdoor_criterion already take – so a graph built here plots and identifies without conversion.

The bundled structures are starting points for justice-system work, not claims about a particular jurisdiction. Both encode the same shape: a set of common causes of the exposure and the outcome, which is exactly the shape that makes an unadjusted comparison misleading.

class morie.mrm_graphs.CausalDag(nodes: 'list[str]', edges: 'list[tuple[str, str]]', exposure: 'str', outcome: 'str', latent: 'list[str]' = <factory>)[source]¶

Bases: object

Parameters:

nodes (list[str])
edges (list[tuple[str, str]])
exposure (str)
outcome (str)
latent (list[str])

nodes: list[str]¶

edges: list[tuple[str, str]]¶

exposure: str¶

outcome: str¶

latent: list[str]¶

parents(node)[source]¶

Parameters:: node (str)
Return type:: list[str]

children(node)[source]¶

Parameters:: node (str)
Return type:: list[str]

backdoor(adjust=())[source]¶

Does adjust satisfy the back-door criterion for this graph?

Delegates to morie.fn.bdcrt, which implements Pearl’s two conditions; the result exposes .satisfied.

morie.mrm_graphs.causal_dag(edges, exposure, outcome, latent=())[source]¶

Build a DAG from edges, rejecting cycles.

edges may be "A -> B" strings or (from, to) pairs. The exposure and the outcome must both appear in the graph: a DAG whose exposure is not in it cannot identify anything, and saying so here beats an empty adjustment set later.

Parameters:

exposure (str)
outcome (str)

Return type:

CausalDag

morie.mrm_graphs.mrm_dags()[source]¶

The bundled MRM starting-point graphs.

placement: race, prior record and age are common causes of both the placement decision and the outcome, so comparing outcomes by placement without adjusting for them confounds the three.

use_of_force: neighbourhood and race cause both police contact and force, so force conditional on contact is not the effect of contact.

Return type:: dict[str, CausalDag]

MRM-framework analyses on Ontario OTIS data.

Five Python callables that mirror the R mrm_otis_* family in r-package/morie/R/mrm_otis.R. Used in the MRM empirical companion paper (papers/morie-empirical-paper/).

Functions:

mrm_otis_placement_concentration: Hill alpha + Gini + top-k%: concentration on b09 banded placement counts.
mrm_otis_seg_duration_km: pooled / stratified survival summary: of segregation-placement durations from b01.
mrm_otis_mortification_cooccurrence: pairwise Cramer’s V on the: three b01 alert columns.
mrm_otis_region_locality: chi-square + Cramer’s V + diagonal-share: on Region_AtTimeOfPlacement x Region_MostRecentPlacement.

OTIS UniqueIndividual_ID (format YYYY-XXXXX-SG) is randomly reassigned every fiscal year; cross-year tracking is invalid by construction. All analyses operate within fiscal year.

morie.mrm_otis.mrm_classify_mandela(data, *, duration_col='NumberConsecutiveDays_Segregation', year_col='EndFiscalYear', id_col='UniqueIndividual_ID', threshold_days=15, denominator='individual_any', broader_rc=False, alert_cols=('MentalHealth_Alert', 'SuicideRisk_Alert', 'SuicideWatch_Alert'), meaningful_contact_col=None)[source]¶

Mandela Rules classifier on OTIS-style placement data.

Python parity of the R-side morie::mrm_classify_mandela(). Classifies placement records under the UN Nelson Mandela Rules (A/RES/70/175) which define prolonged solitary as any continuous placement exceeding 15 days. Three denominator conventions:

“row” per-placement rate

“individual_any” proportion of individuals with any
placement above threshold

“individual_cumulative” proportion of individuals whose
cumulative within-year segregation days exceed threshold

Parameters:

data (DataFrame) – long-format placement frame.
duration_col (str) – OTIS-b01 column names.
year_col (str) – OTIS-b01 column names.
id_col (str) – OTIS-b01 column names.
threshold_days (int) – Mandela duration threshold (default 15).
denominator (str) – one of row/individual_any/individual_cumulative.
broader_rc (bool) – if True, numerator also counts placements with alert-complexity >= 2 (broader restrictive-confinement).
alert_cols (tuple) – alert columns for the broader rate.
meaningful_contact_col (str | None) – optional federal Sprott-Doob style “1 = meaningful contact met” indicator; rows with met contact are excluded from the numerator.

Returns:

DataFrame with year / denominator / n_mandela / rate / pct / n_broader_rc / rate_broader columns, one row per fiscal year plus a final “pooled” row.

Return type:

DataFrame

morie.mrm_otis.mrm_otis_placement_concentration(data, *, year_col='EndFiscalYear', band_col='NumberPlacements_Segregation', count_col='NumberIndividuals_Segregation', gender_col=None, gender_keep=None, x_min=1.0, top_pct=0.05)[source]¶

Per-individual placement-count concentration on OTIS b09.

Parameters:

data (DataFrame) – b09 long-format data.
year_col (str) – OTIS b09 column names.
band_col (str) – OTIS b09 column names.
count_col (str) – OTIS b09 column names.
gender_col (str | None) – optional gender filter.
gender_keep (Iterable[str] | None) – optional gender filter.
x_min (float) – Hill-MLE lower cutoff.
top_pct (float) – top concentration share (e.g. 0.05 = top 5%).

Returns:

DataFrame indexed by year (final row “pooled”) with n_individuals, n_placements, mean_per_individual, gini, hill_alpha, top_pct_share.

Return type:

DataFrame

morie.mrm_otis.mrm_otis_seg_duration_km(data, *, duration_col='NumberConsecutiveDays_Segregation', group_cols=None, probs=(0.5, 0.25, 0.1, 0.05, 0.01), mandela_threshold=15)[source]¶

KM-style summary of b01 segregation placement durations.

Replaces the misreading of UniqueIndividual_ID = YYYY-XXXXX-SG as a persistent person identifier (which produces a spurious ~210-day cross-year TTR artifact).

Parameters:

data (DataFrame) – b01 placement-level data.
duration_col (str) – column with day-count durations.
group_cols (Sequence[str] | None) – optional stratifying columns.
probs (Sequence[float]) – survival levels; days_at_SXX is the day at which the empirical survival function reaches XX% (the 1 - p quantile, type 7), as the R arm.
mandela_threshold (int) – day cutoff defining Mandela-prolonged.

Returns:

DataFrame with per-stratum n, mean, median, q25, fraction above the Mandela threshold, and median duration among those above.

Return type:

DataFrame

morie.mrm_otis.mrm_otis_mortification_cooccurrence(data, *, alert_cols=('MentalHealth_Alert', 'SuicideRisk_Alert', 'SuicideWatch_Alert'))[source]¶

Pairwise Cramer’s V of OTIS b01 alert columns.

Parameters:

data (DataFrame)
alert_cols (Sequence[str])

Return type:

DataFrame

morie.mrm_otis.mrm_otis_region_locality(data, *, region_at_col='Region_AtTimeOfPlacement', region_recent_col='Region_MostRecentPlacement')[source]¶

OTIS b01 region locality: chi-square + Cramer’s V + diagonal share.

Parameters:

data (DataFrame)
region_at_col (str)
region_recent_col (str)

Return type:

RegionLocalityResult

MRM-framework analyses on Ontario SIU (Special Investigations Unit) data.

Python parity for mrm_siu_* (see r-package/morie/R/mrm_siu.R).

Functions:

mrm_siu_case_to_decision_km: KM-style summary of the gap from: date_of_incident_iso to date_of_director_decision_iso, with right-censoring of still-open cases.

mrm_siu_per_service_rate: per-(service, year) case counts. mrm_siu_outcome_classifier: tabulates Director’s-decision outcomes

by service with within-service share.

Unlike OTIS (no placement dates) and TPS (no per-person ID), SIU exposes per-case dates with a stable police_service jurisdiction, enabling a real time-to-outcome analysis. This is the analysis the MA-thesis “210-day TTR” claim should have been.

morie.mrm_siu.mrm_siu_case_to_decision_km(data, *, incident_col='date_of_incident_iso', decision_col='date_of_director_decision_iso', service_col='police_service', censor_open_cases=True, min_n=5)[source]¶

KM-style time-from-incident-to-Director’s-decision summary.

Parameters:

data (DataFrame)
incident_col (str)
decision_col (str)
service_col (str)
censor_open_cases (bool)
min_n (int)

Return type:

SIUCaseDecisionResult

morie.mrm_siu.mrm_siu_per_service_rate(data, *, service_col='police_service', incident_col='date_of_incident_iso', stratify_col=None)[source]¶

Per-police-service case counts by year (and optional stratum).

Parameters:

data (DataFrame)
service_col (str)
incident_col (str)
stratify_col (str | None)

Return type:

DataFrame

morie.mrm_siu.mrm_siu_outcome_classifier(data, *, outcome_col='director_decision_category', service_col='police_service')[source]¶

Tabulate SIU Director’s-decision outcomes by service.

Parameters:

data (DataFrame)
outcome_col (str)
service_col (str)

Return type:

DataFrame

MRM-framework analyses on Toronto Police Service (TPS) open data.

Python parity for mrm_tps_* (see r-package/morie/R/mrm_tps.R).

Functions:

mrm_tps_levy_scaling: Hill-MLE Pareto exponent of inter-incident: step lengths on lat/long event streams.
mrm_tps_moran_clustering: gridded Moran’s I + DBSCAN summary on: WGS84 event coordinates.
mrm_tps_neighbourhood_recurrence_km: per-neighbourhood inter-event: gap summary.
mrm_tps_load_hawkes_refit: convenience loader for the per-category: Hawkes (Markovian + Weibull/sin) manifest.

morie.mrm_tps.mrm_tps_levy_scaling(data, *, date_col='OCC_DATE', lat_col='LAT_WGS84', lon_col='LONG_WGS84', min_step_km=0.5, x_min=None)[source]¶

Hill-MLE exponent on TPS inter-incident step lengths (km).

Parameters:

data (DataFrame)
date_col (str)
lat_col (str)
lon_col (str)
min_step_km (float)
x_min (float | None)

Return type:

LevyResult

morie.mrm_tps.mrm_tps_moran_clustering(data, *, lat_col='LAT_WGS84', lon_col='LONG_WGS84', grid_resolution=40, dbscan_eps=0.3, dbscan_minpts=5)[source]¶

Gridded Moran’s I + DBSCAN summary on TPS WGS84 events.

Parameters:

data (DataFrame)
lat_col (str)
lon_col (str)
grid_resolution (int)
dbscan_eps (float)
dbscan_minpts (int)

Return type:

MoranClusteringResult

morie.mrm_tps.mrm_tps_neighbourhood_recurrence_km(data, *, date_col='OCC_DATE', hood_col='HOOD_158', min_gap_days=0.0)[source]¶

Per-neighbourhood inter-event gap summary on TPS events.

Parameters:

data (DataFrame)
date_col (str)
hood_col (str)
min_gap_days (float)

Return type:

DataFrame

morie.mrm_tps.mrm_tps_load_hawkes_refit(manifest_path=None)[source]¶

Load the precomputed per-category Hawkes refit manifest.

The reference manifest is shipped with the package at morie/data/paper_hawkes_refit.json. Pass manifest_path=None (the default) to read the bundled copy; pass an explicit path to load a user-supplied refit.

Parameters:: manifest_path (str | Path | None)
Return type:: DataFrame

MRM-framework analyses on Use-of-Force (UOF) data.

Six rich-output callables for the MRM-UOF analytic suite. Each function returns a morie.fn._richresult.RichResult carrying a multi-paragraph natural-language interpretation, a structured table / section payload for programmatic access, and an explicit warnings list. The functions mirror the conceptual structure of the R-side r-package/morie/R/mrm_uof.R and the Python-side mrm_otis family, but address the use-of-force domain: concentration of force incidents across services, weapon-by-service contingency, year-on-year change with optional change-point detection, region locality stability across reporting periods, demographic disparity in outcome rates with risk-ratio confidence intervals, and a CKAN-aware data-quality audit.

All callables are pure (no IO, no class state), accept pandas inputs, depend only on numpy + scipy.stats (with an optional native PELT change-point detector), and produce results suitable for inclusion in MRM empirical papers without further post-processing.

Functions¶

mrm_uof_force_concentration: Hill-MLE Pareto alpha + Gini + top-5 / top-10 share for incidents aggregated by force / service.
mrm_uof_weapon_diversity: Weapon-by-service contingency: chi-square, Cramer’s V, top-3 standardised residuals.
mrm_uof_yoy_change: Year-on-year percentage change with native PELT-based change-point detection (PELT / rbf) or a manual largest-gap fallback.
mrm_uof_region_locality: Region-at-time vs region-now contingency: diagonal share, chi-square and Cramer’s V.
mrm_uof_demographic_disparity: Per-category outcome rates with Wilson 95 percent intervals, risk-ratio relative to a baseline group, optional bootstrap percentile interval on the risk ratio.
mrm_uof_data_quality_audit: Schema and null audit; CKAN-sidecar aware, optional duck-typed schema comparison, suspect-flag interpretation.

morie.mrm_uof.mrm_uof_force_concentration(df, force_col, count_col=None)[source]¶

Concentration of UOF incidents across forces / services.

Aggregates per-force incident counts and reports a Hill-MLE Pareto tail exponent, the Gini coefficient, and the top-5 / top-10 concentration shares.

Parameters:

df (pandas.DataFrame) – Long-format UOF data with one row per incident (when count_col is None) or one row per force-period with a numeric count column.
force_col (str) – Column identifying the force / service / agency.
count_col (str, optional) – If supplied, the per-row incident count to sum within each force; otherwise each row counts as one incident.

Returns:

payload keys: n_forces, n_incidents, pareto_alpha_mle, gini, top5_share, top10_share, counts (list of (force, count) tuples sorted descending), n.

Return type:

Examples

>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({"force": ["A"] * 50 + ["B"] * 5})
>>> r = mrm_uof_force_concentration(df, "force")
>>> r.n_forces
2

morie.mrm_uof.mrm_uof_weapon_diversity(df, weapon_col, force_col)[source]¶

Weapon-by-force contingency test.

Builds a weapon x force contingency table, runs a chi-square test of independence, computes Cramer’s V, and reports the top-3 (weapon, force) cells by standardised Pearson residual.

Parameters:

df (DataFrame)
weapon_col (str)
force_col (str)

Return type:

morie.mrm_uof.mrm_uof_yoy_change(dfs_by_year=None, df=None, year_col=None, count_col=None)[source]¶

Year-on-year change in incident counts with optional change-point.

Either supply dfs_by_year (a mapping from integer year to a DataFrame) or df together with year_col. When count_col is None the incident count is the number of rows; otherwise it is the sum of count_col within each year.

Parameters:

dfs_by_year (Mapping[int, DataFrame] | None)
df (DataFrame | None)
year_col (str | None)
count_col (str | None)

Return type:

morie.mrm_uof.mrm_uof_region_locality(df, region_at_col, region_now_col)[source]¶

Region-at-time vs region-now locality contingency.

Parameters:

df (DataFrame)
region_at_col (str)
region_now_col (str)

Return type:

morie.mrm_uof.mrm_uof_demographic_disparity(df, demo_col, outcome_col, baseline=None, bootstrap_reps=0)[source]¶

Demographic disparity in outcome rates with risk-ratio CIs.

Parameters:

df (DataFrame)
demo_col (str)
outcome_col (str)
baseline (str | None)
bootstrap_reps (int)

Return type:

morie.mrm_uof.mrm_uof_data_quality_audit(df, sidecar=None, expected_schema=None)[source]¶

Schema + null + suspect-value audit of a UOF dataframe.

Parameters:

df (DataFrame)
sidecar (Mapping[str, Any] | None)
expected_schema (Any)

Return type:

Stock and flow measures for confinement, and MRM across the OTIS strata.

Full parity with rmorie’s R/mrm_stockflow.R and with the measures in rmoriebricklayer’s R/custody.R. The arithmetic is implemented here rather than delegated: morie is not a wrapper.

The measures are Lakner’s. Given person-days x_i served over a period of t days by N people:

adp = sum(x_i) / t the average daily population – days per DAY,
a STOCK. p.15

alos = sum(x_i) / N the average length of stay – days per PERSON,
the FLOW side. p.16

adp = N_a * alos / t the identity, eq 2.4 p.18 X_t = mean(C) * t person-days from periodic head counts, eq 2.7

after Lakner, A Manual of Statistical Sampling Methods for Corrections Planners (University of Illinois at Urbana-Champaign, April 1976).

Two caveats that change conclusions rather than decorate them:

p.16-17: t must exceed the longest stay, or alos is biased DOWNWARD.
The flow and stock rates can move in OPPOSITE directions, because days = people x stay. On the published Ontario segregation release the number of people fell 24.0 percent while stays grew 43.5 percent, so detention days ROSE 9.0 percent: the flow rate fell 27.6 percent and the stock rate rose 3.9 percent over the same window. Quoting one of them alone reverses the finding, so both are returned and the decomposition is checked to multiply out exactly.

class morie.mrm_stockflow.StockFlowResult(period: 'list[str]', people: 'list[float]', days: 'list[float]', alos: 'list[float]', adp: 'list[float]', people_change: 'list[float | None]', alos_change: 'list[float | None]', days_change: 'list[float | None]', adp_change: 'list[float | None]', t: 'list[float]', baseline: 'str', exposure: 'list[float] | None' = None, flow_rate: 'list[float] | None' = None, stock_rate: 'list[float] | None' = None, flow_rate_change: 'list[float | None] | None' = None, stock_rate_change: 'list[float | None] | None' = None, per: 'float | None' = None)[source]¶

Bases: object

Parameters:

period (list[str])
people (list[float])
days (list[float])
alos (list[float])
adp (list[float])
people_change (list[float | None])
alos_change (list[float | None])
days_change (list[float | None])
adp_change (list[float | None])
t (list[float])
baseline (str)
exposure (list[float] | None)
flow_rate (list[float] | None)
stock_rate (list[float] | None)
flow_rate_change (list[float | None] | None)
stock_rate_change (list[float | None] | None)
per (float | None)

period: list[str]¶

people: list[float]¶

days: list[float]¶

alos: list[float]¶

adp: list[float]¶

people_change: list[float | None]¶

alos_change: list[float | None]¶

days_change: list[float | None]¶

adp_change: list[float | None]¶

t: list[float]¶

baseline: str¶

exposure: list[float] | None = None¶

flow_rate: list[float] | None = None¶

stock_rate: list[float] | None = None¶

flow_rate_change: list[float | None] | None = None¶

stock_rate_change: list[float | None] | None = None¶

per: float | None = None¶

to_frame()[source]¶

opposite_signs()[source]¶

Do the flow and stock rates move in opposite directions?

The reason both are reported: on the Ontario release one fell while the other rose.

Return type:: bool

class morie.mrm_stockflow.StaySummary(n: 'int', total_days: 'float', mean: 'float', sd: 'float | None', median: 'float', iqr: 'float', max: 'float', se: 'float | None', lower: 'float | None', upper: 'float | None')[source]¶

Bases: object

Parameters:

n (int)
total_days (float)
mean (float)
sd (float | None)
median (float)
iqr (float)
max (float)
se (float | None)
lower (float | None)
upper (float | None)

n: int¶

total_days: float¶

mean: float¶

sd: float | None¶

median: float¶

iqr: float¶

max: float¶

se: float | None¶

lower: float | None¶

upper: float | None¶

to_frame()[source]¶

class morie.mrm_stockflow.OtisStockFlowResult(stock_flow: 'StockFlowResult', reconciliation: 'list[dict]' = <factory>, strata_agree: 'bool' = False, decomposition_exact: 'bool' = False)[source]¶

Bases: object

Parameters:

stock_flow (StockFlowResult)
reconciliation (list[dict])
strata_agree (bool)
decomposition_exact (bool)

stock_flow: StockFlowResult¶

reconciliation: list[dict]¶

strata_agree: bool = False¶

decomposition_exact: bool = False¶

to_frame()[source]¶

morie.mrm_stockflow.adp(days, t=365)[source]¶

Average daily population: person-days per day of the period.

Parameters:: t (float)
Return type:: float

morie.mrm_stockflow.alos(days, n)[source]¶

Average length of stay: person-days per person.

n counts people admitted AND released within the period (Lakner p.16); counting everyone present inflates the denominator and biases the stay downward.

Return type:: float

morie.mrm_stockflow.admissions(adp_value, alos_value, t=365)[source]¶

Admissions implied by the identity adp = N_a * alos / t.

Parameters:: t (float)
Return type:: float

morie.mrm_stockflow.adp_from_counts(counts, t=365)[source]¶

Person-days from periodic head counts: mean(counts) * t (eq 2.7).

Parameters:: t (float)
Return type:: float

morie.mrm_stockflow.period_days(start, end)[source]¶

Inclusive length of a period in days.

Return type:: float

morie.mrm_stockflow.stay_summary(days_per_person, conf_level=0.95)[source]¶

Distribution of stay lengths, with a t interval on the mean.

Parameters:: conf_level (float)
Return type:: StaySummary

morie.mrm_stockflow.stock_flow(days, people, period=None, t=365, exposure=None, per=100000, baseline='first')[source]¶

The stock and the flow side by side, with the exact decomposition.

baseline is a reporting decision, not a detail: “first” compares every period against the start of the window, “previous” against the period before it.

Parameters:

t (float)
per (float)
baseline (str)

Return type:

StockFlowResult

morie.mrm_stockflow.mrm_otis_stock_flow(person_days, placements=None, totals=None, year_col='EndFiscalYear', id_col='UniqueIndividual_ID', days_col='TotalAggregatedDays_Segregation', consecutive_col='NumberConsecutiveDays_Segregation', placement_col='Number_Of_Placements', totals_col='NumberIndividuals_Segregation', t=365, exposure=None, per=100000)[source]¶

Stock and flow across the OTIS strata, reconciled.

The same confinement appears at three strata in the OTIS release and they do not automatically agree:

PERSON (b02): one row per individual per year, carrying the days that person served. This is the additive quantity, and the one the measures need.
PLACEMENT (b01): several rows per individual, carrying the consecutive days of a SPELL. A spell length is not an additive share of the year – summing it as though it were understates the published total by about a third – so it is reported for reconciliation, never used as the numerator.
AGGREGATE (c01): the yearly totals as stated.

Reconciling them is the point, so both disagreements are returned rather than assumed away.

Parameters:

year_col (str)
id_col (str)
days_col (str)
consecutive_col (str)
placement_col (str)
totals_col (str)
t (float)
per (float)

Return type:

OtisStockFlowResult

Mandela Rules apples-to-apples spectrum on OTIS b01.

Computes the full grid of provincial Mandela-classified rates across four denominator conventions × three meaningful-contact proxies = 12 cells per fiscal year + a pooled row. Lets a downstream user pick the row that matches their preferred operationalisation, and lets the MRM paper Table 2 reproduce its spectrum from a single function call.

Denominator conventions (per mrm_classify_mandela semantics):

row - per-placement rate (b01 row count)
individual_any - share of within-year individuals with any
placement satisfying the criterion
individual_cumulative- share of within-year individuals whose
cumulative within-year segregation days exceeds the threshold
c11_aggregate - the duration-band aggregate from c11
(placeholder; requires the c11 file path)

Meaningful-contact proxies (Rule 44, derived from b01 alert flags):

none - Rule 43 only, no contact proxy applied
any_alert - Rule 43 ∩ (any of MentalHealth/SuicideRisk/SuicideWatch
alert active); these placements receive STAFF contact from medical or watch staff, so this is the LOOSER (less-restrictive) contact-failure proxy
no_alert - Rule 43 ∩ (no alert active); these placements
receive NO recorded staff contact and are the STRICTEST contact-failure proxy

The function returns a tidy long-format DataFrame with columns:

year, denominator, contact_proxy, n_eligible, n_mandela, rate, pct

so the caller can pivot or filter at will.

Reference for the Rule 43 + Rule 44 definitions:

United Nations General Assembly (2015). The Nelson Mandela Rules, A/RES/70/175.

Used by:

papers/mrm-formulations-paper §5.3 Table 2 (apples-to-apples spectrum row Federal SIU vs Ontario 2023/2024/2025)
papers/morie-empirical-paper §6 verification of all 12 cells.

morie.mrm_mandela_spectrum.mrm_otis_mandela_spectrum(data, *, duration_col='NumberConsecutiveDays_Segregation', year_col='EndFiscalYear', id_col='UniqueIndividual_ID', threshold_days=15, alert_cols=('MentalHealth_Alert', 'SuicideRisk_Alert', 'SuicideWatch_Alert'), contact_proxies=('none', 'any_alert', 'no_alert'), denominators=('row', 'individual_any', 'individual_cumulative'), c11_data=None)[source]¶

Compute the full 4 × 3 Mandela spectrum on OTIS b01.

Parameters:

data (DataFrame) – OTIS b01 frame (placement-level).
duration_col (str) – b01 column names.
year_col (str) – b01 column names.
id_col (str) – b01 column names.
threshold_days (int) – Rule 43 duration threshold (UN: 15 days).
alert_cols (Iterable[str]) – three b01 alert columns; “Yes” is treated as alert-active.
contact_proxies (Iterable[str]) – subset of {“none”, “any_alert”, “no_alert”}.
denominators (Iterable[str]) – subset of {“row”, “individual_any”, “individual_cumulative”, “c11_aggregate”}. The “c11_aggregate” mode requires c11_data to be supplied and is skipped silently otherwise.
c11_data (DataFrame | None) – optional c11 aggregate frame. When supplied, the c11_aggregate denominator joins per-individual cumulative duration from c11.

Returns:

one row per (year, denominator, contact_proxy) combination plus the pooled row, with columns

year, denominator, contact_proxy, n_eligible, n_mandela, rate, pct.

Return type:

Tidy DataFrame

Examples

Four placements over two years; in 2023 one of the two runs past 15 days.

>>> from morie.fn import _frame_core as pd
>>> b01 = pd.DataFrame({
...     "UniqueIndividual_ID": [1, 2, 3, 4],
...     "EndFiscalYear": [2023, 2023, 2024, 2024],
...     "NumberConsecutiveDays_Segregation": [20, 3, 16, 40],
...     "MentalHealth_Alert": ["Yes", "No", "No", "No"],
...     "SuicideRisk_Alert": ["No"] * 4,
...     "SuicideWatch_Alert": ["No"] * 4,
... })
>>> spectrum = mrm_otis_mandela_spectrum(b01)
>>> row = spectrum.to_dict("records")[0]
>>> row["year"], row["denominator"], row["contact_proxy"], row["pct"]
('2023', 'row', 'none', 50.0)
>>> spectrum.pivot_table(index=["year", "denominator"],
...                      columns="contact_proxy", values="pct").shape
(9, 3)

On the real data, pd.read_csv the b01 file from the OTIS release and pass it the same way.

Kulldorff space-time scan statistic on TPS event data.

Implements Kulldorff’s 1995 space-time scan statistic with a Monte-Carlo permutation test for significance. Used for §7.7 of the MRM empirical paper.

The scan iterates over (centre, radius, time-window) tuples, computing the Poisson log-likelihood-ratio statistic against H_0 (events uniformly distributed in space and time). The maximum LRT is the test statistic; Monte-Carlo permutations of event timestamps generate the null distribution.

Reference:: Kulldorff, M. (1997). A spatial scan statistic. Communications in Statistics: Theory and Methods, 26(6), 1481-1496.

class morie.mrm_kulldorff.ScanCluster(center_lat, center_lon, radius_km, t_start, t_end, n_observed, n_expected, relative_risk, log_lrt, p_value)[source]¶

Bases: object

A single space-time cluster returned by the Kulldorff scan.

Parameters:

center_lat (float)
center_lon (float)
radius_km (float)
t_start (datetime)
t_end (datetime)
n_observed (int)
n_expected (float)
relative_risk (float)
log_lrt (float)
p_value (float)

center_lat: float¶

center_lon: float¶

radius_km: float¶

t_start: datetime¶

t_end: datetime¶

n_observed: int¶

n_expected: float¶

relative_risk: float¶

log_lrt: float¶

p_value: float¶

morie.mrm_kulldorff.mrm_tps_kulldorff_scan(data, *, date_col='OCC_DATE', lat_col='LAT_WGS84', lon_col='LONG_WGS84', radii_km=(1.0, 2.0, 3.0, 5.0, 8.0), window_years=4.0, n_centers=60, n_permutations=199, n_top_clusters=1, seed=42)[source]¶

Run a 3-d (lat, lon, time) Kulldorff scan with MC inference.

Parameters:

data (DataFrame) – DataFrame with date_col, lat_col, lon_col.
radii_km (Iterable[float]) – Candidate cylinder radii in km.
window_years (float) – Length of the time cylinder in years.
n_centers (int) – Number of random candidate centres (downsampled from the event lat/long population to keep cost bounded).
n_permutations (int) – Monte-Carlo permutations of the timestamp assignment for the null distribution.
n_top_clusters (int) – Maximum number of non-overlapping clusters to return (sorted by descending LRT).
seed (int) – Random seed for both the centre sub-sample and the MC permutation.
date_col (str)
lat_col (str)
lon_col (str)

Returns:

List of ScanCluster objects, length <= n_top_clusters, sorted by descending log_lrt and pruned to be non-overlapping (no centre within the radius of a higher-LRT cluster).

Return type:

list[ScanCluster]

LISA (Local Indicators of Spatial Association) + per-year polygon Moran’s I on TPS-style geographic crime data.

Two callables:

mrm_tps_lisa(data, year=…) – Local Moran’s I per polygon centroid, with 999-permutation Monte-Carlo significance. Returns per-polygon I_i, lag z-score, quadrant (HH/HL/LH/LL), and p-value. Computes the global Moran’s I as a by-product.
mrm_tps_polygon_moran_per_year(data, year_cols=…) – convenience wrapper that loops mrm_tps_lisa over a set of per-year count columns (e.g. ASSAULT_2014 … ASSAULT_2024) and returns the global Moran’s I time series. Used by the empirical paper §7.11 to document the declining spatial concentration of assault 2014-2024.

Both functions take a GeoJSON-like polygon frame (or a DataFrame with lat/lon centroid columns + per-feature counts) and compute k-nearest-neighbour spatial weights (default k=6).

References

Anselin, L. (1995). Local indicators of spatial association – LISA.: Geographical Analysis, 27(2), 93-115.
Anselin, L. (2010). Thirty years of spatial econometrics.: Papers in Regional Science, 89(1), 3-25.

class morie.mrm_lisa.LISAResult(n_polygons, global_moran_I, permutations, knn_k, table, quadrants_all, quadrants_significant_p05, n_significant_p05)[source]¶

Bases: object

Output of mrm_tps_lisa.

Parameters:

n_polygons (int)
global_moran_I (float)
permutations (int)
knn_k (int)
table (DataFrame)
quadrants_all (dict)
quadrants_significant_p05 (dict)
n_significant_p05 (int)

n_polygons: int¶

global_moran_I: float¶

permutations: int¶

knn_k: int¶

table: DataFrame¶

quadrants_all: dict¶

quadrants_significant_p05: dict¶

n_significant_p05: int¶

morie.mrm_lisa.mrm_tps_lisa(data, *, count_col, lat_col='lat', lon_col='lon', id_col=None, k=6, n_permutations=999, seed=42)[source]¶

Local Moran’s I + quadrant + significance for a polygon surface.

Parameters:

data (DataFrame) – DataFrame with one row per polygon. Must have lat/lon centroid columns + a count_col with per-polygon counts.
count_col (str) – per-polygon count column (e.g. “ASSAULT_2024”).
lat_col (str) – WGS84 centroid columns.
lon_col (str) – WGS84 centroid columns.
id_col (str | None) – optional polygon-id column (passed through to output).
k (int) – k-NN spatial-weights neighbourhood (default 6).
n_permutations (int) – MC permutations for significance. 999 is the spatial-statistics convention. Local p-values use conditional randomization (Anselin 1995): z_i stays fixed and its neighbours are drawn from the other n - 1 values; folded pseudo p = (min(#>=, #<=) + 1) / (R + 1), as GeoDa and spdep::localmoran_perm.
seed (int) – RNG seed for reproducibility.

Returns:

LISAResult with per-polygon table and quadrant summaries.

Return type:

LISAResult

morie.mrm_lisa.mrm_tps_polygon_moran_per_year(data, *, year_cols, lat_col='lat', lon_col='lon', k=6, n_permutations=999, seed=42)[source]¶

Per-year global Moran’s I time series across a polygon surface.

Parameters:

data (DataFrame) – DataFrame with one row per polygon + one count column per year.
year_cols (Sequence[str]) – list of per-year count columns (in year order).
lat_col (str) – WGS84 centroid columns.
lon_col (str) – WGS84 centroid columns.
k (int) – k-NN spatial-weights neighbourhood.
n_permutations (int) – MC permutations for each year’s significance.
seed (int) – RNG seed.

Returns:

year, n_events, moran_I, global_p_value (one-sided permutation test of positive autocorrelation, as spdep::moran.mc).

Return type:

DataFrame with one row per year

Federal SIU, crime feeds and forensics¶

morie.siu – automated mining of Ontario Special Investigations Unit director’s reports.

Reads from siu.on.ca (publicly published reports + paired news releases), parses each report HTML into a structured row, writes a unified SIU.csv keyed on case_number (the SIU’s canonical case ID, format YY-XXX-NNN).

URL invariant (see SIU_PLAN_20260506.md):

drid (URL locator) ≠ nrid (news-release locator) case_number = canonical join key

Public API:

scrape_drid(drid, *, client=None, cache=True) -> dict: Fetch + parse a single director’s report by drid.
scrape_range(drid_min, drid_max, **kw) -> Iterator[dict]: Polite range scrape with concurrency limits.
parse_html(html, *, drid=None, source_url=None) -> dict: Pure parser – no network – turns one HTML page into a row.

write_csv(rows, path) -> None write_jsonl(rows, path) -> None # for narrative_full bodies SIU_COLUMNS # 45-col canonical schema

Examples

>>> from morie.siu import html_to_text, to_iso_date
>>> to_iso_date("2nd February 2018")
'2018-02-02'
>>> html_to_text("<p>Qu&eacute;bec &amp; Ontario</p>").strip()
'Québec & Ontario'

morie.siu.scrape_drid(drid, *, client=None, cache=True, cache_dir=None, fetch_news=True)[source]¶

Fetch + parse one director’s report. Returns a SIU.csv row dict.

On 404, returns a row with case_number=None and the source_url set – the caller decides whether to drop it.

If fetch_news=True (default) and the report links to a news release via news_template.php?nrid=…, that page is also fetched and its fields are merged into the row (news_release_title, _date_iso, _raw, _summary, directors_name).

Parameters:

drid (int)
client (Client | None)
cache (bool)
cache_dir (Path | None)
fetch_news (bool)

Return type:

morie.siu.scrape_range(drid_min, drid_max, *, cache=True, cache_dir=None, delay_s=0.25, progress=True)[source]¶

Yield rows for every drid in [drid_min, drid_max] inclusive.

Sequential, polite (sleeps delay_s between fetches that touch the network – cache hits don’t sleep). Phase-2a default; Phase-2c will introduce a concurrent variant behind the same name.

Parameters:

drid_min (int)
drid_max (int)
cache (bool)
cache_dir (Path | None)
delay_s (float)
progress (bool)

Return type:

Iterator[dict]

morie.siu.parse_html(html, *, drid=None, source_url=None)[source]¶

Turn a director’s-report HTML page into a SIU.csv row dict.

Parameters:

html (str) – raw response body.
drid (int | None) – optional integer drid from the request URL – useful when the page itself doesn’t echo it.
source_url (str | None) – optional canonical URL – used to derive drid/nrid and recorded as source_url_report.

Returns:

dict with every SIU_COLUMNS key (Nones for unfound fields).

Return type:

morie.siu.parse_news_html(html, *, nrid=None, source_url=None)[source]¶

Parse a SIU news-release page into a partial row dict.

News-release pages live at news_template.php?nrid=<N> and have a different layout than the director’s reports – single headline, short summary paragraph, signed-by-Director line. We extract:

news_release_title

news_release_date_iso / _raw (the “Mississauga, ON (Date)” line)

news_release_summary (1-2 paragraphs)

case_number (cross-validation with drid scrape)

nrid (URL locator)

directors_name (e.g. “Tony Loparco”)

The caller (typically scrape_drid after finding the nrid link) merges these into the main row dict by case_number match.

Parameters:

html (str)
nrid (int | None)
source_url (str | None)

Return type:

morie.siu.write_csv(rows, path, *, exclude_narrative=True)[source]¶

Write rows in canonical column order. Returns the count written.

By default, narrative_full is excluded from the CSV (it can be >50KB per row and bloats the file) and written separately to JSONL via write_jsonl. Pass exclude_narrative=False to include it.

Parameters:

rows (Iterable[dict])
path (Path | str)
exclude_narrative (bool)

Return type:

morie.siu.write_jsonl(rows, path, *, only_keys=('case_number', 'drid', 'narrative_full'))[source]¶

Write rows as one JSON object per line. By default only keeps the primary key + drid + narrative_full to keep the file lean.

Parameters:

rows (Iterable[dict])
path (Path | str)
only_keys (tuple)

Return type:

morie.siu.resolve_subject_officials(report_text)[source]¶

Rule-engine SO count for one report’s text. (count|None, reason).

None means “needs a human/panel read” – never a guess.

Parameters:: report_text (str)
Return type:: tuple[int | None, str]

morie.siu.siu_panel(html, mode=4, readers=None, auditors=None, reader_concurrency=3, granularity='all_fields', host=None, timeout=300, api='', base='', key='', chat=None)[source]¶

Mixture-of-Agents reading panel for SIU reports not yet in the corpus.

Modes mirror the panel that built the reviewed corpus: 1 = one reader; 2 = reader + auditor; 3 = two readers + auditor; 4 = three readers + auditor (default). Explicit readers/auditors override the mode. Readers run concurrently (hard barrier before the auditor); auditors chain sequentially, each seeing its predecessors’ verdicts.

Models come from ANY backend (morie.siu.llm): api ("ollama" or "openai" for llama.cpp, vLLM, LM Studio, LocalAI, TGI, OpenRouter …), base (host is kept as an alias), key, or your own chat(model, prompt) -> str. Empty values fall back to the environment (MORIE_LLM_API/BASE/KEY, OLLAMA_HOST, OPENAI_BASE_URL, …); the default model is $MORIE_LLM_MODEL, $OLLAMA_MODEL or the first model the server lists. Nothing is hardcoded.

Parameters:

html (str | int)
mode (int)
readers (list[str] | None)
auditors (list[str] | None)
reader_concurrency (int)
granularity (str)
host (str | None)
timeout (float)
api (str)
base (str)
key (str)

Return type:

morie.siu.siu_reports(update=False, cache_dir=None)[source]¶

Return the panel-reviewed SIU corpus as a pandas DataFrame.

The verified corpus is the single source of truth: reviewed rows carry panel_reviewed == True and their number_of_subject_officials values were verified by the multi-agent panel. update=True only re-downloads the snapshot (same SHA-256 pin); it never re-derives reviewed values.

Parameters:

update (bool)
cache_dir (str | Path | None)

morie.siu.siu_resolve_so(text=None, drid=None)[source]¶

Subject-official count: verified corpus first, rules second.

Exactly one of text / drid must be given. A drid found in the panel-reviewed corpus returns the verified value (reason "panel-reviewed corpus (verified)"); anything else goes through the zero-wrong rule engine.

Parameters:

text (str | None)
drid (int | None)

Return type:

morie.siu.strip_boilerplate(text)[source]¶

Normalise NBSPs and remove the privacy + glossary boilerplate.

Parameters:: text (str)
Return type:: str

morie.siu.html_to_text(html)[source]¶

Strip tags, scripts and entities from report HTML, keeping line structure.

Examples

>>> html_to_text("<p>The&nbsp;Team</p><p>SO #1</p>")
' The Team\nSO #1\n'

Parameters:: html (str)
Return type:: str

morie.siu.parse_report_html(html)[source]¶

HTML -> fields in one call (html_to_text() then parse_report_text()).

Examples

>>> f = parse_report_html("<p>The Investigation</p><p>It happened in the City of Barrie, a 34-year-old man ...</p>")
>>> f["age_affected"], f["sex_gender_affected"], f["location_of_call"]
('34', 'man', 'City of Barrie')

Parameters:: html (str)
Return type:: dict[str, str]

morie.siu.parse_report_text(text)[source]¶

Parse plain report text into the 16 schema fields plus _language ("" when not stated).

Parameters:: text (str)
Return type:: dict[str, str]

morie.siu.siu_audit_panel(report_text, parsed=None, mode=4, readers=None, auditors=None, num_readers=0, num_auditors=0, reader_concurrency=0, auditor_sequential=True, reader_granularity='all', auditor_granularity='all', health_check=True, api='', base='', key='', timeout=300.0, temperature=0.0, chat=None)[source]¶

Run the Mixture-of-Agents audit panel on one SIU report with any model.

mode 1 = one reader, no auditor; 2 = 1 reader + auditor; 3 = 2 readers + auditor; 4 = 3 readers + auditor. Readers (cycled over readers) read the full report and answer every field with a quote and confidence; after a hard barrier the auditors form a review chain, each auditing the previous stage. Readers run concurrently up to reader_concurrency (0 = all) against an HTTP backend; with your own chat callable they run one after another. *_granularity="per-field" re-reads the whole report once per field. Models returning an empty reply are dropped first. The backend is api/base/key (see morie.siu.llm) or chat.

Returns {"fields": dict, "json": str}.

Examples

>>> def fake(model, prompt):
...     if prompt.startswith("Reply with"):
...         return "OK"
...     if "You are the AUDITOR" in prompt:
...         return '{"police_service": "Barrie Police Service"}'
...     return '{"police_service": {"value": "Barrie", "quote": "Barrie", "confidence": "high"}}'
>>> siu_audit_panel("The Barrie Police Service ...", {}, mode=2, readers=["r1"], chat=fake)["json"]
'{"police_service":"Barrie Police Service"}'

Parameters:

report_text (str)
parsed (dict | str | None)
mode (int)
num_readers (int)
num_auditors (int)
reader_concurrency (int)
auditor_sequential (bool)
reader_granularity (str)
auditor_granularity (str)
health_check (bool)
api (str)
base (str)
key (str)
timeout (float)
temperature (float)

Return type:

morie.siu.to_iso_date(human)[source]¶

"January 5, 2023" -> "2023-01-05" ("" if unparseable).

Examples

>>> to_iso_date("January 5, 2023"), to_iso_date("2023-01-05"), to_iso_date("soon")
('2023-01-05', '2023-01-05', '')

Parameters:: human (str)
Return type:: str

On-demand scraper for the Ontario Special Investigations Unit (SIU).

The SIU publishes Director’s Reports at https://siu.on.ca/en/directors_reports.php. Each case has a public PDF or HTML report listing the incident date, the notifying police service, and the Director’s decision. This module scrapes the index page(s) and per-case detail pages on demand, caching results as a single CSV in the morie cache directory.

Distribution policy (2026-05): the scraped corpus is NOT shipped with the package because the legal status of redistributing aggregated copies of publicly-posted oversight reports is unsettled. Each user runs the scraper themselves, which is unambiguously fair use.

The scraper is conservative: a 2-second delay between requests, retries on 5xx, respects robots.txt. Run fetch_siu_cases(year=…) to populate the cache, then morie_load_dataset(“siu”) returns a tidy data.frame.

morie.siu_fetch.fetch_siu_cases(*, years=None, cache_dir='~/.cache/morie/siu', overwrite=False, progress=True)[source]¶

Scrape SIU Director’s Reports into a single CSV.

Parameters:

years (Iterable[int] | None) – Iterable of fiscal years to fetch (default = all years indexed on the SIU site).
cache_dir (str | Path) – Directory for the cache CSV.
overwrite (bool) – If False and SIU.csv already exists, return it.
progress (bool) – Print one line per scraped page when True.

Returns:

Path to the populated SIU.csv.

Raises:

urllib.error.URLError on persistent network failure. –

Return type:

morie.siu_fetch.fetch_siu_dataframe(**kwargs)[source]¶

Scrape SIU Director’s Reports and return them as a DataFrame.

Thin wrapper over fetch_siu_cases() (which returns the CSV path); used as a morie.data.DATASET_CATALOG fetcher, whose dispatch expects a DataFrame.

morie.siu_fetch.siu_cache_path(cache_dir='~/.cache/morie/siu')[source]¶

Path of the cached SIU director’s-report table, creating its directory.

Parameters:: cache_dir (str | Path) – directory for the cache (~ is expanded).
Returns:: <cache_dir>/SIU.csv; the file itself need not exist yet.
Return type:: Path

Examples

>>> import tempfile
>>> siu_cache_path(tempfile.mkdtemp()).name
'SIU.csv'

TPS (Toronto Police Service) Open Data ArcGIS REST fetcher.

The Toronto Police Service publishes per-category crime-incident GeoJSON through ArcGIS Online at services.arcgis.com. Each category has a stable service layer URL; this module pages through the /query endpoint, pulls all features, and writes a tidy CSV to the morie cache directory.

Public categories (as of 2026-05): Assault, AutoTheft, BicycleTheft, BreakAndEnter, Homicides, Robbery, ShootingAndFirearmDiscarges, TheftFromMV, TheftOver, HateCrimes, IntimatePartnerViolence, CommunitySafetyIndicators, NeighbourhoodCrimeRates.

morie.tps_fetch.fetch_tps_category(category, *, cache_dir='~/.cache/morie/tps', where='1=1', overwrite=False, max_records_per_page=2000)[source]¶

Fetch a TPS category as a CSV, paging through ArcGIS until exhausted.

Parameters:

category (str) – One of list_tps_categories().
cache_dir (str | Path) – Directory to write the CSV into.
where (str) – SQL where clause for the ArcGIS query (default “1=1”).
overwrite (bool) – If False and the output file exists, return it without re-downloading.
max_records_per_page (int) – ArcGIS pagination size (server caps at 2000).

Returns:

Path to the CSV file in cache_dir.

Raises:

ValueError – if category is unknown.
urllib.error.URLError – on network failure.

Return type:

morie.tps_fetch.fetch_tps_dataframe(category, **kwargs)[source]¶

Fetch a TPS category and return it as a DataFrame.

Thin wrapper over fetch_tps_category() (which returns the CSV path); used as a morie.data.DATASET_CATALOG fetcher, whose dispatch expects a DataFrame.

Parameters:: category (str)

morie.tps_fetch.list_tps_categories()[source]¶

Return the list of TPS categories known to this fetcher.

Return type:: list[str]

Numba-JIT’d negative log-likelihood paths for tps_hawkes_advanced.

Reason this exists: tps_hawkes_advanced._neg_loglik_general did a pure- Python O(n²) inner loop over events; at n=21,160 the optimizer spent hours per fit. This module replaces that hot loop with kernel code plus the well-known O(n) recursive update for the exponential kernel, so a 21k-event 2019-contiguous Markovian fit lands in seconds and the Weibull×sinusoidal non-Markovian fit lands in minutes.

Public:

has_jit_path(kernel, baseline) -> bool neg_loglik_jit(theta, t, T, kernel, baseline) -> float

The caller (_neg_loglik_general) checks has_jit_path first and falls through to the original NumPy implementation only for parameter combinations no JIT path covers.

morie.tps_hawkes_jit.njit(*args, **kwargs)[source]¶

Identity stand-in for numba.njit: returns the function unchanged, with or without arguments.

Examples

>>> njit(len) is len, njit(cache=True)(len) is len
(True, True)

morie.tps_hawkes_jit.has_jit_path(kernel, baseline)[source]¶

True if a compiled likelihood exists for this kernel and baseline.

Examples

>>> has_jit_path("no-such-kernel", "constant")
False

Parameters:

kernel (str)
baseline (str)

Return type:

morie.tps_hawkes_jit.neg_loglik_jit(theta, t, T, kernel, baseline)[source]¶

JIT-routed negative log-likelihood. Caller must check has_jit_path().

Parameters:

theta (marr)
t (marr)
T (float)
kernel (str)
baseline (str)

Return type:

morie.tps_hawkes_jit.hawkes_loglik_custom(t, T, nu, eta, kernel, kernel_integral)[source]¶

Hawkes negative log-likelihood with a user-supplied triggering kernel.

The “bring your own kernel” path: supply your own triggering kernel when the built-in exponential / Weibull / Lomax / gamma kernels misspecify your process.

Parameters:

t (array) – Sorted event times.
T (float) – Observation horizon.
nu (float) – Constant baseline intensity (must be > 0).
eta (float) – Branching ratio.
kernel (callable) – The triggering kernel g(dt). A plain Python callable runs in the pure-Python O(n^2) engine; a pre-built native callback (an object exposing .address, the C-function-pointer convention) runs GIL-free inside the C++ engine.
kernel_integral (callable) – G(u) = the integral of g over [0, u] – the compensator term.

Returns:

The negative log-likelihood (a large sentinel if infeasible).

Return type:

Examples

>>> import math
>>> def g(u):
...     return 1.5 * math.exp(-1.5 * u)
>>> def G(u):
...     return 1.0 - math.exp(-1.5 * u)
>>> hawkes_loglik_custom(t, T, nu=0.4, eta=0.3,
...                      kernel=g, kernel_integral=G)

morie.tps_hawkes_jit.soe_fit_lomax(alpha, c, horizon, *, tol=1e-07, m_max=256)[source]¶

Sum-of-exponentials representation of the Lomax triggering kernel, accurate over the lag range u in [0, horizon].

The Lomax density g(u) = (alpha-1)/c * (1 + u/c)**(-alpha) is completely monotone, so by Bernstein’s theorem it is exactly the Laplace transform of a non-negative mixing measure:

g(u) = integral_0^inf rho(s) e**(-s u) ds, rho(s) = (alpha-1) c**(alpha-1) / Gamma(alpha)

s**(alpha-1) e**(-s c).

A finite SoE has an exponential tail and so can never match the power-law tail as u -> inf; but the Hawkes log-likelihood only evaluates inter-event lags up to the observation horizon, so the quadrature is targeted at [0, horizon] alone. Substituting s = exp(v) makes the integrand analytic and doubly-decaying in v, for which the trapezoidal rule converges geometrically (Trefethen & Weideman, SIAM Review 2014): the node count grows only logarithmically in both the timescale range and the inverse tolerance.

The decay-rate grid spans s_min ~ 1/horizon (the slowest mode resolvable over the data window) up to s_max ~ 40/c (past which rho is negligible – e**(-s c) has decayed). M is not hand-tuned: it is doubled until the measured maximum relative error of the SoE against the exact kernel on [0, horizon] falls to tol (or m_max is hit).

Returns (w, beta, max_rel_err) – the float64 arrays for _core.hawkes_ll_soe plus the achieved, verified error bound.

morie.tps_hawkes_jit.soe_fit_gamma_tail(alpha, beta, u_split, *, span=20.0, n_samples=240)[source]¶

Matrix-pencil SoE fit of the gamma kernel’s tail, for u >= u_split.

The gamma triggering kernel g(u) = (beta**alpha / Gamma(alpha)) * u**(alpha-1) * exp(-beta*u) is not completely monotone for shape alpha > 1, so it has no Bernstein quadrature; the tail is fitted empirically instead. The kernel is sampled uniformly on [u_split, u_split + span/beta] and the matrix pencil (_soe_fit_matrix_pencil) recovers the exponential modes.

The fit is returned in the SHIFTED convention – the modes describe g(u_split + s) = sum_m w[m] * exp(-beta_soe[m] * s) for s >= 0 – which the hybrid engine’s graduation recursion consumes directly, with no exp(beta*u_split) rescaling, so the weights stay well scaled.

Modes are complex: real poles plus, for alpha above ~2.5, complex-conjugate pairs (an accurate damped-oscillatory mode, not an artefact – see hawkes_ll_soe_cplx). Every Re(beta_soe) must be > 0 (decaying); a non-decaying mode raises ValueError.

Returns (w, beta_soe, max_rel_err) – complex128 arrays plus the measured relative error of the SoE against the exact tail.

morie.tps_hawkes_jit.hawkes_em(t, T, kernel_kind, baseline_kind, x0, bounds, max_iter=100, tol=0.0001, inner_iter=8)[source]¶

EM for the Hawkes MLE (Veen & Schoenberg 2008): the branching structure as missing data.

E-step: the intensity at each event (compiled core). M-step, with p_i0 = nu_i/lambda_i and p_ij = eta g(t_i - t_j)/lambda_i: the baseline maximises sum p_i0 log nu_i - int nu (closed form for a constant one); the kernel maximises sum p_ij log g(.; psi) - P log sum_j G(T - t_j; psi), with eta = P / sum_j G (P = sum p_ij), its weights recomputed in the pair loop. Each step does not lower the likelihood; the same maximum as the direct fit. Returns (theta, nll, iterations).

Generalised EM: each M-step takes at most inner_iter quasi-Newton steps (every one of them an O(n^2) pair pass, which is where a full inner optimisation spent 20-140 s on 1,083 events), since the E-step is recomputed next round anyway, and it stops once the likelihood moves by less than tol (relative): fit_hawkes_general() then seeds the direct optimiser from EM’s iterate, so the answer is the direct method’s optimum at its precision.

morie.tps_hawkes_jit.hawkes_inar(t, T, kernel_kind, baseline_kind, bounds, delta=None, support=None)[source]¶: Kirchner’s (2017, Quant. Finance 17:571) INAR(p) estimator: binned counts X_k (width delta) regressed on their p lags by conditional least squares (here the equivalent Yule-Walker equations from the sample autocovariances), so alpha_0 = nu delta and alpha_l ~ eta delta g(l delta); the parametric kernel is then fitted to the alpha_l by least squares. A different estimator from the MLE (consistent as delta -> 0 and p delta -> infinity), fast, and a starting point for it. Constant baseline only (it assumes stationarity).

Spatiotemporal (self-exciting) Hawkes point process.

Models the conditional intensity of clustered space-time events – near-miss airspace violations, crime, seismic aftershocks – where each event transiently raises the risk of further events nearby and soon after. The purely temporal Hawkes lives in morie.hawkes; this adds the spatial dimension:

lambda(t,x,y) = mu + sum_{t_i<t} alpha*beta*exp(-beta(t-t_i))

(1/(2*pi*sigma^2)) * exp(-((x-x_i)^2+(y-y_i)^2)/(2*sigma^2))

with background rate mu (per unit area-time), branching ratio alpha (alpha<1 for stability), temporal decay beta, Gaussian spatial spread sigma. The triggering kernel integrates to alpha in time and 1 in space (Reinhart 2018). R parity: rmorie R/hawkes_spatial.R (morie_hawkes_st_*).

References

Reinhart A (2018). A review of self-exciting spatio-temporal point processes. Statistical Science 33(3), 299-318.

morie.hawkes_spatial.hawkes_st_intensity(events, t_q, x_q, y_q, params)[source]¶

Conditional intensity lambda(t_q, x_q, y_q). R parity: morie_hawkes_st_intensity.

Examples

>>> events = {"t": [0.5, 1.0, 1.5], "x": [0.1, 0.2, 0.15], "y": [0.1, 0.1, 0.2]}
>>> round(hawkes_st_intensity(events, 2.0, 0.15, 0.15, {"mu": 0.5, "alpha": 0.6, "beta": 1.0, "sigma": 0.1}), 4)
10.0067

Parameters:

t_q (float)
x_q (float)
y_q (float)
params (dict[str, float])

Return type:

morie.hawkes_spatial.hawkes_st_loglik(events, params, end_time=None, area=1.0)[source]¶

Log-likelihood sum(log lambda) - compensator. R parity: morie_hawkes_st_loglik.

Compensator = mu*T*A + sum_i alpha(1 - exp(-beta(T - t_i))); the spatial kernel integrates to 1 over the plane (interior-region boundary effects neglected).

Parameters:

params (dict[str, float])
end_time (float | None)
area (float)

Return type:

morie.hawkes_spatial.hawkes_st_simulate(params, end_time, region, seed=None, max_events=100000)[source]¶

Exact branching (immigrant-offspring) simulation. Requires alpha < 1.

R parity: morie_hawkes_st_simulate. region = (xmin, xmax, ymin, ymax). Returns a DataFrame with t, x, y, gen (0 = immigrant), sorted by t.

Parameters:

params (dict[str, float])
end_time (float)
seed (int | None)
max_events (int)

Return type:

DataFrame

morie.hawkes_spatial.hawkes_st_fit(events, end_time=None, area=1.0, start=None)[source]¶

Maximum-likelihood fit over (mu, alpha, beta, sigma) on the log scale.

R parity: morie_hawkes_st_fit. Check fit['params']['alpha'] < 1 for a stable fit. Note: the background/branching split (mu vs alpha) is weakly identified at small samples and the likelihood neglects spatial edge effects, so interpret mu vs alpha cautiously and prefer large regions relative to sigma and long records.

Parameters:

end_time (float | None)
area (float)
start (dict[str, float] | None)

Return type:

morie.cpd_all_analyze – Chicago Police Department analysis aggregator.

Python parity with R’s morie_cpd_all_analyses. Loads Chicago crime + arrests data (bundled samples offline by default) and runs the applicable descriptive, predictive-policing, and fairness surfaces, each emitting a RichResult. Mirrors morie.otis_all_analyze.

analyze_<name>(…) -> RichResult analyze_all(*, out_dir=None) -> dict[str, RichResult]

morie.cpd_all_analyze.analyze_crime_by_type(crime_df=None)[source]¶

Incident volume by Chicago primary crime type.

Parameters:: crime_df (DataFrame | None)
Return type:: RichResult

morie.cpd_all_analyze.analyze_arrests_by_area(crime_df=None)[source]¶

Predpol area aggregation: arrest concentration by community area.

Parameters:: crime_df (DataFrame | None)
Return type:: RichResult

morie.cpd_all_analyze.analyze_temporal(crime_df=None)[source]¶

Incident counts by year.

Parameters:: crime_df (DataFrame | None)
Return type:: RichResult

morie.cpd_all_analyze.analyze_arrest_race_disparity(arrests_df=None)[source]¶

Disparate-impact of arrest representation across race categories.

Parameters:: arrests_df (DataFrame | None)
Return type:: RichResult

morie.cpd_all_analyze.analyze_all(*, out_dir=None)[source]¶

Run every CPD surface; returns a name -> RichResult mapping.

Parameters:: out_dir (str | None)
Return type:: dict[str, RichResult]

morie.nypd_all_analyze – New York Police Department analysis aggregator.

Python parity with R’s morie_nypd_all_analyses. Loads NYPD arrests + complaint data (bundled samples offline by default) and runs descriptive and fairness surfaces. NYPD arrests carry perp_race and law_cat_cd (felony/misdemeanour), so a felony-charge disparate-impact by race is computable directly from the data.

analyze_<name>(…) -> RichResult analyze_all(*, out_dir=None) -> dict[str, RichResult]

morie.nypd_all_analyze.analyze_arrests_by_offense(arrests_df=None)[source]¶

Arrest volume by NYPD offense description.

Parameters:: arrests_df (DataFrame | None)
Return type:: RichResult

morie.nypd_all_analyze.analyze_arrests_by_boro(arrests_df=None)[source]¶

Arrest volume by borough.

Parameters:: arrests_df (DataFrame | None)
Return type:: RichResult

morie.nypd_all_analyze.analyze_felony_race_disparity(arrests_df=None)[source]¶

Disparate impact + demographic parity of felony (vs lesser) charging.

Parameters:: arrests_df (DataFrame | None)
Return type:: RichResult

morie.nypd_all_analyze.analyze_complaints_by_race(complaint_df=None)[source]¶

Complaint volume by recorded suspect race.

Parameters:: complaint_df (DataFrame | None)
Return type:: RichResult

morie.nypd_all_analyze.analyze_all(*, out_dir=None)[source]¶

Run every NYPD surface; returns a name -> RichResult mapping.

Parameters:: out_dir (str | None)
Return type:: dict[str, RichResult]

morie.run_crime_analysis – one-command end-to-end pipeline.

Usage:

from morie import run_crime_analysis
run_crime_analysis.full()                  # everything
run_crime_analysis.tps_only("Assault")     # one category
run_crime_analysis.otis_only()             # OTIS Goffmanian + churn

Reproduces the full paper-205 analysis surface against the canonical data/datasets/{OTIS,TPS}/ folders and writes outputs to data/manifest/outputs/{otis,tps,overlay,otis_churn,tps_stochastic, tps_spatial_advanced,figures}/. No further setup required.

Designed for users who want the paper-205 results “for free” on their own copy of the OTIS / TPS data.

morie.run_crime_analysis.otis_only(*, run_descriptive=True, run_churn=True, run_overlay=False)[source]¶

Run OTIS-only analyses: 28 dataset descriptive + 6 Goffmanian churn tests.

Parameters:

run_descriptive (bool)
run_churn (bool)
run_overlay (bool)

Return type:

morie.run_crime_analysis.tps_only(category='Assault', *, sample_rows=50000)[source]¶

Run TPS-only stack on one category.

Parameters:

category (str)
sample_rows (int | None)

Return type:

morie.run_crime_analysis.overlay()[source]¶

Pipeline step: run the OTIS x TPS overlay (otis_tps_overlay.analyze_all).

Return type:: dict[str, Any]

morie.run_crime_analysis.render_figures()[source]¶

Re-render all paper-205 figures from existing JSON outputs. Tries R/ggplot first, falls back to matplotlib if R unavailable.

Return type:: dict[str, str]

morie.run_crime_analysis.full(sample_rows=50000, tps_categories=None)[source]¶

Run everything end-to-end on the bundled OTIS+TPS data.

Wall-clock on a 2024 MacBook Air M2 / external VSR drive: ~5 min for the OTIS half + 30 s per TPS category.

Parameters:

sample_rows (int | None)
tps_categories (list[str] | None)

Return type:

morie.arsau_analyze — per-record-type ARSAU analysis pipelines.

Each public callable in this module loads one ARSAU dataset via morie.arsau_datasets.arsau_load_*() and chains the jurisdiction-agnostic MRM Use-of-Force primitives from morie.mrm_uof over it, producing a single morie.fn._richresult.RichResult that holds the loaded data, all sub-analyses, and a multi-paragraph natural-language summary.

The analyzers do not invent new statistical methods — they wire up the generic callables against the column names that the Ontario open-data release actually publishes. If the upstream schema changes, the generic callables in morie.mrm_uof will still work; the only patch needed will be the column-name constants below.

Public callables¶

arsau_analyze_main_records() (per year, 2023 / 2024)
arsau_analyze_individual_records() (per year, 2023 / 2024)
arsau_analyze_probe_cycle_records() (per year, 2023 / 2024)
arsau_analyze_weapon_records() (per year, 2023 / 2024; 2023 requires allow_invalid=True)
arsau_analyze_aggregate_summary() (2020-2022)
arsau_analyze_detailed_dataset() (2020-2022)

Every analyzer returns a RichResult whose payload contains the constituent sub-results under named keys, so callers can drill into specific tests without re-running the whole pipeline.

morie.arsau_analyze.arsau_analyze_main_records(year, *, language='en', data_dir=None)[source]¶

End-to-end analysis of the ARSAU main_records CSV for one year.

Chains:

morie.mrm_uof.mrm_uof_force_concentration() over PoliceService
morie.mrm_uof.mrm_uof_weapon_diversity() over IncidentType x PoliceService
morie.mrm_uof.mrm_uof_data_quality_audit() against the published CKAN sidecar (when present)

Region-locality is NOT meaningful for main_records — only the OPP_PoliceService_Region field is published, and it pairs one column with itself. See arsau_analyze_detailed_dataset() for the 2020-2022 layout that exposes more region columns.

Parameters:

year (str | int)
language (str)
data_dir (str | Path | None)

Return type:

morie.arsau_analyze.arsau_analyze_individual_records(year, *, language='en', data_dir=None, bootstrap_reps=0)[source]¶

End-to-end analysis of the ARSAU individual_records CSV for one year.

Chains demographic-disparity tests over Race, Gender, and AgeCategory against the IndivInjuries_PhysicalInjuries outcome column, plus a data-quality audit against the sidecar.

Parameters:

bootstrap_reps (int, default 0) – Forwarded to morie.mrm_uof.mrm_uof_demographic_disparity(). Set to e.g. 1000 to get percentile-bootstrap CIs on the risk ratios.
year (str | int)
language (str)
data_dir (str | Path | None)

Return type:

morie.arsau_analyze.arsau_analyze_probe_cycle_records(year, *, language='en', data_dir=None)[source]¶

Analysis of ARSAU probe_cycle_records (CEW telemetry).

The probe-cycle file is intentionally narrow (BatchFileName + Indiv_Index + a comma-separated cycle string). We compute the cycle-count distribution per incident and run a data-quality audit.

Parameters:

year (str | int)
language (str)
data_dir (str | Path | None)

Return type:

morie.arsau_analyze.arsau_analyze_weapon_records(year, *, allow_invalid=False, language='en', data_dir=None)[source]¶

Analysis of ARSAU weapon_records.

Chains morie.mrm_uof.mrm_uof_weapon_diversity() over Weapon x Location (the only two categorical columns the file publishes) plus a data-quality audit.

The 2023 file is the ministry-flagged-invalid release and requires allow_invalid=True.

Parameters:

year (str | int)
allow_invalid (bool)
language (str)
data_dir (str | Path | None)

Return type:

morie.arsau_analyze.arsau_analyze_aggregate_summary(year_range='2020-2022', *, language='en', data_dir=None)[source]¶

Analysis of the ARSAU aggregate-summary-by-year file.

The aggregate file is a long-format YEAR_2020 / YEAR_2021 / YEAR_2022 panel keyed by (SECTION, CATEGORY, UNITS OF MEASURE). We rebuild the implied time series, run year-on-year change against the REPORT_SCOPE rows (the headline volume series), and surface a data-quality audit.

Parameters:

year_range (str)
language (str)
data_dir (str | Path | None)

Return type:

morie.arsau_analyze.arsau_analyze_detailed_dataset(year_range='2020-2022', *, language='en', data_dir=None)[source]¶

Wide-format analysis of the 2020-2022 detailed-incident dataset.

Chains:

force_concentration on POLICE_SERVICE
weapon_diversity on (POLICE_SERVICE x ASSIGNMENT_TYPE)
yoy_change on REPORTING_YEAR
data_quality audit

Parameters:

year_range (str)
language (str)
data_dir (str | Path | None)

Return type:

morie.arsau_datasets — loaders + registry for Ontario ARSAU data.

ARSAU = the Ontario Ministry of the Solicitor General’s provincial release of Police Use-of-Force incident records (formally “Race-Based and Identity-Based Data on Police Use of Force in Ontario”). The data are published on the Ontario Data Catalogue (data.ontario.ca) as a multi-year CKAN package; the public-facing landing page is https://data.ontario.ca/dataset/police-use-of-force-race-based-data.

This module ships:

ARSAU_REGISTRY — every (year_or_range, kind) tuple the ministry currently publishes, with the corresponding CSV filename, CKAN resource_id for the sidecar, expected row / column counts observed on disk, and a per-entry is_valid flag. The 2023 uof_weapon_records_invaliddata.csv is flagged is_valid=False because the ministry itself marked the file as containing invalid data (per the published technical notes); no loader will return its rows without the user opting in via allow_invalid=True.
read_sidecar() / sidecar_to_frame() / sidecar_schema() — generic CKAN datastore_search JSON readers, used both by the loaders here and by callers who want offline access to column metadata.
Per-record-type loaders, returning morie.fn._richresult.RichResult: arsau_load_main_records(), arsau_load_individual_records(), arsau_load_probe_cycle_records(), arsau_load_weapon_records(), arsau_load_aggregate_summary(), arsau_load_detailed_dataset().
Discovery callables: arsau_available_years(), arsau_available_datasets(), arsau_describe().

Portability¶

This module NEVER hard-codes paths to the maintainer’s workstation. All file resolution goes through morie._datapaths.resolve_data_dir(), which honours (in order):

an explicit data_dir= argument
the MORIE_ARSAU_DIR environment variable
MORIE_DATA_DIR/arsau
the platform-appropriate user-data directory (~/Library/Application Support/morie/arsau on macOS, ~/.local/share/morie/arsau on Linux, %APPDATA%/morie/arsau on Windows) — only if it already exists
a bundled tiny fixture under morie/data/arsau/ — present in the wheel for unit tests + tutorials only

Set the env var or pass data_dir= on every loader call to opt in to the full upstream release.

2023 weapon data — explicit invalidity gate¶

The 2023 release of ARSAU shipped a separate uof_weapon_records_invaliddata.csv file rather than the usual uof_weapon_records.csv. The ministry’s technical report for 2023 flagged the weapon-record submissions as data-quality non-compliant (under-reporting + format drift across the participating services), and the open-data release was renamed accordingly. We therefore:

expose the file in ARSAU_REGISTRY so it remains discoverable,
mark its entry is_valid=False,
raise ValueError from arsau_load_weapon_records() for year 2023 unless the caller passes allow_invalid=True,
if allow_invalid=True is set, return a RichResult whose interpretation opens with an explicit warning paragraph and whose warnings list flags the invalidity so downstream consumers cannot accidentally treat the rows as comparable to the 2022 / 2024 weapon records.

Examples

>>> from morie.arsau_datasets import (
...     arsau_available_years, arsau_available_datasets,
...     arsau_load_main_records,
... )
>>> arsau_available_years()
RichResult(... years=['2020-2022', '2023', '2024'])
>>> arsau_load_main_records(2024).n_rows
11326

class morie.arsau_datasets.ArsauDataset(year_or_range, kind, csv_filename, sidecar_filename, expected_rows, expected_cols, is_valid, description_en, description_fr)[source]¶

Bases: object

Metadata for a single (year_or_range, kind) entry in ARSAU.

Parameters:

year_or_range (str)
kind (str)
csv_filename (str)
sidecar_filename (str | None)
expected_rows (int)
expected_cols (int)
is_valid (bool)
description_en (str)
description_fr (str)

year_or_range: str¶: e.g. "2020-2022", "2023", "2024".

kind: str¶

Type:: One of

csv_filename: str¶: Filename of the CSV under <arsau_root>/<year_or_range>/.

sidecar_filename: str | None¶: Filename of the matching CKAN sidecar JSON, or None if the ministry did not publish one (e.g. the 2023 weapon-records file).

expected_rows: int¶: Number of data rows observed when the registry was last refreshed, used for smoke-test sanity checking.

expected_cols: int¶: Number of columns observed.

is_valid: bool¶: True for all ARSAU files except the 2023 weapon-records, which the ministry flagged as containing invalid data.

description_en: str¶

description_fr: str¶

csv_path(root)[source]¶

Parameters:: root (Path)
Return type:: Path

sidecar_path(root)[source]¶

Parameters:: root (Path)
Return type:: Path | None

morie.arsau_datasets.read_sidecar(path)[source]¶

Parse a CKAN datastore_search JSON file from disk.

Handles both bare {"fields": [...], "records": [...]} and the common {"result": {"fields": [...], "records": [...]}} wrapper.

Parameters:: path (str or Path)
Returns:: With keys "fields" and "records" (always present; empty lists if the file is malformed).
Return type:: dict

morie.arsau_datasets.sidecar_schema(sidecar)[source]¶

Extract a simplified [{name, type, notes}, ...] from a sidecar dict.

Parameters:: sidecar (dict)
Return type:: list[dict[str, Any]]

morie.arsau_datasets.sidecar_to_frame(sidecar)[source]¶

Convert a CKAN sidecar’s records array-of-arrays to a DataFrame.

The fields[].id array supplies the column names. Records are list-of-list, so the column order in the JSON matches the column order in the resulting DataFrame.

Parameters:: sidecar (dict)
Return type:: DataFrame

morie.arsau_datasets.arsau_load_main_records(year, *, language='en', data_dir=None)[source]¶

Load the ARSAU uof_main_records CSV for the given year.

Available years (incident-level main records): 2023, 2024. The 2020-2022 release uses a different layout — see arsau_load_detailed_dataset() for that range.

Parameters:

year (str | int)
language (str)
data_dir (str | Path | None)

Return type:

morie.arsau_datasets.arsau_load_individual_records(year, *, language='en', data_dir=None)[source]¶

Load the ARSAU uof_individual_records CSV for the given year.

Available years: 2023, 2024.

Parameters:

year (str | int)
language (str)
data_dir (str | Path | None)

Return type:

morie.arsau_datasets.arsau_load_probe_cycle_records(year, *, language='en', data_dir=None)[source]¶

Load the ARSAU uof_probe_cycle_records (CEW telemetry) CSV.

Available years: 2023, 2024.

Parameters:

year (str | int)
language (str)
data_dir (str | Path | None)

Return type:

morie.arsau_datasets.arsau_load_weapon_records(year, *, allow_invalid=False, language='en', data_dir=None)[source]¶

Load the ARSAU uof_weapon_records CSV for the given year.

Available years: 2023 (INVALID — requires allow_invalid=True), 2024.

Parameters:

year (str | int) – 2023 or 2024.
allow_invalid (bool, default False) – Required to be True to load the 2023 file, which the publishing ministry marked as invalid. See the module docstring for details.
language (str)
data_dir (str | Path | None)

Raises:

ValueError – If year == 2023 and allow_invalid is False.

Return type:

morie.arsau_datasets.arsau_load_aggregate_summary(year_range='2020-2022', *, language='en', data_dir=None)[source]¶

Load the ARSAU aggregate-summary-by-year CSV.

Currently only published as a single combined 2020-2022 file.

Parameters:

year_range (str)
language (str)
data_dir (str | Path | None)

Return type:

morie.arsau_datasets.arsau_load_detailed_dataset(year_range='2020-2022', *, language='en', data_dir=None)[source]¶

Load the ARSAU detailed-incident-level CSV.

Currently only published as a single combined 2020-2022 file (167 columns wide; the 2023+ releases switched to the main/individual/probe_cycle/weapon four-file split).

Parameters:

year_range (str)
language (str)
data_dir (str | Path | None)

Return type:

morie.arsau_datasets.arsau_available_years(*, data_dir=None, language='en')[source]¶

List ARSAU year/year-range buckets the package knows about.

If the user has the data files on disk, the result also flags which entries are present vs missing in their tree.

Parameters:

data_dir (str | Path | None)
language (str)

Return type:

morie.arsau_datasets.arsau_available_datasets(year=None, *, data_dir=None, language='en')[source]¶

List ARSAU dataset kinds, optionally restricted to one year.

Parameters:

year (str | int | None) – If supplied, only list kinds available for this year/range. None lists everything in ARSAU_REGISTRY.
data_dir (str | Path | None)
language (str)

Return type:

morie.arsau_datasets.arsau_describe(kind, year, *, language='en', data_dir=None, n_preview_rows=3)[source]¶

Describe a single ARSAU dataset entry.

Returns the registry metadata, the CKAN sidecar schema if present, and a small preview of the actual rows if the CSV is on disk.

Parameters:

kind (str) – One of ARSAU_KINDS.
year (str | int) – One of ARSAU_YEARS.
language (str) – "en" or "fr".
n_preview_rows (int, default 3) – How many rows from the CSV head to include.
data_dir (str | Path | None)

Return type:

ActuarialRiskDisparityResult

morie.laniyonu — policing & corrections methodology from Laniyonu et al.

Three reproducible-replication modules, each calling out to the v0.6.1 morie.mrm_primitives building blocks:

gentrification_policing() — Laniyonu (2018) UAR 54(5):898–930

smi_force_disparity() — Laniyonu & Goff (2021) BMC Psych
21(1):500

actuarial_risk_disparity() — O’Connell & Laniyonu (2025)
Race & Justice 15(3):428–453

Each module composes the corresponding MRM primitives and adds the paper-specific wrapper (column-name conventions, reporting structure, sensitivity grid). Users who already have data in canonical form can call the primitives directly; this module is the “I have the data the way the paper has it” entry point.

morie.laniyonu.gentrification_policing(df, *, year_col='year', tract_id_col='tract_id', stops_col='stops', population_col='population', crime_col='felony_count', demand_col='calls_311_omp', baseline_income_col='median_inc_2000', baseline_rent_col='median_rent_2000', growth_college_col=None, growth_rent_col=None, follow_income_col='median_inc_2014', follow_rent_col='median_rent_2014', baseline_college_col='pct_ba_2000', follow_college_col='pct_ba_2014', additional_controls=None, weight_matrix=None, weight_matrix_kind='queen', fitted_rho=None, fitted_beta_direct=None, fitted_beta_spatial=None, years=None, log_outcome=True)[source]¶

Replicate the Laniyonu (2018) SDM by year, return per-year decompositions.

Two modes:

Pre-fitted mode (preferred). Pass fitted_rho, fitted_beta_direct, fitted_beta_spatial from your own SDM fit (spreg, pysal, R’s spatialreg, etc.). This function does the diagnostic ladder + spillover decomposition + gentrification-coding sensitivity report.
Pure-numpy lite mode. When fitted_* aren’t supplied, we run OLS, compute Moran’s I on residuals, and report the decomposition with rho=0 as a fall-back. Useful for sanity checks; for paper-ready results you almost certainly want mode 1.

Parameters:

df (pd.DataFrame) – Tract-year panel. One row per tract per year.
year_col (str)
tract_id_col (str)
stops_col (str)
population_col (str)
crime_col (str)
demand_col (str)
baseline_income_col (str)
baseline_rent_col (str)
growth_college_col (str | None)
growth_rent_col (str | None)
follow_income_col (str)
follow_rent_col (str)
baseline_college_col (str)
follow_college_col (str)
additional_controls (list[str] | None)
weight_matrix (marr | None)
weight_matrix_kind (Literal['queen', 'knn'])
fitted_rho (float | dict[int, float] | None)
fitted_beta_direct (dict[int, marr] | None)
fitted_beta_spatial (dict[int, marr] | None)
years (list[int] | None)
log_outcome (bool)

Return type:

list[GentrificationPolicingResult]

:param : :param crime_col: Column names. Defaults match the schema documented in

ROADMAP.md’s load_gentrification_nyc() toy bundle.

Parameters:

demand_col (str) – Column names. Defaults match the schema documented in ROADMAP.md’s load_gentrification_nyc() toy bundle.
baseline_income_col (str) – Income + rent at baseline period (2000 in the paper).
baseline_rent_col (str) – Income + rent at baseline period (2000 in the paper).
growth_college_col (str or None) – Growth columns. If None, computed from follow_<x>_col minus baseline_<x>_col.
growth_rent_col (str or None) – Growth columns. If None, computed from follow_<x>_col minus baseline_<x>_col.
follow_income_col (str)
follow_rent_col (str)
baseline_college_col (str)
df (DataFrame)
year_col (str)
tract_id_col (str)
stops_col (str)
population_col (str)
crime_col (str)
follow_college_col (str)
additional_controls (list[str] | None)
weight_matrix (marr | None)
weight_matrix_kind (Literal['queen', 'knn'])
fitted_rho (float | dict[int, float] | None)
fitted_beta_direct (dict[int, marr] | None)
fitted_beta_spatial (dict[int, marr] | None)
years (list[int] | None)
log_outcome (bool)

Return type:

list[GentrificationPolicingResult]

:param : :param follow_college_col: Used when growth columns are not pre-computed. :type follow_college_col: str :param additional_controls: Extra tract-year controls (pct_black, pct_latino, etc.). :type additional_controls: list[str], optional :param weight_matrix: Pre-computed (N, N) row-standardised spatial weights.

Required when fitted_* mode is in use. In lite mode, if None, we synthesise a uniform-block W (placeholder).

Parameters:

weight_matrix_kind ("queen" | "knn") – Provenance label only; we don’t recompute W here.
fitted_rho (float | dict[int, float] | None) – Pass these to bypass the lite-mode fall-back.
fitted_beta_direct (dict[int, marr] | None) – Pass these to bypass the lite-mode fall-back.
fitted_beta_spatial (dict[int, marr] | None) – Pass these to bypass the lite-mode fall-back.
years (list[int], optional) – Subset of years to analyse. Default = all years in df.
log_outcome (bool, default True) – If True, the outcome is log(stops / population). If False, raw stops / population.
df (DataFrame)
year_col (str)
tract_id_col (str)
stops_col (str)
population_col (str)
crime_col (str)
demand_col (str)
baseline_income_col (str)
baseline_rent_col (str)
growth_college_col (str | None)
growth_rent_col (str | None)
follow_income_col (str)
follow_rent_col (str)
baseline_college_col (str)
follow_college_col (str)
additional_controls (list[str] | None)
weight_matrix (np.ndarray, optional)

Return type:

list[GentrificationPolicingResult], one per year analysed.

morie.laniyonu.smi_force_disparity(df, *, survey_df, survey_trait_col='smi', survey_covariate_cols, area_covariate_cols=None, force_count_col='force_events', non_smi_count_col=None, geog_col='tract_id', year_col='year', population_col='pop_18plus', baseline_year=None, include_year_fe=True, include_area_re=True, max_iter=500, tol=1e-06, return_design=False)[source]¶

Replicate Laniyonu & Goff (2021): hierarchical NB on SMI force disparity.

Composes morie.mrm_primitives.synthetic_area_exposure() to build the per-area PwSMI exposure offset, then fits a negative- binomial GLM with year fixed effects and an area random intercept via hand-rolled MLE (scipy.optimize.minimize, no statsmodels).

The outcome is a single relative-risk: force-against-PwSMI versus force-against-non-SMI. The PwSMI events use the SAE-derived offset n_{vti}; the non-SMI events use (pop_18plus - SAE_exposure) as their offset. The vulnerability dummy alpha_v picks up the log-RR.

Parameters:

df (pd.DataFrame) – Force-event panel, one row per (area, year). Must contain force_count_col (PwSMI events), the area covariates, the geog id, year, and adult population.
survey_df (pd.DataFrame) – Survey microdata (e.g., NCS-R) for fitting P(SMI | covariates). Passed through to the SAE primitive.
survey_trait_col (str, default "smi") – Binary column in survey_df flagging SMI.
survey_covariate_cols (list[str]) – Covariates available in BOTH survey_df and df (after renaming via area_covariate_cols if needed).
area_covariate_cols (list[str], optional) – Column names for the SAE covariates in df. If None, we assume the columns are named identically to survey_covariate_cols.
force_count_col (str, default "force_events") – Count of force events against PwSMI per (area, year).
non_smi_count_col (str or None, default None) – Count of force events against non-SMI per (area, year). If None, treated as total_force_events - force_count_col and the user must supply a column named "total_force_events".
geog_col (str, default "tract_id") – Area unit identifier (tract or precinct).
year_col (str, default "year") – Year column.
population_col (str, default "pop_18plus") – Adult population per area (used for the non-SMI offset and passed to the SAE primitive).
baseline_year (int, optional) – Year to drop as the reference (default = min year in df).
include_year_fe (bool, default True) – If False, omits delta_t.
include_area_re (bool, default True) – If True, an area-level random intercept is approximated as a free per-area shift estimated jointly (penalised by an L2 prior with SD estimated from the data — a marginalised approximation, NOT a full Bayesian hierarchical fit).
max_iter (optimiser controls.)
tol (optimiser controls.)
return_design (bool, default False) – If True, attaches X, y, and offset to the result’s exposure_summary dict so the caller can hand off to pymc.

Return type:

SMIForceDisparityResult

Warns:

UserWarning – Always: about the SMI-flag proxy bias (officers miss more SMI than they over-attribute), so the estimated alpha_v is a lower bound on the true disparity.

Notes

This is a frequentist MLE approximation to the paper’s Bayesian hierarchical model. The “random intercept” is implemented as free per-area shifts with an L2 ridge penalty whose strength is set so the implied SD matches the residual area-level variance. For paper-grade credible intervals, fit the model in pymc / brms using the design matrix you can extract with return_design=True.

class morie.laniyonu.SMIForceDisparityResult(alpha_v, intercept, year_effects, area_random_effect_sd, dispersion, n_events, n_area_years, log_likelihood, converged, exposure_summary=<factory>, note='')[source]¶

Bases: object

Replication output for Laniyonu & Goff (2021).

Variables:

alpha_v (SMIForceDisparityCoefficient) – Headline coefficient: log relative-risk of force against PwSMI vs non-SMI. alpha_v.rr is the multiplicative relative risk.
intercept (SMIForceDisparityCoefficient) – Grand intercept mu.
year_effects (list[SMIForceDisparityCoefficient]) – Year fixed effects delta_t (one per year minus baseline).
area_random_effect_sd (float) – Estimated SD of the area-level random intercept beta_i. Approximated by a single dispersion parameter under the random-intercept assumption (see note).
dispersion (float) – Negative-binomial dispersion phi. Variance = mu + mu^2/phi.
n_events (int) – Total force events across the panel.
n_area_years (int) – Number of (area x year) observation rows.
log_likelihood (float) – Final log-likelihood at the MLE.
converged (bool) – Whether scipy.optimize reports convergence.
exposure_summary (dict) – Summary stats on the SAE-derived offset for sanity checking.
note (str) – Caveats / methodology notes for the user.

Parameters:

alpha_v (SMIForceDisparityCoefficient)
intercept (SMIForceDisparityCoefficient)
year_effects (list[SMIForceDisparityCoefficient])
area_random_effect_sd (float)
dispersion (float)
n_events (int)
n_area_years (int)
log_likelihood (float)
converged (bool)
exposure_summary (dict)
note (str)

alpha_v: SMIForceDisparityCoefficient¶

intercept: SMIForceDisparityCoefficient¶

year_effects: list[SMIForceDisparityCoefficient]¶

area_random_effect_sd: float¶

dispersion: float¶

n_events: int¶

n_area_years: int¶

log_likelihood: float¶

converged: bool¶

exposure_summary: dict¶

note: str = ''¶

interpret()[source]¶

Return type:: str

class morie.laniyonu.SMIForceDisparityCoefficient(name, estimate, std_error)[source]¶

Bases: object

Single coefficient with point estimate, SE, and Wald 95% CI.

Parameters:

name (str)
estimate (float)
std_error (float)

name: str¶

estimate: float¶

std_error: float¶

property ci_low: float¶

property ci_high: float¶

property rr: float¶: exp(estimate) — relative risk on the multiplicative scale.

property rr_ci_low: float¶

property rr_ci_high: float¶

morie.laniyonu.actuarial_risk_disparity(df, *, outcome, race_cols, gender_col='gender', score_col=None, control_cols=None, ordinal_levels=None, outcome_col=None, split_by_gender=True, bootstrap_replicates=200, random_state=20260513)[source]¶

Audit CSC actuarial risk and downstream outcomes for race × gender bias.

Parameters:

df (pd.DataFrame) – Sentence-level (one row per sentence) CSC microdata. Race columns may be either pre-dummied 0/1 indicators or a single categorical race column — pass the indicator column names in race_cols (e.g. ["race_indigenous", "race_black"] with White as the implicit reference).
outcome ({"static", "dynamic", "osl", "reintegration",) – “parole”, “housing”} Which audit to run. The first four are the ordinal risk scores (stage 1 — threshold-specific ordinal logit). The last two are the binary downstream outcomes (stage 2 — score- net-residual audit).
race_cols (list[str]) – Race indicator columns. Must be 0/1 numeric. Reference category (typically White) is the omitted column.
gender_col (str, default "gender") – Categorical gender column. Used either as a stratifying variable (when split_by_gender=True) or one-hot-dummied and stacked into the design matrix (when False).
score_col (str, optional) – Required when outcome is "parole" or "housing" — the actuarial score on which CSC claims to make the decision. Typically "reintegration_score_numeric" for parole and "osl_score_numeric" for housing.
control_cols (list[str], optional) – Additional controls (age, prior offences, sentence length, offence-severity dummies, etc.). Pre-dummy any categoricals.
ordinal_levels (list[str], optional) – Override level ordering for ordinal outcomes. Default ["low", "medium", "high"] for all four risk scores — matches the CSC released form. Ignored for binary outcomes.
outcome_col (str, optional) – Override the default column name for the chosen outcome. Defaults to the canonical name in the CSC FOI release (see _ORDINAL_OUTCOMES / _BINARY_OUTCOMES).
split_by_gender (bool, default True) – Stage-1 only. When True, fits the threshold-specific ordinal separately within each gender stratum — recovers the “Indigenous men +0.59” style estimate cleanly. When False, adds a one-hot gender column to the pooled design.
bootstrap_replicates (int, default 200) – Stage-2 only; bootstrap reps for residual race-coef SEs.
random_state (int, default 20260513)

Return type:

Notes

O’Connell & Laniyonu fit the Bayesian version; the primitive morie composes is the frequentist analogue (see morie.mrm_primitives.ordinal for the explicit trade-off). The threshold-specific coefficient pattern (β_low->med >> β_med->high for race) is recovered identically; only the posterior-credible-interval reporting is different.

class morie.laniyonu.ActuarialRiskDisparityResult(outcome, outcome_kind, outcome_col, n_obs, race_cols, gender_col, ordinal_result=None, residual_result=None, per_threshold_logodds=<factory>, proportional_odds_stat=None, proportional_odds_df=None, proportional_odds_p=None, note='')[source]¶

Bases: object

Per-outcome disparity audit + diagnostics.

Exactly one of ordinal_result / residual_result is populated, depending on whether outcome is one of the four ordinal risk scores or one of the two binary downstream outcomes.

The per_threshold_logodds field is a flat per-stratum dict of the form {(gender_stratum, threshold_label, race_column): logodds}, populated for stage-1 (ordinal) runs. When split_by_gender=False the gender stratum is "pooled".

Parameters:

outcome (str)
outcome_kind (Literal['ordinal', 'binary'])
outcome_col (str)
n_obs (int)
race_cols (list[str])
gender_col (str)
ordinal_result (dict[str, ThresholdSpecificOrdinalResult] | None)
residual_result (ScoreNetResidualResult | None)
per_threshold_logodds (dict[tuple[str, str, str], float])
proportional_odds_stat (float | None)
proportional_odds_df (int | None)
proportional_odds_p (float | None)
note (str)

outcome: str¶

outcome_kind: Literal['ordinal', 'binary']¶

outcome_col: str¶

n_obs: int¶

race_cols: list[str]¶

gender_col: str¶

ordinal_result: dict[str, ThresholdSpecificOrdinalResult] | None = None¶

residual_result: ScoreNetResidualResult | None = None¶

per_threshold_logodds: dict[tuple[str, str, str], float]¶

proportional_odds_stat: float | None = None¶

proportional_odds_df: int | None = None¶

proportional_odds_p: float | None = None¶

note: str = ''¶

residual_race_coefficient(race_col)[source]¶

Stage-2 residual race coefficient (None when stage 1).

Parameters:: race_col (str)
Return type:: float | None

bias_concentrated_at_low_medium(race_col, gender_stratum='pooled', ratio_threshold=1.5)[source]¶

True iff |β_threshold1| / |β_threshold2| >= ratio_threshold.

Returns None when not a stage-1 result, or when the second threshold’s coefficient is too small to take the ratio safely.

Parameters:

race_col (str)
gender_stratum (str)
ratio_threshold (float)

Return type:

bool | None

interpret()[source]¶

Return type:: str

Taphonomic preservation as a causal-inference problem.

A thin domain layer that recasts “is this body’s preservation natural or anomalous?” as a treatment-effect estimate over MORIE’s existing causal estimators. It adds no new statistics: it documents the taphonomy variable set, dispatches to morie.estimate_irm() / morie.estimate_cate(), and attaches an E-value (morie.sensitivity.e_value_d()) that quantifies how strong an unmeasured cause would have to be to explain away the natural preservation effect – i.e. it turns a claim of “incorruptibility” into a bounded statistical one.

This module ships no data. Forensic-taphonomy comparanda (documented lime/desiccation burials with preservation outcomes) and any non-invasive readings of a specific case (CT/micro-CT density, pXRF elemental signatures, hyperspectral surface composition) must be supplied by the caller from real sources. taphonomy_schema() returns a typed, zero-row frame describing the expected columns; it never fabricates rows.

What it can and cannot do. With comparanda it estimates the average preservation effect attributable to burial processing (e.g. quicklime desiccation) and reports the E-value residual. It cannot falsify a miracle: a small natural-preservation probability yields a delta, not a disproof (Chernozhukov et al. 2018 give the effect; VanderWeele & Ding 2017 bound the unmeasured confounding needed to nullify it).

R parity: r-morie-oss R/taphonomy.R (morie_taphonomy_schema / morie_taphonomy_preservation_delta).

morie.taphonomy.taphonomy_schema()[source]¶

Return an empty, typed DataFrame documenting the expected columns.

No rows are fabricated – the caller fills it with real comparanda. The role of each column (treatment / covariate / measurement / outcome) is stored in df.attrs['role'].

Return type:: DataFrame

morie.taphonomy.taphonomy_preservation_delta(data, *, treatment='lime_treatment', outcome='preservation_score', covariates=None, estimator='irm', se_method='none', n_boot=199, boot_seed=42, **kwargs)[source]¶

Estimate the natural preservation “delta” and its E-value.

Recasts a taphonomy question as a treatment-effect estimate: how much of the observed preservation is attributable to burial processing (the treatment, e.g. quicklime desiccation), holding environment and handling fixed, and how strong an unmeasured cause (a “miracle”) would need to be to explain the rest. A thin dispatch over morie.estimate_irm() (default) or morie.estimate_cate() plus morie.sensitivity.e_value_d(). Adds no new statistics.

Parameters:

data (pandas.DataFrame) – Real comparanda (see taphonomy_schema()). Must be non-empty.
treatment (str) – Binary treatment column (default "lime_treatment").
outcome (str) – Continuous preservation outcome (default "preservation_score").
covariates (list[str], optional) – Confounder/measurement columns. Defaults to every schema covariate + measurement present in data.
estimator (str) – "irm" (DoubleML, default) or "cate".
se_method (str) – Inference for the "cate" path only: "none" (default) reports the point estimate + cate_sd dispersion with no SE/CI/p-value; "bootstrap" resamples rows and refits the CATE procedure n_boot times for a valid SE, percentile CI, and p-value. (sd(tau)/sqrt(n) is deliberately not offered – the per-unit effects are correlated fitted predictions, so it understates the SE; use cate_sd for heterogeneity or bootstrap for inference.) Ignored for "irm".
n_boot (int) – Bootstrap resamples when se_method="bootstrap" (default 199).
boot_seed (int) – RNG seed for the bootstrap (default 42).
**kwargs – Passed to the chosen estimator.

Returns:

All estimates are float (double); n is an int count. Keys: value (preservation delta = ATE / mean CATE), se, p_value (Wald; None on the CATE path – no valid SE), ci_lower, ci_upper, n, e_value, e_value_ci, estimator, cate_per_unit and cate_sd (None unless estimator="cate"), warnings, interpretation.

Return type:

morie.taphonomy.taphonomy_decay_chain(preservation=0.0, decay_rate=0.5, mummify_rate=0.5, states=None)[source]¶

Build a taphonomic decay Markov chain (absorbing DTMC).

From each transient decomposition stage a body either progresses one step toward the terminal "skeletal" state (ordinary decay) or diverts to the terminal "mummified" state (preserved). preservation in [0, 1] (quicklime desiccation, aridity, sealing) shifts mass from progression toward diversion. Compare against the preservation=0 chain to quantify how much the burial practice changes the fate distribution.

Returns a dict with P (row-stochastic float transition matrix), states, transient, absorbing (["skeletal", "mummified"]), and preservation. R parity: morie_taphonomy_decay_chain.

Examples

>>> chain = taphonomy_decay_chain(decay_rate=0.6, mummify_rate=0.1)
>>> sorted(k for k in chain if not k.startswith("_"))[:6]
['P', 'absorbing', 'preservation', 'states', 'transient']

Parameters:

preservation (float)
decay_rate (float)
mummify_rate (float)
states (list[str] | None)

Return type:

morie.taphonomy.taphonomy_decay_absorption(chain, start=None)[source]¶

Absorption analysis via the fundamental matrix N = (I - Q)^-1.

For a body entering at start, returns P(each terminal fate) and the expected steps to absorption (Grinstead & Snell, Ch. 11). Keys: absorption (dict over absorbing states, sums to 1), expected_steps, fundamental (N), B. R parity: morie_taphonomy_decay_absorption.

Parameters:

chain (dict[str, Any])
start (str | None)

Return type:

morie.taphonomy.taphonomy_decay_simulate(chain, start=None, n_steps=100, seed=42)[source]¶

Simulate one realised decay trajectory to an absorbing fate.

Deterministic given seed. R parity: morie_taphonomy_decay_simulate.

Parameters:

chain (dict[str, Any])
start (str | None)
n_steps (int)
seed (int)

Return type:

morie.taphonomy.taphonomy_decay_delta(preservation, start=None, **kwargs)[source]¶

Natural-vs-treated fate delta: change in P(mummified) from preservation.

The Markov-chain analogue of the preservation delta – the rise in the probability of ending "mummified" when a preservation factor is applied, relative to the natural (preservation=0) baseline. Keys: p_mummified_natural, p_mummified_treated, delta, interpretation. R parity: morie_taphonomy_decay_delta.

Parameters:

preservation (float)
start (str | None)
kwargs (Any)

Return type:

morie.taphonomy.taphonomy_evidence_loglik(evidence, mean, sd)[source]¶

Gaussian log-likelihood of measured evidence under a model.

Sum of independent normal log-densities for measured evidence (e.g. pXRF calcium, CT density) under a model with expected mean and sd (broadcast over evidence). R parity: morie_taphonomy_evidence_loglik.

Return type:: float

morie.taphonomy.taphonomy_likelihood_ratio(loglik_h1, loglik_h2)[source]¶

Forensic likelihood ratio LR = P(E|H1) / P(E|H2).

Given the evidence log-likelihood under a natural/target hypothesis and an alternative, returns the LR, its base-10 log, and the ENFSI (2015) verbal equivalent. Computed in log space for stability. The LR reports how much the evidence favours H1 over H2 – it is not posterior odds and does not prove either hypothesis. R parity: morie_taphonomy_likelihood_ratio.

Parameters:

loglik_h1 (float)
loglik_h2 (float)

Return type:

morie.taphonomy.taphonomy_preservation_lr(evidence, natural, alternative)[source]¶

Preservation likelihood ratio from measured evidence.

Evaluate measured non-invasive evidence under a natural-preservation model (H1) and an alternative model (H2), each a dict {"mean": .., "sd": ..}, and return the forensic LR (with loglik_h1/loglik_h2 attached). R parity: morie_taphonomy_preservation_lr.

Parameters:

natural (dict[str, Any])
alternative (dict[str, Any])

Return type:

morie.taphonomy.taphonomy_bhm(data, *, outcome='preservation_score', covariates=None, group=None, priors=None, prior_sd_default=10.0, backend='conjugate', chains=4, iter=1000, seed=42)[source]¶

Bayesian hierarchical preservation model (conjugate + EB pooling).

Conjugate Gaussian-linear Bayesian model with informative Normal priors on the coefficients (domain knowledge, e.g. quicklime’s desiccant effect, enters as a prior). Closed-form posterior; noise variance from OLS residuals (empirical Bayes). With group, adds empirical-Bayes partial-pooled random intercepts (normal-normal shrinkage) for a two-level hierarchical model.

backend="conjugate" (default) uses the dependency-free closed form; backend="mcmc" fits the same model by full-Bayes HMC/NUTS via cmdstanpy + a built CmdStan (returns the stanfit too). chains/ iter/seed control the sampler. R parity: morie_taphonomy_bhm (its backend="cmdstanr").

Returns a dict: coefficients (DataFrame: term, post_mean, post_sd, ci_lower, ci_upper, prob_positive), sigma, group_effects (None unless group), fitted, n, interpretation.

Parameters:

data (DataFrame)
outcome (str)
covariates (list[str] | None)
group (str | None)
priors (dict[str, dict[str, float]] | None)
prior_sd_default (float)
backend (str)
chains (int)
iter (int)
seed (int)

Return type:

morie.taphonomy.taphonomy_simulate_pxrf(n, condition='control', elements=None, alpha=None, seed=None, as_ppm=False, total_ppm=1000000.0)[source]¶

Simulate synthetic pXRF compositional data (Dirichlet).

Synthetic data for testing/calibration ONLY – never a substitute for real comparanda, and never written to bundled data. pXRF elemental concentrations are closed compositions on the simplex, so this samples from a Dirichlet (not Gaussian). Control = natural matrix; treatment = quicklime (calcium-skewed). R parity: morie_taphonomy_simulate_pxrf.

Returns a DataFrame with one column per element plus condition and lime_treatment (1 = treatment); df.attrs carries elements, alpha, synthetic=True.

Parameters:

n (int)
condition (str)
elements (list[str] | None)
seed (int | None)
as_ppm (bool)
total_ppm (float)

Return type:

DataFrame

morie.taphonomy.taphonomy_clr(x, pseudocount=1e-06)[source]¶

Centred log-ratio (CLR) transform: clr(x) = log x - mean(log x), row-wise.

Removes simplex closure. Rank-deficient (columns sum to 0); for regression inputs prefer taphonomy_ilr(). R parity: morie_taphonomy_clr.

Parameters:: pseudocount (float)
Return type:: marr

morie.taphonomy.taphonomy_ilr(x, pseudocount=1e-06)[source]¶

Isometric log-ratio (ILR): D-part composition -> D-1 orthonormal coords.

Full-rank (unlike CLR), so it feeds DML/BHM without a singular design. Egozcue et al. (2003) pivot-coordinate basis, closed form – identical to the R sibling. R parity: morie_taphonomy_ilr.

Parameters:: pseudocount (float)
Return type:: marr

morie.taphonomy.taphonomy_fetch_usgs_soil(dest=None, nrows=1000, url='https://mrdata.usgs.gov/ngdb/soil/ngdbsoil-csv.zip', refresh=False)[source]¶

Fetch USGS National Geochemical Database soil geochemistry.

Downloads the USGS NGDB (soil) bulk CSV – real, open elemental concentrations (the compositional analogue of pXRF spectra) to calibrate the clr/ilr/DML pipeline on genuine open data before real scans exist. Dependency-free (base zipfile + pandas; the WFS endpoint is GML-only and not used). ~54 MB download (482 MB uncompressed), cached in dest. R parity: morie_taphonomy_fetch_usgs_soil.

Parameters:

nrows (int | None)
url (str)
refresh (bool)

Return type:

DataFrame

morie.taphonomy.taphonomy_pmi_schema()[source]¶

STO-2022 taphonomic-observation schema (PMI nuisance variables).

Typed zero-row template of the taphonomic-observation + environmental variables for post-mortem-interval work (Standard for Taphonomic Observations in Support of the PMI, 2022 – the geoFOR variable family). geoFOR itself is an app with no open data API, so this structures your own case observations into the DML/Bayesian nuisance set X. No fabricated rows. R parity: morie_taphonomy_pmi_schema.

Return type:: DataFrame

morie.taphonomy.taphonomy_morphosource_search(query=None, kind='media', media_type=None, taxonomy_gbif=None, visibility=None, media_tag=None, per_page=10, page=1, api_key=None)[source]¶

Search MorphoSource for 3D media or physical objects.

Public search needs no key; a key (restricted records) comes from api_key or MORPHOSOURCE_API_KEY and travels only in the Authorization header. kind is "media" or "physical-objects". Returns a dict with items, n, total_pages, and df (id + title). R parity: morie_taphonomy_morphosource_search.

Return type:: dict

morie.taphonomy.taphonomy_morphosource_fetch(media_id, use_statement, use_categories=None, use_category_other=None, dest=None, api_key=None)[source]¶

Download a MorphoSource media bundle (requires a data-use statement).

MorphoSource enforces a use agreement on every download, so use_statement is required (agreements_accepted=True is sent); restricted media also need per-item permission on the site. Key from api_key/MORPHOSOURCE_API_KEY, Authorization header only. Returns the path to the downloaded zip. R parity: morie_taphonomy_morphosource_fetch.

Return type:: str

Python counterparts for core investigation modules from the old R workflow.

morie.investigation.run_weighted_logistic_analysis(data, *, outcome='heavy_drinking_30d', treatment='cannabis_any_use', weight_col='weight', predictors=None)[source]¶

Mirror the core outputs of the old 06_logistic.R module.

Parameters:

data (DataFrame)
outcome (str)
treatment (str)
weight_col (str)
predictors (list[str] | None)

Return type:

dict[str, DataFrame]

morie.investigation.compare_nested_logistic_models(data, *, outcome='heavy_drinking_30d', weight_col='weight')[source]¶

Mirror the core outputs of the old 06_model_comparison.R module.

Parameters:

data (DataFrame)
outcome (str)
weight_col (str)

Return type:

dict[str, DataFrame]

morie.investigation.run_treatment_effects_analysis(data, *, treatment='cannabis_any_use', outcome='heavy_drinking_30d', covariates=None)[source]¶

Mirror the core outputs of the old 07_treatment_effects.R module.

ATE / ATT / ATC Estimators¶

All three estimands use IPW weights derived from the estimated propensity score $e_i = P(T_i=1 \mid X_i)$.

ATE – Hájek (normalised Horvitz-Thompson) estimator. See _hajek_ate() for the formula. This is preferred over the unnormalised HT estimator because self-normalisation gives better finite-sample performance (Lunceford & Davidian, 2004).

ATT – weighted mean of control outcomes under ATT weights:

\[\widehat{\text{ATT}} = \bar{Y}_1 - \frac{\sum_{T_i=0} Y_i e_i / (1-e_i)}{\sum_{T_i=0} e_i / (1-e_i)}\]

ATC – weighted mean of treated outcomes under ATC weights:

\[\widehat{\text{ATC}} = \frac{\sum_{T_i=1} Y_i (1-e_i) / e_i}{\sum_{T_i=1} (1-e_i) / e_i} - \bar{Y}_0\]

CATE Subgroup Estimates¶

Conditional average treatment effects (CATE) for each subgroup level are computed with the Hájek estimator applied within each subgroup. Raw (unadjusted) within-subgroup means are not used because they are confounded by covariate imbalance within the subgroup. Propensity scores are re-used from the global model; this is a standard stratified-IPW approach for subgroup CATE estimation.

References

Lunceford, J. K., & Davidian, M. (2004). Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study. Statistics in Medicine, 23(19), 2937–2960. https://doi.org/10.1002/sim.1903

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41

Parameters:

data (DataFrame)
treatment (str)
outcome (str)
covariates (list[str] | None)

Return type:

dict[str, DataFrame]

Data capsules¶

Provenance records, results manifests, signed bundles and drift checks in the format rmoriebricklayer verifies (vrfcps(), cpsbndlvrf(), mnfrcmp(), cpsdrft()) and the rootcoder007/rmbl-action GitHub Action checks. morie signs and verifies with hmac (a shared secret), ML-DSA (ML-DSA-44, ML-DSA-65, ML-DSA-87, FIPS 204) and XMSS (xmss-sha256, RFC 8391), with the keys, signatures and attestations of fipskg(), pqckg(), cpssign() and cpsatst() byte for byte, so a bundle signed on either side verifies on the other; an XMSS key shares rmoriebricklayer’s used-index record and never signs one index twice. Bundles signed with SLH-DSA keys are verified by the R package and the action.

from morie import capsule

key = capsule.keygen("ML-DSA-65")          # or "xmss-sha256", height=10
capsule.write_key(key, "key.json")         # SECRET; capsule.public_key(key) is not
capsule.bundle("dir", capsule.read_manifest("dir/manifest.json"), key)
capsule.verify_bundle("dir/capsule_bundle.json", "dir", public_key=capsule.public_key(key))

Reproducible data capsules, in the format rmoriebricklayer verifies.

A capsule is a directory holding a data file pinned by a provenance record (data_provenance.json), optionally a results manifest (manifest.json) and a signed bundle (capsule_bundle.json). Everything written here is read and verified by the R package rmoriebricklayer (vrfcps(), cpsbndlvrf(), mnfrcmp(), cpsdrft()) and by the GitHub Action rootcoder007/rmbl-action, and capsules written by them are verified here.

Provenance: write(), load(), pin(), verify().
Manifests: manifest(), record(), write_manifest(), read_manifest(), canon(), digest(), compare().
Signing: keygen(), public_key(), read_key(), write_key(), sign(), verify_signature(), attest(), bundle(), read_bundle(), verify_bundle(), with a shared secret (hmac), an ML-DSA key (ML-DSA-44, ML-DSA-65, ML-DSA-87) or an XMSS key (xmss-sha256), byte-compatible with fipskg(), pqckg() and cpssign(). Bundles signed with SLH-DSA keys are verified by the R package and the action, not here.
Drift: drift(), benford().

morie.capsule.attest(manifest, key, note=None, context=None, prehash='none')[source]¶

An attestation over digest(manifest), as cpsatst() writes it, in its field order: digest, scheme, public, context, prehash, note, signed_utc, signature. The signature is over the SHA-256 (hex) of the canonical JSON of the first seven fields. context and prehash are bound into an ML-DSA signature (XMSS records them and its signature does not use them); an hmac attestation takes neither.

Parameters:

note (str | None)
context (str | None)
prehash (str)

Return type:

morie.capsule.benford(x)[source]¶

First-digit test against Benford’s law, as benfordtst(): counts, expected, proportion (digits "1"..``”9”), ``statistic, df (8), p_value, logpvalue and n, over the finite non-zero values.

Return type:: dict

morie.capsule.bundle(dir, manifest, key, files=None, note=None, path=None, context=None, prehash='none')[source]¶

Sign a whole capsule into capsule_bundle.json, as cpsbndl() does.

Hashes files (default: every regular file under dir, dotfiles included, except the bundle itself), sorted, records digest(manifest) and the time, and signs the lot with attest() under key (a shared secret, or an ML-DSA or XMSS key from keygen(); context and prehash as there). The file has the R package’s layout (files, manifest_digest, bundled_utc, attestation) and is written to path (default dir/capsule_bundle.json). Returns the bundle. An XMSS key is advanced in place: save it (write_key()) before signing again.

rmoriebricklayer::cpsbndlvrf() verifies such a bundle: an ML-DSA or XMSS one under the public key it carries (keyexp = pins it), an hmac one with the shared key (key =, rmoriebricklayer >= 1.0.0 (srr)). verify_bundle() checks it here.

>>> import tempfile, pathlib
>>> d = pathlib.Path(tempfile.mkdtemp())
>>> _ = (d / "d.csv").write_text("x\n1\n")
>>> k = keygen("ML-DSA-44")
>>> b = bundle(d, {"meta": {}, "results": []}, k)
>>> b["attestation"]["scheme"], verify_bundle(str(d / BUNDLE), d, public_key=public_key(k))["ok"]
('ML-DSA-44', True)

Parameters:

note (str | None)
context (str | None)
prehash (str)

Return type:

morie.capsule.canon(man)[source]¶

The canonical one-line JSON of a manifest (or any JSON value), as mnfcanon().

Return type:: str

morie.capsule.check_attestation(att, manifest, key=None, public_key=None)[source]¶

The rows cpschkatst() reports: attestation_complete, manifest_digest, key_as_expected (with public_key: the expected public hex, or a key object) and signature. An hmac signature needs the shared key; an ML-DSA or XMSS one is verified under the public key the attestation carries (pin it with public_key).

Parameters:: att (dict)
Return type:: dict

morie.capsule.compare(manifest, data, stats, tol=None)[source]¶

Recompute recorded statistics against data, as mnfrcmp() does.

manifest is a manifest or the path of a manifest file.

stats maps result names to functions of data. Each row has name, recorded, recomputed, delta, tol, status (MATCH, DIFFER, NOT_RECORDED or ERROR) and detail. Numbers match when equal (NaN with NaN too) or at most tol apart (else the result’s recorded tol, else 0); anything else is compared as text. unchecked lists recorded names not recomputed; ok needs every row MATCH and nothing unchecked.

Parameters:

stats (dict)
tol (float | None)

Return type:

morie.capsule.digest(man)[source]¶

SHA-256 of canon(), as mnfdgst(): 64 lower-case hex characters.

Return type:: str

morie.capsule.drift(ref, current, provenance_file='data_provenance.json', alpha=0.01, psi_threshold=0.25, psi_min_n=1000, bins=10, id_levels=100, seed=1)[source]¶

Compare a reference CSV with a current one column by column, as cpsdrft().

ref is a capsule directory (its pinned CSV, resource.filename of provenance_file), a CSV path, or a {column: (type, values)} frame; current a CSV path or a frame. Returns columns (one row per shared column: column, type, statistic, p_value, psi, jsdiv, unseen_share, drifted, note), added, removed, nref, ncur, alpha and anydrft, the fields of cpsdrft(). A column has drifted when its p-value is below alpha or, with at least psi_min_n values on both sides, PSI exceeds psi_threshold; a categorical column with more than id_levels distinct values is an identifier and gets no verdict.

Parameters:

provenance_file (str)
alpha (float)
psi_threshold (float)
psi_min_n (int)
bins (int)
id_levels (int)
seed (int)

Return type:

morie.capsule.key_id(key)[source]¶

The attestation’s public field. For an HMAC key, HMAC-SHA-256 of "public" under the key: it names the key (so a verifier can pin it) without revealing it. For an ML-DSA key its public key, for an XMSS key root || pubseed.

Return type:: str

morie.capsule.keygen(scheme='ML-DSA-65', height=10)[source]¶

A new post-quantum signing key, as fipskg(scheme) or, for "xmss-sha256", pqckg(height) returns it (a dict with the R object’s fields). Save it with write_key(); give others public_key().

>>> k = keygen("ML-DSA-44")
>>> s = sign("a manifest digest", k)
>>> s["scheme"], verify_signature("a manifest digest", s, public_key(k))
('ML-DSA-44', True)

Parameters:

scheme (str)
height (int)

Return type:

morie.capsule.load(path)[source]¶

Read and validate a provenance file; raises ValueError listing every problem.

Valid: a JSON object whose resource has a non-empty filename and a 64-hex sha256, with size_bytes and row_count_data_rows whole numbers >= 0 when present and script.sha256 64 hex when present – the rules of the rmbl-action’s capsule.schema.json.

Return type:: dict

morie.capsule.manifest(meta=None, checks=None, path=None, env=True, canonical=False)[source]¶

A new results manifest, as mkmnf() makes one, optionally filled and written.

meta is the run’s metadata (project, author, …). synthetic is set to True when meta says so or the environment variable BRICKLAYER_SYNTHETIC is set to anything but 0/false/no/off, and is written only when true. Empty meta and results are kept as [], which is how R writes an empty list, so the digest of an empty manifest is the one R computes. With env the Python session is recorded under environment (python_version, platform, os, captured_utc, packages). checks is a list of dicts with name, observed, expected and optionally tol and group, each added with record(); with path the manifest is written by write_manifest() (canonical as there).

Parameters:

meta (dict | None)
checks (list | None)
env (bool)
canonical (bool)

Return type:

morie.capsule.pin(dir, provenance_file='data_provenance.json')[source]¶

Re-pin a capsule’s provenance to the files as they are now.

Updates resource.sha256, size_bytes and row_count_data_rows (for a CSV) and, when a script is named, script.sha256; every other field is kept. Returns the updated record.

Parameters:: provenance_file (str)
Return type:: dict

morie.capsule.public_key(key)[source]¶

The public half of a key: fipspubkey() / signpubkey().

Parameters:: key (dict)
Return type:: dict

morie.capsule.read_bundle(path)[source]¶

A bundle file read back; ValueError when it is not one.

Return type:: dict

morie.capsule.read_key(path)[source]¶

A key object read from a JSON key file.

Return type:: dict

morie.capsule.read_manifest(path)[source]¶

A manifest file read back (any JSON object).

Return type:: dict

morie.capsule.record(manifest, name, observed, expected, tol=0.0001, group='general', synthetic=False)[source]¶

Add one cross-check to manifest (in place, and returned), as recrd() does.

The status is INFO on synthetic data (synthetic or meta.synthetic), PASS when both values are numbers at most tol apart, DIFFER when further apart, and INFO otherwise. Recording an existing name replaces it.

Parameters:

manifest (dict)
name (str)
tol (float)
group (str)
synthetic (bool)

Return type:

morie.capsule.sha256file(path)[source]¶

SHA-256 of a file’s bytes, 64 lower-case hex characters (sha256file()).

Return type:: str

morie.capsule.sign(message, key, scheme=None, context=None, prehash='none')[source]¶

The object cpssign(message, key, ...) returns.

With a shared secret (a string or bytes), {"scheme": "hmac", "signature": <64 hex>}: HMAC-SHA-256 of message under key. With a key from keygen() (or a key file of fipskg() / pqckg()), its scheme’s signature; context (a string or bytes, at most 255 bytes) and prehash ("none", "sha256", "sha512", …) apply to ML-DSA. An XMSS key’s next_index is advanced in place: save the key after signing. scheme, when given, must be the key’s.

>>> sign("what do ya want for nothing?", "Jefe")["signature"][:16]
'5bdcc146bf60754e'

Parameters:

scheme (str | None)
prehash (str)

Return type:

morie.capsule.verify(dir, provenance_file='data_provenance.json', data_file=None, manifest_file=None, script_file=None)[source]¶

Re-verify a capsule directory, check by check, as vrfcps() does.

Returns {"ok", "checks", "required", "missing"}; checks is a list of {"check", "ok", "detail"} with the R package’s names, in its order: provenance_readable, data_present, data_sha256, data_size_bytes (when recorded), data_readable, data_row_count (when recorded), schema_valid (when a schema is recorded), manifest_consistent (with a manifest), script_sha256 (with a script) and data_not_synthetic. A check that cannot be made is a failed row; ok needs every row to pass and every required check to be made. Offline, writes nothing; the only error is a cpsdir that is not a directory.

Parameters:: provenance_file (str)
Return type:: dict

morie.capsule.verify_bundle(bundle, dir, manifest=None, key=None, public_key=None)[source]¶

Check a bundle against a directory, as cpsbndlvrf() does.

bundle is a path or a bundle from read_bundle(). Rows: file:<path> per listed file (digest), no_unlisted_files, the attestation:* rows of check_attestation() and, with manifest, manifest_digest. key is the shared secret of an hmac bundle; public_key (hex or a key object) pins the key any bundle must be signed with. Returns {"ok", "checks"}.

Return type:: dict

morie.capsule.verify_signature(message, sig, key, context=None, prehash=None)[source]¶

True when sig (from sign() or cpssign()) is valid for message under key: the shared secret for hmac (compared in constant time), else the public (or signing) key, with the context it was signed with. A malformed signature is False, as in cpsvrf().

Parameters:: prehash (str | None)
Return type:: bool

morie.capsule.write(dir, data_file, url=None, name=None, retrieved=None, provenance_file='data_provenance.json', table=None, script_file=None)[source]¶

Pin a data file of a capsule directory in a provenance record.

Writes provenance_file in dir with name (default: the data file’s name without .csv), source.url (omitted when url is None) and source.retrieved (default: today, YYYY-MM-DD), and resource.filename, sha256, size_bytes and, for a CSV, row_count_data_rows (data rows as read.csv() counts them): the fields verify() and rmoriebricklayer::vrfcps() check. table is stored as given; script_file, a file in the capsule, is pinned as script.filename and script.sha256. data_file and script_file are relative to dir and must stay inside it. For a CSV beside its provenance the bytes are those of the rmbl-action’s own examples. Returns the record.

Parameters:

data_file (str)
url (str | None)
name (str | None)
retrieved (str | None)
provenance_file (str)
table (dict | None)
script_file (str | None)

Return type:

morie.capsule.write_key(key, path)[source]¶

Write a key object as JSON, readable by the owner only, replacing path atomically (an XMSS key’s next_index must never go back).

Parameters:: key (dict)
Return type:: None

morie.capsule.write_manifest(manifest, path, canonical=False)[source]¶

Write manifest to path as wrmnfjs() does: pretty, or with canonical the one line digest() hashes. A synthetic manifest always says so.

Parameters:

manifest (dict)
canonical (bool)

Return type:

Datasets, ingestion and data hygiene¶

morie.datasets — one-call dataset loaders.

This module wraps the lower-level morie.ingest adapters so that fresh users can pull common Canadian sociolegal datasets with a single function call:

>>> import morie.datasets as md
>>> df = md.tps_major_crime(year=2024)
>>> df.shape
(..., 29)

>>> df = md.cpads()                   # synthetic fallback if no real PUMF
>>> df.shape
(1200, 11)

>>> idx = md.siu_director_reports()   # may be empty; see siu module docstring
>>> text = md.siu_report_text("https://www.siu.on.ca/.../22-OFD-001.pdf")

The functions here are stable user-facing API. The underlying adapters in morie.ingest may change layer URLs / endpoint shapes as portals reorganise, but the names here will not.

Every function returns a plain pandas.DataFrame (or, for text-mining helpers, a dict). No bespoke result type, no required configuration, no manual paging.

morie.datasets.tps_major_crime(*, year=None, max_features=None, include_geometry=False, offline=False)[source]¶

Toronto Police Service “Major Crime Indicators” feed.

Parameters:

year (int | None) – If set, returns only rows with OCC_YEAR == year. Default None returns the entire feed (~700K rows; large).
max_features (int | None) – Cap on returned rows. Useful for inspection.
include_geometry (bool) – If True, includes geom_x/geom_y (web-mercator) columns.
offline (bool) – If True, returns the bundled 500-row synthetic frame instead of hitting the live ArcGIS endpoint. Use this in CI, on air-gapped hosts, or for fast deterministic tests.

Returns:

pd.DataFrame with TPS’s documented columns (REPORT_DATE, OCC_DATE,
OFFENCE, CSI_CATEGORY, etc.) — see
https (//data.torontopolice.on.ca/ for the schema.)

Return type:

DataFrame

morie.datasets.tps_shootings(*, year=None, max_features=None)[source]¶

TPS Shooting and Firearm Discharges feed.

Parameters:

year (int | None)
max_features (int | None)

Return type:

DataFrame

morie.datasets.tps_homicide(*, year=None, max_features=None)[source]¶

TPS Homicides feed.

Parameters:

year (int | None)
max_features (int | None)

Return type:

DataFrame

morie.datasets.tps_layers()[source]¶

List all TPS open-data layers shipped with morie.

Return type:: DataFrame

morie.datasets.cpads()[source]¶

Canadian Postsecondary Alcohol and Drug Use Survey microdata.

Resolves in this order:

The real Statistics Canada PUMF file if it’s available at the documented project-root path (data/datasets/oc/CPADS/…).
The 1,200-row shipped-in-wheel synthetic frame with the correct schema but random values.

A warning fires when the synthetic frame is used so toy outputs aren’t mistaken for findings about the real population.

Returns:

pd.DataFrame with morie’s canonical CPADS analysis columns
(weight, alcohol_past12m, heavy_drinking_30d, ebac_tot, ebac_legal,
cannabis_any_use, age_group, gender, province_region, mental_health,
physical_health).

Return type:

DataFrame

morie.datasets.otis_a01(*, offline=True)[source]¶

Ontario Tracking Information System A01-RCDD (restrictive-confinement) correctional-placement records.

Returns the bundled 800-row synthetic frame by default — real OTIS data is FOI-only and cannot be shipped publicly. Replace with your own pull when you have access.

Schema (canonical morie names):: person_id, obs_date, fiscal_year, region, alert_complexity, volatility_movement, cell_disposition, restrictive_confinement_days, age_band.

Parameters:: offline (bool)
Return type:: DataFrame

morie.datasets.siu_director_reports()[source]¶

SIU director’s-reports index: one row per reviewed report.

The SIU site renders its case list with JavaScript since 2025, so the index comes from the reviewed corpus shipped by rmoriedata (5,157 reports): case number, report id, source URL and the key dates. Pass a report URL to siu_report_text() for the full text.

Return type:: DataFrame

morie.datasets.siu_report_text(url=None, *, offline=False)[source]¶

Download an SIU director’s-report PDF and return its plain text.

Parameters:

url (str | None) – Direct PDF URL of a single report. Required unless offline.
offline (bool) – If True, returns a bundled synthetic SIU director’s-report text (case 24-OFD-001). Useful for testing without hitting the SIU website.

Return type:

morie.datasets.siu_report_fields(text_or_url)[source]¶

Pull structured fields (sections, conclusion, report_id, date) from a report.

Pass either the text from siu_report_text() (re-uses it) or a PDF URL (fetches first).

Parameters:: text_or_url (str)
Return type:: dict[str, Any]

morie.datasets.chicago_crime(*, year=None, max_features=None, offline=False)[source]¶

City of Chicago “Crimes — 2001 to Present” feed.

Schema (Socrata column names, preserved verbatim):: id, case_number, date, block, iucr, primary_type, description, location_description, arrest, domestic, beat, district, ward, community_area, fbi_code, x_coordinate, y_coordinate, year, updated_on, latitude, longitude.

Parameters:

year (int | None) – Server-side filter to a single year (e.g. 2024). None returns the entire feed (millions of rows; large).
max_features (int | None) – Cap on returned rows. Useful for inspection.
offline (bool) – If True, returns the bundled tiny synthetic Chicago frame instead of hitting the live Socrata endpoint. Use this in CI, on air-gapped hosts, or for fast deterministic tests.

Return type:

DataFrame

Notes

The live endpoint is https://data.cityofchicago.org/resource/ijzp-q8t2.json (resource id ijzp-q8t2). Anonymous calls share a throttled rate-limit pool; for production pulls register a Socrata application token and pass it via the lower-level morie.ingest.chicago.fetch_crime() helper.

morie.datasets.nyc_stop_and_frisk(*, year=None, max_features=None, offline=False)[source]¶

NYPD Stop, Question and Frisk (SQF) microdata via NYC OpenData.

NYC publishes SQF as a separate Socrata resource per release year. Pass year to pick which release to pull; None defaults to the most-recent registered year (currently 2024).

Parameters:

year (int | None) – Release year. Must be a key of the internal resource map (currently 2022–2024 inclusive). None → most recent.
max_features (int | None) – Cap on returned rows.
offline (bool) – If True, returns the bundled synthetic NYC SQF frame.

Return type:

DataFrame

Notes

Schema is NOT normalised across years — NYPD has rotated field names between releases. Inspect df.columns to see the exact schema of the requested release.

morie.datasets.bigquery(project, dataset, table, *, where=None, limit=None, select='*', billing_project=None)[source]¶

Pull a BigQuery table (or filtered slice) as a DataFrame.

Lightweight wrapper around morie.ingest.bigquery for the common “give me this public table, optionally filtered” case.

Parameters:

project (str) – Fully-qualified table, e.g. project="bigquery-public-data", dataset="chicago_crime", table="crime".
dataset (str) – Fully-qualified table, e.g. project="bigquery-public-data", dataset="chicago_crime", table="crime".
table (str) – Fully-qualified table, e.g. project="bigquery-public-data", dataset="chicago_crime", table="crime".
where (str | None) – Raw SQL WHERE clause (without the leading WHERE).
limit (int | None) – Optional LIMIT.
select (str) – Projection list; defaults to *.
billing_project (str | None) – GCP project to bill the query to. Public datasets cost the caller’s project, not the dataset owner. None uses the ADC-discovered project.

Return type:

DataFrame

Notes

Requires the bigquery extra:

pip install 'morie[bigquery]'

Authenticates via Application Default Credentials (ADC), matching the HADES-LLM Pi-rendered architecture.

morie.datasets.ckan_search(portal, query, *, rows=50)[source]¶

Search a CKAN open-data portal by free-text query.

Examples of portals:

https://open.canada.ca/data — Government of Canada
https://data.gov.uk — UK government
https://data.europa.eu — European Union
https://data.ontario.ca — Ontario provincial

Parameters:

portal (str)
query (str)
rows (int)

Return type:

DataFrame

morie.datasets.ckan_package(portal, package_id)[source]¶

Pull every CSV resource of a CKAN package as a dict of DataFrames.

Returns a mapping {resource_name: DataFrame}.

Parameters:

portal (str)
package_id (str)

Return type:

dict[str, DataFrame]

morie.datasets.nibrs(*, year=None, max_features=None, state=None, offense=None, api_key=None, offline=False)[source]¶

FBI National Incident-Based Reporting System (NIBRS) offence events.

Pulls offence-event records from the FBI Crime Data Explorer (https://api.usa.gov/crime/fbi/cde/). An API key is required; sign up free at https://api.data.gov/signup/ and pass via api_key= or export FBI_CDE_API_KEY.

Parameters:

year (int | None) – Reporting year (required unless offline=True).
max_features (int | None) – Cap on returned rows. Recommended for the national feed.
state (str | None) – Two-letter US state code (e.g. "GA"). None = national.
offense (str | None) – NIBRS offence slug (e.g. "aggravated-assault"). None = all.
api_key (str | None) – FBI CDE API key. Falls back to $FBI_CDE_API_KEY.
offline (bool) – If True, returns a bundled synthetic NIBRS frame for CI/airgap.

Returns:

pd.DataFrame with one row per offence-event; nested NIBRS sub-objects
flattened to dotted-key columns (offense.code, victim.age).

Return type:

DataFrame

morie.datasets.namus_missing_persons(*, state=None, max_features=None, offline=False)[source]¶

NamUs missing-persons case metadata.

Pulls case metadata from the National Missing and Unidentified Persons System (https://www.namus.gov/). No API key required.

Parameters:

state (str | None) – Two-letter US state code (e.g. "CA"). None = national.
max_features (int | None) – Cap on returned rows.
offline (bool) – If True, returns a bundled synthetic NamUs frame.

Returns:

pd.DataFrame with columns (case_number, state, county, dlc_date, sex,)
race, age_min, age_max, height_cm_min, height_cm_max, weight_kg_min,
weight_kg_max, first_name, last_name, city, circumstances.

Return type:

DataFrame

morie.datasets.nist_rds(*, dataset_id=None, query=None, max_features=None, offline=False)[source]¶

NIST Reference Datasets catalog metadata.

Returns the catalog records for NIST RDS holdings (CSAFE, NSRL, …). The reference datasets themselves are multi-gigabyte and served on dedicated download servers — pull them out-of-band using the landing_page column.

Parameters:

dataset_id (str | None) – Specific NIST RDS / EDI id; returns a single-row frame.
query (str | None) – Free-text search over title/description/keyword.
max_features (int | None) – Cap on returned rows.
offline (bool) – If True, returns a bundled synthetic NIST RDS catalog frame.

Returns:

pd.DataFrame with columns (dataset_id, title, description, publisher,)
issued, modified, keyword, landing_page, size_bytes, license.

Return type:

DataFrame

Victorian crime data (Crime Statistics Agency, Victoria, Australia).

The agency publishes one multi-sheet .xlsx workbook per topic each quarter. The files are not bundled – roughly 65 MB per release, and refreshed quarterly – so the catalog below carries the download URLs and the loader caches to disk, mirroring R/datasets_vic.R.

Licence: Creative Commons Attribution 4.0 International (CC BY 4.0). Attribution: Crime Statistics Agency, Victoria. Source: https://www.crimestatistics.vic.gov.au/crime-statistics/

latest-victorian-crime-data/download-data

morie.datasets_vic.vic_catalog()[source]¶: Return the catalog of published workbooks as a list of dicts.

morie.datasets_vic.vic_cache_dir(cache_dir=None)[source]¶: Cache directory for downloaded workbooks, created if absent.

morie.datasets_vic.vic_table(key, table=1, cache_dir=None, offline=True)[source]¶

Read one sheet of a Victorian workbook.

table is an integer table number (1 gives “Table 01”) or a sheet name. With offline=True (the default) nothing is downloaded: a cached file is read, otherwise an empty frame comes back, so tests never touch the network.

morie.datasets_vic.vic_sheets(key, cache_dir=None)[source]¶: Sheet names in a cached workbook, or [] when it is not cached.

morie.datasets_vic.vic_offence_trend(data)[source]¶: Incidents by year and offence division, with the change over the observed window. Mirrors morie_vic_offence_trend.

morie.datasets_vic.vic_lga_rates(data, year=None, top=10)[source]¶: Local government areas ranked by offence rate.

morie.datasets_vic.vic_indigenous_ratio(data, by='Offence Division', year=None)[source]¶

Indigenous vs non-Indigenous victim COUNT ratio, per group.

Not a population-adjusted rate ratio: the published table carries counts only. The table’s “Total People” rows are excluded – folding them into the non-Indigenous count double-counts every victim – and one year is used at a time.

morie.dataset_dictionary — unified parser for dataset documentation.

Real-world open-data releases ship their schemas in a half-dozen different formats, all describing the same logical thing: “what columns does this CSV have, what type, what valid values, what does each one mean in English and in French.” This module reads those sources and projects them onto a single in-memory schema model so the rest of morie can validate / describe / cross-reference without caring whether the source was Markdown, multi-sheet XLSX, or CKAN JSON sidecar.

Sources currently supported¶

Markdown: OTIS_DATA_DICTIONARY.md style — one file containing multiple ### N. Dataset Title sections, each with a **Dataset Name**: \`b01df\` line and a **Variables** bullet list.
XLSX: official Ontario open-data dictionary releases — multi-sheet workbooks where one or two sheets carry English column definitions and one or two carry French. The exact sheet layout differs by publication; this module auto-detects the header row.
CKAN sidecar JSON: the response payload of CKAN’s datastore_search endpoint, saved next to a CSV. The fields array carries column names + datastore types + optional info.notes; records carries the data.

Public API¶

ColumnSpec, DatasetSchema — the in-memory model
parse_markdown_dictionary(), parse_xlsx_dictionary(), parse_ckan_sidecar() — low-level parsers
merge_schemas() — combine English + French (or MD + XLSX) into one bilingual schema
validate_dataframe_against_schema() — cross-check a loaded DataFrame against a parsed schema; returns a RichResult
load_otis_dictionary(), load_arsau_dictionary() — high-level convenience that resolves the right files via morie._datapaths.resolve_data_dir() and merges MD + XLSX (OTIS) or English + French XLSX sheets (ARSAU)

Notes

A handful of upstream artefacts — particularly in the Ontario open-data releases — contain small annotation bugs that we surface through validate_dataframe_against_schema() rather than fix:

OTIS c05_..._race_by_region is the dictionary entry’s filename but the on-disk CSV is c05_..._religion_by_region (variable list confirms it’s religion, not race; the dict filename is a typo)
OTIS Data Dictionary_French rows often re-state the type as “Booléen” where the English type was “Texte” (translation drift)
ARSAU 2020-2022 File Name cells carry a leading space (”` UseOfForce_DetailedDataset`”) — stripped here on read.

The point is not to fix them; the point is to make sure our parser notices and reports them when it cross-validates against a loaded DataFrame.

class morie.dataset_dictionary.ColumnSpec(name, dtype, description_en=None, description_fr=None, valid_values=None, nullable=True, source_notes=None, raw_type=None)[source]¶

Bases: object

A single column in a dataset, projected to the canonical model.

Parameters:

name (str)
dtype (str)
description_en (str | None)
description_fr (str | None)
valid_values (tuple[str, ...] | None)
nullable (bool)
source_notes (str | None)
raw_type (str | None)

name: str¶: Canonical column name as it appears in the data file.

dtype: str¶

Type:: One of

description_en: str | None = None¶

description_fr: str | None = None¶

valid_values: tuple[str, ...] | None = None¶: Closed-set categorical values, if any. None for free-form columns.

nullable: bool = True¶

source_notes: str | None = None¶: Free-form upstream notes (CKAN info.notes, Additional Notes column, etc.). Retained verbatim for debugging cross-source disagreements.

raw_type: str | None = None¶: The exact upstream type string before canonicalisation — useful for surfacing the OTIS Texte → Booléen translation-drift quirk.

localised_description(language='en')[source]¶

Return the description in the requested language, falling back gracefully.

Parameters:: language (str)
Return type:: str

class morie.dataset_dictionary.DatasetSchema(dataset_name, source_path, source_kind, language, columns=())[source]¶

Bases: object

The schema of a single dataset — an ordered tuple of ColumnSpec.

Parameters:

dataset_name (str)
source_path (str)
source_kind (str)
language (str)
columns (tuple[ColumnSpec, ...])

dataset_name: str¶: Canonical dataset identifier (e.g. b01df, uof_main_records).

source_path: str¶: File from which this schema was parsed.

source_kind: str¶: xlsx | markdown | ckan_sidecar

language: str¶: en | fr | bilingual

columns: tuple[ColumnSpec, ...] = ()¶

find_column(name)[source]¶

Return the ColumnSpec whose name matches name (case-insensitive).

Parameters:: name (str)
Return type:: ColumnSpec | None

describe(language='en')[source]¶

Multi-paragraph plain-language description of this schema.

Returns:: Human-readable summary: dataset name + source + column count + per-column one-line entries with name, type, and localised description.
Return type:: str
Parameters:: language (str)

property column_names: tuple[str, ...]¶

morie.dataset_dictionary.parse_markdown_dictionary(path)[source]¶

Parse a OTIS-style Markdown data dictionary.

The Markdown is expected to be one file with multiple ### N. <Title> sections, each containing at least:

a **Dataset Name**: \`<id>\` line
a **Variables**: heading
a bullet list of variable definitions

Parameters:: path (str | Path) – Path to the .md file.
Returns:: Keyed by dataset_name.
Return type:: dict[str, DatasetSchema]

Examples

>>> schemas = parse_markdown_dictionary("OTIS_DATA_DICTIONARY.md")
>>> 'b01df' in schemas
True

morie.dataset_dictionary.parse_xlsx_dictionary(path, *, header_row=None, sheet_filter=None)[source]¶

Parse a multi-sheet XLSX data dictionary.

Auto-detects English and French sheets by name patterns; merges them per dataset so each returned DatasetSchema has both description_en and description_fr populated where possible.

Parameters:

path (str | Path) – Path to the .xlsx file.
header_row (int | None, default None) – Force a specific 0-indexed header row. None means auto-detect by scanning the first ~12 rows for “Variable Name” / “Nom de la variable”.
sheet_filter (iterable of str, optional) – Only parse sheets whose name contains one of these substrings. Useful when the workbook also contains diagrams or notes sheets that should be skipped (e.g. the Relationships-Relations sheet in the 2024 ARSAU dict).

Returns:

Keyed by normalised dataset id. Each schema is bilingual when both an English and a French sheet describe the same dataset.

Return type:

dict[str, DatasetSchema]

Examples

>>> schemas = parse_xlsx_dictionary(
...     "/path/datadictionary_2023uof_en_fr.xlsx"
... )
>>> 'uof_main_records' in {k.lower() for k in schemas}
True

morie.dataset_dictionary.parse_ckan_sidecar(path)[source]¶

Parse a CKAN datastore_search JSON sidecar saved alongside a CSV.

The expected payload shape is:

{
    "fields": [
        {"id": "_id", "type": "int"},
        {"id": "col_a", "type": "text", "info": {"notes": "..."}},
        ...
    ],
    "records": [...],
}

Some CKAN responses wrap the above under a "result" key; this parser handles both shapes.

Parameters:: path (str | Path) – Path to the JSON file. The filename (without .json) is used as dataset_name.
Return type:: DatasetSchema

morie.dataset_dictionary.merge_schemas(primary, secondary)[source]¶

Merge two schemas describing the same dataset.

primary wins on conflict; secondary fills missing fields (description_fr, source_notes, valid_values, etc.). The merged schema’s language is "bilingual" if the two inputs differ.

Column matching is case-insensitive on name. Columns present in only one of the two schemas are kept in their original position.

Parameters:

primary (DatasetSchema)
secondary (DatasetSchema)

Return type:

DatasetSchema

morie.dataset_dictionary.validate_dataframe_against_schema(df, schema, *, language='en')[source]¶

Cross-check a loaded DataFrame against a parsed schema.

Parameters:

df (pandas.DataFrame) – The DataFrame to check.
schema (DatasetSchema) – The schema to check against.
language (str, default "en") – Language for the interpretation block.

Returns:

payload contains:

is_valid (bool) — True iff missing_columns + dtype_mismatches empty
missing_columns (list[str]) — in schema, absent in df
extra_columns (list[str]) — in df, absent in schema
dtype_mismatches (list[dict]) — column with surprising pandas dtype
n_columns_schema / n_columns_df / n_rows_df

Return type:

morie.dataset_dictionary.load_otis_dictionary(data_dir=None)[source]¶

Load the OTIS data dictionary by combining its Markdown + XLSX sources.

Resolves the OTIS data directory via morie._datapaths.resolve_data_dir(), so the file works regardless of where the user has stored their OTIS download — as long as it’s discoverable through the documented cascade (data_dir argument → MORIE_OTIS_DIR env var → MORIE_DATA_DIR + otis/ → user cache → bundled fixture).

The Markdown carries the canonical English prose; the XLSX is authoritative for column dtypes and valid values, plus carries French translations. The returned schemas are bilingual where both sources agree on the dataset id.

Parameters:: data_dir (str | Path | None, default None) – Override for the OTIS data directory. None triggers the env-var cascade.
Returns:: Keyed by OTIS dataset id (b01df, c12, etc.).
Return type:: dict[str, DatasetSchema]

morie.dataset_dictionary.load_arsau_dictionary(year_or_range, *, data_dir=None)[source]¶

Load the ARSAU data dictionary for the given year (or year range).

year_or_range must be one of: "2020-2022", "2023", "2024".

The right XLSX in ARSAU/Suppl/ is selected and parsed bilingually.

Parameters:

year_or_range (str)
data_dir (str | Path | None)

Return type:

dict[str, DatasetSchema]

morie.ingest — adapters for open-data portals.

Currently shipped:

morie.ingest.ckan — CKAN-portal package_search / package_show /
resource fetch, with auto-paging and pandas-DataFrame return.
morie.ingest.tps — Toronto Police Service open-data
JSON feeds (ArcGIS Hub).
morie.ingest.siu — Special Investigations Unit director’s
reports (PDF index + text extraction).
morie.ingest.chicago — City of Chicago Socrata feeds, plus a
generic fetch_socrata() helper that works against any Socrata portal (NYC OpenData, data.seattle.gov, …).
morie.ingest.bigquery — Google BigQuery public-data adapter
(lazy import; needs the bigquery extra).
morie.ingest.a2aj — A2AJ Canadian Legal Data: court and
tribunal decisions, statutes and regulations (api.a2aj.ca, Hugging Face Parquet), citation network edges.
morie.ingest.canlii — CanLII REST API (needs a free key):
the courts A2AJ does not carry.

Each sub-module exposes:

A thin client object (Client) for advanced usage.

One-shot helpers that take a URL or dataset id and return a pandas.DataFrame.

An optional CLI handler reachable as morie ingest <portal> ....

All sub-modules use httpx (already a hard morie dep) so users need nothing additional to ingest a remote feed once morie is installed. morie.ingest.bigquery is the one exception: it lazy-imports google-cloud-bigquery and asks for the bigquery extra at first call.

A2AJ Canadian Legal Data client.

Access to Algorithmic Justice (https://a2aj.ca) publishes Canadian court and tribunal decisions plus federal and provincial statutes and regulations through three open channels, documented at https://github.com/a2aj-ca/canadian-legal-data:

a REST API at https://api.a2aj.ca (coverage / search / fetch);
Hugging Face datasets a2aj/canadian-case-law and a2aj/canadian-laws, one Parquet file per court or law type;
the same Parquet files by direct download.

Every text is an unofficial copy; each record carries an upstream_license field that may restrict commercial use.

Quick usage¶

>>> from morie.ingest import a2aj
>>> a2aj.coverage("cases")[["dataset", "number_of_documents"]]
>>> hits = a2aj.search("roncarelli", search_type="name", dataset="SCC")
>>> doc = a2aj.fetch("2023 SCC 17", end_char=400)
>>> sct = a2aj.load("SCT", columns=["citation_en", "cases_cited_en"])
>>> edges = a2aj.citation_edges(sct)

The R arm exposes the same surface as morie_ingest_a2aj_*.

exception morie.ingest.a2aj.A2AJError[source]¶

Bases: RuntimeError

An A2AJ API call returned an HTTP error or a non-JSON body.

morie.ingest.a2aj.coverage(doc_type='cases', *, timeout=60.0, user_agent='morie/python (+https://github.com/rootcoder007/morie)', transport=None)[source]¶

Every court, tribunal or law collection in the corpus.

Returns a DataFrame with dataset, description_en, description_fr, earliest_document_date, latest_document_date (both datetime.date) and number_of_documents.

Parameters:

doc_type (str)
timeout (float)
user_agent (str)
transport (BaseTransport | None)

Return type:

DataFrame

morie.ingest.a2aj.search(query, *, search_type='full_text', doc_type='cases', size=10, search_language='en', sort_results='default', dataset=None, start_date=None, end_date=None, timeout=60.0, user_agent='morie/python (+https://github.com/rootcoder007/morie)', transport=None)[source]¶

Full-text or title search; at most 50 hits, no paging.

dataset restricts to one or more dataset codes ("SCC" or ["SCC", "ONCA"]); see coverage().

Parameters:

query (str)
search_type (str)
doc_type (str)
size (int)
search_language (str)
sort_results (str)
dataset (str | list[str] | None)
start_date (str | None)
end_date (str | None)
timeout (float)
user_agent (str)
transport (BaseTransport | None)

Return type:

DataFrame

morie.ingest.a2aj.fetch(citation, *, doc_type='cases', output_language='en', section=None, start_char=0, end_char=-1, include_citations=False, citations_limit=100, citations_offset=0, timeout=60.0, user_agent='morie/python (+https://github.com/rootcoder007/morie)', transport=None)[source]¶

One document by citation, as a dict; None when not in the corpus.

Text is under unofficial_text_en / unofficial_text_fr. With include_citations=True (cases only) cases_cited_* and cases_citing_* are lists. For a court the corpus does not cover see gaps() and morie.ingest.canlii.

Parameters:

citation (str)
doc_type (str)
output_language (str)
section (str | None)
start_char (int)
end_char (int)
include_citations (bool)
citations_limit (int)
citations_offset (int)
timeout (float)
user_agent (str)
transport (BaseTransport | None)

Return type:

dict[str, Any] | None

morie.ingest.a2aj.parquet_url(dataset, doc_type='cases')[source]¶

Hugging Face download URL of one dataset’s train.parquet.

Parameters:

dataset (str)
doc_type (str)

Return type:

morie.ingest.a2aj.download(dataset, doc_type='cases', *, cache_dir=None, refresh=False, timeout=600.0, user_agent='morie/python (+https://github.com/rootcoder007/morie)', transport=None)[source]¶

Fetch one dataset’s Parquet file into the cache and return its path.

Files range from a few megabytes (small tribunals) to several hundred (the Supreme Court of Canada, the BC Supreme Court).

Parameters:

dataset (str)
doc_type (str)
cache_dir (str | Path | None)
refresh (bool)
timeout (float)
user_agent (str)
transport (BaseTransport | None)

Return type:

morie.ingest.a2aj.load(dataset, doc_type='cases', *, columns=None, cache_dir=None, refresh=False)[source]¶

Download (or reuse) one dataset and decode it with the native reader.

unofficial_text_* holds the full texts and dominates the file; pass columns to skip it. cases_cited_* and cases_citing_* come back as lists per row.

Parameters:

dataset (str)
doc_type (str)
columns (list[str] | None)
cache_dir (str | Path | None)
refresh (bool)

Return type:

DataFrame

morie.ingest.a2aj.citation_edges(x, language='en')[source]¶

Edge list (from, to) from the cases_cited_<language> field.

Accepts a DataFrame from load() or the dict from fetch() with include_citations=True. Order follows first appearance in each decision.

Parameters:

x (DataFrame | dict[str, Any])
language (str)

Return type:

DataFrame

morie.ingest.a2aj.cli(args, *, transport=None)[source]¶

morie ingest a2aj {coverage,search,fetch} ... handler.

Writes CSV to --out or stdout (fetch writes the text itself).

Parameters:

args (list[str])
transport (BaseTransport | None)

Return type:

morie.ingest.a2aj.gaps()[source]¶

Courts and tribunals the A2AJ corpus does not carry.

Maps each gap named in the project’s open issues (HRTO, BCHRT and ONSC in #4, Alberta in #3, Quebec and the remaining provinces in #1) to the CanLII database id used on canlii.org, for morie.ingest.canlii.

Return type:: DataFrame

CanLII REST API client.

The Canadian Legal Information Institute (https://www.canlii.org) publishes every Canadian court and most tribunals, which makes it the route to the decisions the A2AJ corpus does not yet carry (see morie.ingest.a2aj.gaps()): the Human Rights Tribunal of Ontario, the BC Human Rights Tribunal, the Ontario Superior Court, Alberta, Quebec and the remaining provinces. Its API (https://api.canlii.org/v1) returns metadata and the citation network, not full text; every record carries the canlii.org URL of the decision. A key is free on request from CanLII and is read from the CANLII_API_KEY environment variable when not passed explicitly.

Quick usage¶

>>> from morie.ingest import canlii
>>> canlii.databases()                       # every court and tribunal
>>> canlii.cases("onhrt", result_count=200, decision_date_after="2024-01-01")
>>> ids = canlii.case_id("2007 BCSC 1700")   # {'database_id': 'bcsc', 'case_id': '2007bcsc1700'}
>>> canlii.case(ids["database_id"], ids["case_id"])
>>> canlii.citator("csc-scc", "2008scc9", "citingCases")

The R arm exposes the same surface as morie_ingest_canlii_*.

exception morie.ingest.canlii.CanLIIError[source]¶

Bases: RuntimeError

A CanLII API call returned an HTTP error or a non-JSON body.

morie.ingest.canlii.databases(language='en', *, api_key=None, timeout=60.0, user_agent='morie/python (+https://github.com/rootcoder007/morie)', transport=None)[source]¶

Every court and tribunal: databaseId, jurisdiction, name.

Parameters:

language (str)
api_key (str | None)
timeout (float)
user_agent (str)
transport (BaseTransport | None)

Return type:

DataFrame

morie.ingest.canlii.cases(database_id, *, offset=0, result_count=100, published_before=None, published_after=None, modified_before=None, modified_after=None, changed_before=None, changed_after=None, decision_date_before=None, decision_date_after=None, language='en', api_key=None, timeout=60.0, user_agent='morie/python (+https://github.com/rootcoder007/morie)', transport=None)[source]¶

Decisions of one database, newest first; at most 10,000 per call.

Returns databaseId, caseId, title and citation.

Parameters:

database_id (str)
offset (int)
result_count (int)
published_before (str | None)
published_after (str | None)
modified_before (str | None)
modified_after (str | None)
changed_before (str | None)
changed_after (str | None)
decision_date_before (str | None)
decision_date_after (str | None)
language (str)
api_key (str | None)
timeout (float)
user_agent (str)
transport (BaseTransport | None)

Return type:

DataFrame

morie.ingest.canlii.case(database_id, case_id, *, language='en', api_key=None, timeout=60.0, user_agent='morie/python (+https://github.com/rootcoder007/morie)', transport=None)[source]¶

Metadata of one decision (url, title, citation, decisionDate, …).

Parameters:

database_id (str)
case_id (str)
language (str)
api_key (str | None)
timeout (float)
user_agent (str)
transport (BaseTransport | None)

Return type:

morie.ingest.canlii.citator(database_id, case_id, metadata_type='citedCases', *, api_key=None, timeout=60.0, user_agent='morie/python (+https://github.com/rootcoder007/morie)', transport=None)[source]¶

Cited cases, citing cases or cited legislation of one decision (English only).

Parameters:

database_id (str)
case_id (str)
metadata_type (str)
api_key (str | None)
timeout (float)
user_agent (str)
transport (BaseTransport | None)

Return type:

DataFrame

morie.ingest.canlii.legislation_databases(language='en', *, api_key=None, timeout=60.0, user_agent='morie/python (+https://github.com/rootcoder007/morie)', transport=None)[source]¶

Legislation databases: databaseId, type, jurisdiction, name.

Parameters:

language (str)
api_key (str | None)
timeout (float)
user_agent (str)
transport (BaseTransport | None)

Return type:

DataFrame

morie.ingest.canlii.legislations(database_id, *, language='en', api_key=None, timeout=60.0, user_agent='morie/python (+https://github.com/rootcoder007/morie)', transport=None)[source]¶

Statutes or regulations of one legislation database.

Parameters:

database_id (str)
language (str)
api_key (str | None)
timeout (float)
user_agent (str)
transport (BaseTransport | None)

Return type:

DataFrame

morie.ingest.canlii.legislation(database_id, legislation_id, *, language='en', api_key=None, timeout=60.0, user_agent='morie/python (+https://github.com/rootcoder007/morie)', transport=None)[source]¶

Metadata of one statute or regulation.

Parameters:

database_id (str)
legislation_id (str)
language (str)
api_key (str | None)
timeout (float)
user_agent (str)
transport (BaseTransport | None)

Return type:

morie.ingest.canlii.case_id(citation)[source]¶

CanLII database_id and case_id from a neutral citation.

"2007 BCSC 1700" names its court (BCSC) and CanLII’s case id is the citation in lower case without spaces ("2007bcsc1700"). The Supreme Court of Canada is the one court whose database id differs from its citation code ("csc-scc"). A reporter-style citation ("[1959] SCR 121") has no derivable id: both are None.

Parameters:: citation (str)
Return type:: dict[str, str | None]

morie.schema — schema-agnostic column mapping.

A morie module names canonical concepts: “weight”, “alcohol_past12m”, “heavy_drinking_30d”, etc. Users in the same field (sociolegal stats, public health, criminology) frequently have datasets that hold those concepts under different column names: wt, drink_yn, binge30, …

This module lets those datasets flow through morie modules WITHOUT forcing the user to rename their columns first.

Two paths are supported:

Explicit mapping — pass column_mapping={"my_wt": "weight", "drinks_yn": "alcohol_past12m"} and the loader renames before handing to the canonical analysis layer. Direction: user-column -> canonical-name.

Inferred mapping — call infer_mapping(df, canonical=CPADS_REQUIRED_VARIABLES)() and morie attempts a fuzzy + dtype-aware match between the user’s columns and the canonical names. Confidence scores are returned so the caller can review before applying.

Both paths are intentionally simple. The threshold for “matched” defaults to 0.6 on the Levenshtein similarity ratio — high enough to reject “id” -> “age_group” but low enough to accept “wt” -> “weight” and “drink_yn” -> “alcohol_past12m”. Tune via the threshold parameter.

Example

>>> import pandas as pd
>>> import morie.schema as ms
>>> user_df = pd.DataFrame({
...     "wt": [1, 2, 3],
...     "drinks_yn": [0, 1, 1],
...     "binge30": [0, 0, 1],
...     "age_band": ["18-24", "25-34", "35-44"],
... })
>>> mapping, scores = ms.infer_mapping(user_df,
...     canonical=["weight", "alcohol_past12m", "heavy_drinking_30d",
...                "age_group"])
>>> mapping
{'wt': 'weight', 'drinks_yn': 'alcohol_past12m', 'binge30':
 'heavy_drinking_30d', 'age_band': 'age_group'}
>>> canon_df = ms.apply_mapping(user_df, mapping)
>>> sorted(canon_df.columns)
['age_group', 'alcohol_past12m', 'heavy_drinking_30d', 'weight']

morie.schema.infer_mapping(df, *, canonical, threshold=0.6)[source]¶

Infer a user-column → canonical-name mapping for one DataFrame.

Parameters:

df (pd.DataFrame) – User’s input data.
canonical (iterable of str) – The canonical names morie expects (e.g. CPADS_REQUIRED_VARIABLES).
threshold (float, default 0.6) – Minimum similarity to accept a match.

Returns:

mapping (dict[str, str]) – user_col -> canonical_name pairs above the threshold.
scores (dict[str, float]) – The similarity score for each accepted pair. Use this to decide whether to surface the mapping for user review.

Return type:

tuple[dict[str, str], dict[str, float]]

Notes

Greedy: each user_col is assigned to its best-scoring canonical, and each canonical can only be matched once (first user_col wins). Re-run with a tighter threshold for ambiguous datasets, or override with an explicit mapping.

morie.schema.apply_mapping(df, mapping)[source]¶

Rename columns in-order, return a new DataFrame.

Columns not in the mapping are kept under their original name — morie modules tolerate extra columns; missing canonical names will be detected by validate_cpads_frame (or the equivalent) downstream.

Parameters:

df (DataFrame)
mapping (dict[str, str])

Return type:

DataFrame

morie.schema.parse_cli_mapping(spec)[source]¶

Parse a CLI-style mapping string into a dict.

Format: my_wt:weight,drinks_yn:alcohol_past12m. Whitespace around items is tolerated. Empty input yields an empty dict.

Parameters:: spec (str)
Return type:: dict[str, str]

morie.variable_taxonomy — classify every variable in every dataset.

Goal: drive a method dispatcher that picks the right statistical analysis for each variable based on its formal level of measurement (Stevens 1946: nominal / ordinal / interval / ratio), its cardinality, its functional role (identifier / outcome / covariate / metadata), and any cross-year-safety constraints documented in the data dictionary.

The taxonomy is sourced from three things, in priority order:

Explicit overrides in the per-dataset registry (e.g. OTIS UniqueIndividual_ID is hard-coded as cross_year_safe = FALSE regardless of what the dictionary says, because the dictionary description itself states the random reassignment).
Parsed dictionary entries via morie.dataset_dictionary.load_otis_dictionary() / load_arsau_dictionary. The dictionary supplies the raw Data Type (Integer, Text, Boolean, Date) and Data Values list (closed-set valid values).
Heuristics on the column name + sample values when the dictionary is silent (e.g. trailing _Alert / _Yes_No -> nominal binary; Number* + *Days / *Count -> ratio).

Outputs are VariableTaxonomy dataclasses, one per (dataset, column) tuple. They feed into the method dispatcher (built next) and the audit walker.

References

Stevens, S.S. (1946). “On the theory of scales of measurement.”: Science, 103(2684), 677-680.
Velleman, P.F. and Wilkinson, L. (1993). “Nominal, ordinal, interval,: and ratio typologies are misleading.” The American Statistician, 47(1), 65-72. (We acknowledge the critique; the dispatcher uses measurement level as a default that the caller can override per analysis.)

class morie.variable_taxonomy.LevelOfMeasurement(*values)[source]¶

Bases: str, Enum

Stevens-1946 levels, plus the practical extensions we need.

NOMINAL = 'nominal'¶: Categorical with no order (Race, Weapon, IncidentType).

ORDINAL = 'ordinal'¶: Categorical with order (Age_Category, severity buckets).

INTERVAL = 'interval'¶: Equal-interval scale, no true zero (date-as-day-number, temperature in C).

RATIO = 'ratio'¶: Equal-interval scale WITH true zero (counts, durations, proportions, monetary).

DATE = 'date'¶: Calendar date. Treated specially because date arithmetic is valid but date means depend on calendar conventions.

DATETIME = 'datetime'¶

IDENTIFIER = 'identifier'¶: ID columns (UniqueIndividual_ID, BatchFileName). No statistical analysis applies directly; used for joins / grouping only.

BOOLEAN = 'boolean'¶: Strict yes/no (or 1/0). A special case of nominal binary but flagged because many tests are optimised for the dichotomous case (Wilson CI, McNemar).

FREE_TEXT = 'free_text'¶: Free-text narrative (rare in administrative releases). No direct statistical analysis without NLP preprocessing.

UNKNOWN = 'unknown'¶

class morie.variable_taxonomy.Cardinality(*values)[source]¶

Bases: str, Enum

Coarse cardinality classes, used to short-circuit slow methods.

BINARY = 'binary'¶

DISCRETE_LOW = 'discrete_low'¶

DISCRETE_MEDIUM = 'discrete_medium'¶

DISCRETE_HIGH = 'discrete_high'¶

CONTINUOUS = 'continuous'¶: Effectively continuous (>>1000 distinct values relative to n, or floating-point).

UNKNOWN = 'unknown'¶

class morie.variable_taxonomy.Role(*values)[source]¶

Bases: str, Enum

Functional role. Drives whether the variable is treated as an identifier, an outcome to predict, a covariate to control for, a weight, or pure metadata.

IDENTIFIER = 'identifier'¶

OUTCOME = 'outcome'¶

COVARIATE = 'covariate'¶

WEIGHT = 'weight'¶

METADATA = 'metadata'¶

UNKNOWN = 'unknown'¶

class morie.variable_taxonomy.VariableTaxonomy(dataset_name, column_name, level, cardinality, role, cross_year_safe=True, dictionary_described=True, valid_values=None, nullable=True, raw_dtype=None, notes=None, source='unknown')[source]¶

Bases: object

The classification of a single (dataset, column) variable.

Parameters:

dataset_name (str)
column_name (str)
level (LevelOfMeasurement)
cardinality (Cardinality)
role (Role)
cross_year_safe (bool)
dictionary_described (bool)
valid_values (tuple[str, ...] | None)
nullable (bool)
raw_dtype (str | None)
notes (str | None)
source (str)

dataset_name: str¶

column_name: str¶

level: LevelOfMeasurement¶

cardinality: Cardinality¶

role: Role¶

cross_year_safe: bool = True¶: If False, joins / aggregations across fiscal years are statistically invalid for this column. OTIS UniqueIndividual_ID is the canonical example: random reassignment each fiscal year means cross-year linking is meaningless.

dictionary_described: bool = True¶: True if the column appears in the official data dictionary.

valid_values: tuple[str, ...] | None = None¶: Closed-set valid values from the dictionary, if any.

nullable: bool = True¶

raw_dtype: str | None = None¶

notes: str | None = None¶

source: str = 'unknown'¶

recommended_summary()[source]¶

Plain-language hint at which summary statistic suits this variable.

Used by the audit report and by morie_analyze_variable.

Return type:: str

recommended_pair_test(other)[source]¶

Hint at the right bivariate test between this variable and other.

Levels-of-measurement → test mapping (Velleman & Wilkinson 1993’s critique noted: caller may override).

Parameters:: other (VariableTaxonomy)
Return type:: str

morie.variable_taxonomy.classify_variable(spec, *, dataset_name)[source]¶

Classify a single variable given its dictionary entry.

Parameters:

spec (ColumnSpec) – The parsed column entry from morie.dataset_dictionary.
dataset_name (str) – The owning dataset id (e.g. "b01" for OTIS, "uof_main_records" for ARSAU).

Return type:

VariableTaxonomy

morie.variable_taxonomy.classify_schema(schema)[source]¶

Classify every column in a DatasetSchema.

Parameters:: schema (DatasetSchema)
Return type:: list[VariableTaxonomy]

morie.audit_variables — comprehensive per-variable audit walker.

Walks every (dataset, column) pair across the OTIS and ARSAU catalogues, classifies each variable via morie.variable_taxonomy, and reports:

how many variables in total
distribution across levels of measurement
distribution across functional roles
which variables are currently exercised by an analyzer in mrm_otis.R / mrm_arsau.py
which variables are flagged cross_year_safe = False
which variables are present in the CSVs but NOT in the dictionary (and vice versa)
the recommended summary statistic per variable
the recommended bivariate test per interesting pair

Output is a morie.fn._richresult.RichResult plus an auxiliary Markdown report saved to papers/data-dictionary-audit.md so the v0.9.5.5 release ships with a defensible record of “every variable accounted for”.

Public callables¶

audit_otis_variables() — walks the 29 OTIS datasets
audit_arsau_variables() — walks the 10 ARSAU datasets
audit_all_variables() — both; the headline runbook
write_audit_markdown() — persists a Markdown report (default location: docs/data-dictionary-audit.md)

morie.audit_variables.audit_otis_variables(*, data_dir=None)[source]¶

Walk every column in every OTIS dataset; classify; report.

Parameters:: data_dir (str | Path | None)
Return type:: RichResult

morie.audit_variables.audit_arsau_variables(*, data_dir=None, year_ranges=('2020-2022', '2023', '2024'))[source]¶

Walk every column in every ARSAU dictionary; classify; report.

Concatenates the dictionaries for all year ranges (the same dataset_name can appear in multiple years; the resulting taxonomy list contains one row per (year, dataset, column)).

Parameters:

data_dir (str | Path | None)
year_ranges (Iterable[str])

Return type:

morie.audit_variables.audit_all_variables(*, otis_data_dir=None, arsau_data_dir=None)[source]¶

Run both audits; return a dict with each RichResult.

Parameters:

otis_data_dir (str | Path | None)
arsau_data_dir (str | Path | None)

Return type:

morie.audit_variables.write_audit_markdown(out_path, *, otis_data_dir=None, arsau_data_dir=None)[source]¶

Write a comprehensive Markdown report to out_path.

Parameters:

out_path (str | Path)
otis_data_dir (str | Path | None)
arsau_data_dir (str | Path | None)

Return type:

Categorical-integrity guards: the Python arm of rmorie’s module 25.

Category-mapping errors are among the most damaging silent failures in applied statistics: numeric codes imported without their value labels, positional recodes that reassign groups wholesale, and alphabetical releveling that silently changes the reference category. Any of them can relabel entire demographic groups and multiply a reported odds ratio severalfold without a warning. The functions here make every recode explicit and name-based, refuse anything unmapped, audit imported columns for the known hazards, prove a recode with a before/after cross-tabulation, verify recoded counts against the counts a release published, and check reported odds ratios against a labelled table under every relabelling, so a swapped label is named as such rather than blamed on the software that carried it.

Every function here agrees with the R arms (morie and rmorie) on identical inputs to twelve digits, including the wording of its errors.

morie.categorical_guard.transfer_verify(imported, source_counts, value_labels=None, code_book=None, tolerance=0, strict=True)[source]¶

Verify a categorical variable that crossed from SPSS/Stata/SAS. value_labels is what arrived with the file (pyreadstat), code_book what the source program’s variable view says; source_counts its frequency table. Any disagreement is an error naming the permutation.

Parameters:

source_counts (dict)
value_labels (dict | None)
code_book (dict | None)
tolerance (float)
strict (bool)

Return type:

morie.categorical_guard.safe_relabel(x, mapping, keep=())[source]¶

Relabel by NAME only. A list of labels assigned by position is the mechanism behind the documented four-way rotation (OHRC correction, 26 January 2023), so anything that is not a dict is refused.

Return type:: dict

morie.categorical_guard.decode_labelled(codes, value_labels=None)[source]¶

Decode imported codes by their value labels, BY CODE. value_labels maps code -> label (pyreadstat’s value_labels[var] shape). Levels come out in code order, never alphabetical.

Parameters:: value_labels (dict | None)
Return type:: dict

morie.categorical_guard.relabel_forensics(value_labels, observed, counts=None)[source]¶

Which deterministic step, applied to the code book, reproduces the observed permutation? A match on a sort-based mechanism exonerates the transfer: no import routine sorts value labels onto codes.

Parameters:

value_labels (dict)
observed (dict)
counts (dict | None)

Return type:

morie.categorical_guard.safe_recode(x, mapping, keep=())[source]¶

Recode by explicit name-to-name mapping; unmapped values are an error.

Returns {"values": [...], "audit": {"mapping", "kept", "checksum"}}. The checksum is the SHA-256 of old=new;old=new;... in mapping order, the same digest the R arms attach as the morie_recode_audit attribute.

Parameters:: mapping (dict)
Return type:: dict

morie.categorical_guard.safe_factor(x, levels, reference=None)[source]¶

A factor with explicit, verified levels: the first level is the reference.

Returns {"values": [...], "levels": [...], "codes": [...], "reference": str} where codes are 1-based level indices (R’s integer codes) or None.

Return type:: dict

morie.categorical_guard.audit_categories(data, cols=None, factor_levels=None)[source]¶

Audit categorical columns for coding hazards.

data maps column name to a list of values. factor_levels may give the declared level order of a column (an R factor); such columns are reported with storage factor and their unused levels flagged. Returns one dict per column with column, storage, n_levels, levels (first eight, joined by |), reference and hazards (joined by ' ;; '), plus clean on the last entry’s parent list via audit_categories.clean semantics mirrored in the R print method.

Parameters:

data (dict)
factor_levels (dict | None)

Return type:

list

morie.categorical_guard.crosstab_verify(original, recoded, declared)[source]¶

Prove a recode with a before/after cross-tabulation; errors otherwise.

Parameters:: declared (dict)
Return type:: dict

morie.categorical_guard.marginals_verify(x, published, tolerance=0, strict=True)[source]¶

Recoded counts per label must equal the published counts; names a permutation otherwise.

Parameters:

published (dict)
tolerance (float)
strict (bool)

Return type:

morie.categorical_guard.odds_ratio_check(counts, reference, reported, tolerance=0.05)[source]¶

Check reported odds ratios against a labelled k-by-2 table under every relabelling.

counts maps group label to [outcome_absent, outcome_present].

Parameters:

counts (dict)
reference (str)
reported (dict)
tolerance (float)

Return type:

morie.categorical_guard.guard_binary_treatment(x, col)[source]¶

Refuse a categorical column where a numeric 0/1 treatment is required.

Parameters:: col (str)
Return type:: bool

Data and model validation framework for epidemiological analysis.

Provides schema validation, data quality scoring, cross-validation strategies, calibration assessment, discrimination metrics, decision curve analysis, overfitting detection, and temporal/external validation frameworks.

class morie.validation.ColumnRule(name, dtype=None, required=True, nullable=True, null_threshold=1.0, min_val=None, max_val=None, allowed_values=None, unique=False, regex=None, custom=None)[source]¶

Bases: object

Validation rule for a single column.

Parameters:

name (str) – Column name.
dtype (str or None) – Expected pandas dtype family (e.g. "numeric", "object", "datetime").
required (bool) – Whether the column must be present.
nullable (bool) – Whether null values are allowed.
null_threshold (float) – Maximum fraction of nulls allowed (0.0 – 1.0).
min_val (float or None) – Minimum allowed value (numeric columns).
max_val (float or None) – Maximum allowed value.
allowed_values (set or None) – If provided, all non-null values must be in this set.
unique (bool) – Whether all values must be unique.
regex (str or None) – A regex pattern that all string values must match.
custom (Callable or None) – A lambda/function (pd.Series) -> bool. Returns True if the column passes.

name: str¶

dtype: str | None = None¶

required: bool = True¶

nullable: bool = True¶

null_threshold: float = 1.0¶

min_val: float | None = None¶

max_val: float | None = None¶

allowed_values: set | None = None¶

unique: bool = False¶

regex: str | None = None¶

custom: Callable[[Series], bool] | None = None¶

class morie.validation.SchemaValidationResult(passed, errors=<factory>, warnings=<factory>)[source]¶

Bases: object

Result container for schema validation.

Parameters:

passed (bool)
errors (list[str])
warnings (list[str])

passed: bool¶

errors: list[str]¶

warnings: list[str]¶

morie.validation.validate_schema(data, rules, *, raise_on_error=False)[source]¶

Validate a DataFrame against a schema defined by column rules.

Parameters:

data (DataFrame) – Data to validate.
rules (list of ColumnRule) – Validation rules for each column.
raise_on_error (bool) – If True, raise ValueError on the first error.

Return type:

SchemaValidationResult

morie.validation.check_referential_integrity(child, parent, child_key, parent_key)[source]¶

Check that all values in child_key appear in parent_key.

Parameters:

child (DataFrame) – Table with the foreign key.
parent (DataFrame) – Table with the primary key.
child_key (str) – Column in child.
parent_key (str) – Column in parent.

Return type:

SchemaValidationResult

class morie.validation.DataQualityReport(completeness, consistency, timeliness, uniqueness, overall, details=<factory>)[source]¶

Bases: object

Data quality assessment scores.

Parameters:

completeness (float)
consistency (float)
timeliness (float)
uniqueness (float)
overall (float)
details (dict[str, Any])

completeness: float¶

consistency: float¶

timeliness: float¶

uniqueness: float¶

overall: float¶

details: dict[str, Any]¶

morie.validation.score_data_quality(data, *, date_cols=None, freshness_days=365, key_cols=None, consistency_rules=None)[source]¶

Compute a multi-dimensional data quality score.

Parameters:

data (DataFrame) – Input data.
date_cols (list of str or None) – Datetime columns used for timeliness scoring.
freshness_days (int) – Maximum age in days for full timeliness score.
key_cols (list of str or None) – Columns that should be unique together (for uniqueness score).
consistency_rules (list of callable or None) – Each callable takes the DataFrame and returns True if the consistency check passes.

Returns:

Scores between 0.0 and 1.0 for each dimension and an overall composite score.

Return type:

DataQualityReport

class morie.validation.CVResult(scores, mean, std, ci_lower, ci_upper, fold_details=<factory>)[source]¶

Bases: object

Cross-validation result.

Parameters:

scores (marr)
mean (float)
std (float)
ci_lower (float)
ci_upper (float)
fold_details (list[dict[str, Any]])

scores: marr¶

mean: float¶

std: float¶

ci_lower: float¶

ci_upper: float¶

fold_details: list[dict[str, Any]]¶

morie.validation.cross_validate(estimator, X, y, *, method='stratified_kfold', n_folds=5, n_repeats=10, scoring='roc_auc', groups=None, confidence=0.95, random_state=42)[source]¶

Cross-validate a model using various strategies.

Parameters:

estimator (sklearn estimator) – An unfitted estimator (will be cloned per fold).
X (array-like) – Feature matrix.
y (array-like) – Target vector.
method (str) – Cross-validation strategy: "kfold", "stratified_kfold", "grouped_kfold", "loo", "monte_carlo", "time_series", "nested" (outer only – see nested_cross_validate()).
n_folds (int) – Number of folds (ignored for LOO).
n_repeats (int) – Number of repeats (for "monte_carlo").
scoring (str) – Scoring metric name (scikit-learn convention).
groups (array-like or None) – Group labels for grouped k-fold.
confidence (float) – Confidence level for the CI on mean score.
random_state (int) – Random seed.

Return type:

CVResult

morie.validation.nested_cross_validate(estimator, X, y, *, param_grid, outer_folds=5, inner_folds=3, scoring='roc_auc', random_state=42)[source]¶

Nested cross-validation for unbiased performance estimation.

Parameters:

estimator (sklearn estimator)
X (array-like)
y (array-like)
param_grid (dict) – Hyperparameter grid for inner-loop search.
outer_folds (int)
inner_folds (int)
scoring (str)
random_state (int)

Return type:

CVResult

morie.validation.bootstrap_validate(estimator, X, y, *, n_bootstraps=200, scoring='roc_auc', method='632plus', random_state=42)[source]¶

Bootstrap validation with .632 or .632+ correction.

Parameters:

estimator (sklearn estimator)
X (array-like)
y (array-like)
n_bootstraps (int)
scoring (str) – "roc_auc" or "accuracy".
method (str) – "632" or "632plus".
random_state (int)

Return type:

CVResult

References

Efron, B., & Tibshirani, R. (1997). Improvements on cross-validation: the .632+ bootstrap method. Journal of the American Statistical Association, 92(438), 548-560. https://doi.org/10.1080/01621459.1997.10474007

class morie.validation.CalibrationResult(hosmer_lemeshow_stat, hosmer_lemeshow_p, calibration_slope, calibration_intercept, brier_score, scaled_brier, calibration_in_the_large)[source]¶

Bases: object

Calibration assessment metrics.

Parameters:

hosmer_lemeshow_stat (float)
hosmer_lemeshow_p (float)
calibration_slope (float)
calibration_intercept (float)
brier_score (float)
scaled_brier (float)
calibration_in_the_large (float)

hosmer_lemeshow_stat: float¶

hosmer_lemeshow_p: float¶

calibration_slope: float¶

calibration_intercept: float¶

brier_score: float¶

scaled_brier: float¶

calibration_in_the_large: float¶

morie.validation.assess_calibration(y_true, y_pred, *, n_groups=10)[source]¶

Comprehensive calibration assessment for a binary outcome model.

Parameters:

y_true (array-like) – True binary labels.
y_pred (array-like) – Predicted probabilities.
n_groups (int) – Number of groups for the Hosmer-Lemeshow test.

Return type:

References

Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied Logistic Regression (3rd ed.). Wiley. https://doi.org/10.1002/9781118548387

Steyerberg, E. W., et al. (2010). Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology, 21(1), 128-138. https://doi.org/10.1097/EDE.0b013e3181c30fb2

class morie.validation.DiscriminationResult(auroc, auroc_ci_lower, auroc_ci_upper, c_statistic, somers_d, discrimination_slope, nri=None, idi=None)[source]¶

Bases: object

Discrimination assessment metrics.

Parameters:

auroc (float)
auroc_ci_lower (float)
auroc_ci_upper (float)
c_statistic (float)
somers_d (float)
discrimination_slope (float)
nri (float | None)
idi (float | None)

auroc: float¶

auroc_ci_lower: float¶

auroc_ci_upper: float¶

c_statistic: float¶

somers_d: float¶

discrimination_slope: float¶

nri: float | None = None¶

idi: float | None = None¶

morie.validation.assess_discrimination(y_true, y_pred, *, y_pred_ref=None, n_bootstrap=1000, confidence=0.95, random_state=42, ci_method='delong')[source]¶

Comprehensive discrimination assessment for a binary classifier.

Parameters:

y_true (array-like) – True binary labels.
y_pred (array-like) – Predicted probabilities from the model being evaluated.
y_pred_ref (array-like or None) – Predicted probabilities from a reference model. If provided, NRI and IDI are computed comparing y_pred to y_pred_ref.
n_bootstrap (int) – Number of bootstrap replicates when ci_method="bootstrap".
ci_method (str) – "delong" (default): AUC +/- z SE with the DeLong, DeLong & Clarke-Pearson (1988) variance, clipped to [0, 1], as pROC::ci.auc; "bootstrap": percentile bootstrap.
confidence (float)
random_state (int)

Return type:

DiscriminationResult

References

DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves. Biometrics, 44(3), 837-845. https://doi.org/10.2307/2531595

Pencina, M. J., et al. (2008). Evaluating the added predictive ability of a new marker. Statistics in Medicine, 27(2), 157-172. https://doi.org/10.1002/sim.2929

class morie.validation.DecisionCurveResult(thresholds, net_benefit, net_benefit_all, net_benefit_none)[source]¶

Bases: object

Decision curve analysis result.

Parameters:

thresholds (marr)
net_benefit (marr)
net_benefit_all (marr)
net_benefit_none (marr)

thresholds: marr¶

net_benefit: marr¶

net_benefit_all: marr¶

net_benefit_none: marr¶

morie.validation.decision_curve_analysis(y_true, y_pred, *, thresholds=None)[source]¶

Decision curve analysis for clinical prediction models.

Computes net benefit across a range of probability thresholds.

Parameters:

y_true (array-like) – True binary labels.
y_pred (array-like) – Predicted probabilities.
thresholds (array-like or None) – Probability thresholds to evaluate. Defaults to np.arange(0.01, 0.99, 0.01).

Return type:

DecisionCurveResult

References

Vickers, A. J., & Elkin, E. B. (2006). Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making, 26(6), 565-574. https://doi.org/10.1177/0272989X06295361

class morie.validation.OverfitResult(apparent_performance, optimism, corrected_performance, shrinkage_factor, recommendation)[source]¶

Bases: object

Overfitting assessment result.

Parameters:

apparent_performance (float)
optimism (float)
corrected_performance (float)
shrinkage_factor (float)
recommendation (str)

apparent_performance: float¶

optimism: float¶

corrected_performance: float¶

shrinkage_factor: float¶

recommendation: str¶

morie.validation.detect_overfitting(estimator, X, y, *, scoring='roc_auc', n_bootstrap=200, random_state=42)[source]¶

Detect overfitting via bootstrap optimism correction.

Estimates the optimism (the gap between apparent and honest performance) and computes a shrinkage factor.

Parameters:

estimator (sklearn estimator)
X (array-like)
y (array-like)
scoring (str)
n_bootstrap (int)
random_state (int)

Return type:

OverfitResult

References

Steyerberg, E. W. (2019). Clinical Prediction Models (2nd ed.). Springer. https://doi.org/10.1007/978-3-030-16399-0

class morie.validation.TemporalValidationResult(train_score, test_score, degradation, split_date, train_n, test_n)[source]¶

Bases: object

Temporal (time-split) validation result.

Parameters:

train_score (float)
test_score (float)
degradation (float)
split_date (str)
train_n (int)
test_n (int)

train_score: float¶

test_score: float¶

degradation: float¶

split_date: str¶

train_n: int¶

test_n: int¶

morie.validation.temporal_validate(estimator, X, y, date_col, *, split_date=None, split_quantile=0.7, scoring='roc_auc')[source]¶

Temporal validation: train on earlier data, test on later data.

Parameters:

estimator (sklearn estimator)
X (DataFrame) – Feature matrix (must include or align with date_col).
y (Series) – Target variable.
date_col (str) – Date column name in X.
split_date (str, Timestamp, or None) – Date to split on. If None, uses split_quantile.
split_quantile (float) – Quantile of dates for the split (default 0.7 = first 70% for training).
scoring (str)

Return type:

TemporalValidationResult

class morie.validation.ExternalValidationResult(discrimination, calibration, n_external, domain_shift)[source]¶

Bases: object

External validation result.

Parameters:

discrimination (DiscriminationResult)
calibration (CalibrationResult)
n_external (int)
domain_shift (dict[str, float])

discrimination: DiscriminationResult¶

calibration: CalibrationResult¶

n_external: int¶

domain_shift: dict[str, float]¶

morie.validation.external_validate(model, X_external, y_external, *, X_development=None)[source]¶

External validation of a fitted model on new data.

Parameters:

model (fitted sklearn estimator) – Must have a predict_proba method.
X_external (array-like) – External data features.
y_external (array-like) – External data outcome.
X_development (array-like or None) – Original development data features (for domain-shift diagnostics).

Return type:

ExternalValidationResult

class morie.validation.ReproducibilityManifest(python_version, package_versions, random_seeds, data_checksum, parameters, timestamp)[source]¶

Bases: object

Manifest capturing all information needed to reproduce an analysis.

Parameters:

python_version (str)
package_versions (dict[str, str])
random_seeds (dict[str, int])
data_checksum (str)
parameters (dict[str, Any])
timestamp (str)

python_version: str¶

package_versions: dict[str, str]¶

random_seeds: dict[str, int]¶

data_checksum: str¶

parameters: dict[str, Any]¶

timestamp: str¶

morie.validation.create_reproducibility_manifest(data, *, parameters=None, seeds=None)[source]¶

Create a reproducibility manifest for an analysis.

Parameters:

data (DataFrame) – The analysis data (used to compute a checksum).
parameters (dict or None) – Key analysis parameters.
seeds (dict or None) – Random seeds used.

Return type:

ReproducibilityManifest

morie bricklayer / morie r-install – offer to install the rest of the morie family.

Python is already present (you ran morie), so this focuses on the other ecosystems:

R: rmorie, rmoriedata and rmoriebricklayer, all from r-universe (CRAN carries older companions).
the rmorie command-line launcher ships inside rmorie; rmorie::install_cli() links it onto PATH.

The whole family is built on a shared C/C++ numeric core (libmorie -> morie._core in Python; rmoriebricklayer’s compiled kernels in R). This command verifies that core actually loaded and warns – loudly – that the project does not work properly without a C/C++ toolchain.

Pure stdlib, cross-platform (works for Windows pip users with no shell).

morie.bricklayer.register_subparser(subparsers)[source]¶

Called from morie.runner.build_parser() to register this command.

Return type:: None

morie.bricklayer.run(args)[source]¶

Run morie bricklayer: report which members of the morie family are installed.

Prints one line each for morie (Python), rmorie with its data and bricklayer packages (R), the rmorie launcher and a C/C++ toolchain, and warns when a compiled core is inactive. With args.check it only reports. Otherwise, when the R side is missing (or args.github asks for this repository’s R arm), it installs it – rmorie pinned to this morie’s version – after a confirmation that args.yes skips; off a terminal without --yes it prints the install command instead.

Parameters:: args – the parsed argparse namespace of the bricklayer verb.
Returns:: The process exit code (0 when everything checked is present).
Return type:: int

CPADS data contract helpers for local-private analysis.

morie.cpads.cpads_contract()[source]¶

Return the local-private CPADS contract used by MORIE workflows.

Return type:: dict[str, object]

morie.cpads.missing_cpads_variables(columns)[source]¶

Return missing canonical CPADS variables.

Parameters:: columns (Iterable[str])
Return type:: list[str]

morie.cpads.validate_cpads_frame(frame, *, strict=True)[source]¶

Validate that a frame contains the canonical CPADS analysis columns.

Parameters:

frame (DataFrame)
strict (bool)

Return type:

morie.cpads.has_raw_cpads_columns(frame)[source]¶

Return True when the frame looks like raw CPADS PUMF data.

Accepts both wtpumf (PUMF release) and wtdf (full dataset) as the weight column.

Parameters:: frame (DataFrame)
Return type:: bool

morie.cpads.canonicalize_cpads_frame(frame)[source]¶

Convert raw CPADS PUMF columns into the canonical MORIE analysis columns.

This is a first-pass canonicalization layer based on the current public CPADS PUMF field names present in the repository CSV.

Parameters:: frame (DataFrame)
Return type:: DataFrame

morie.cpads.infer_file_format(path)[source]¶

Infer a dataset format from a local path.

Parameters:: path (str | Path)
Return type:: str

Further estimators and diagnostics¶

Cluster-robust double machine learning for the ATE.

A general policy-evaluation entry point for double/debiased machine learning when observations are clustered – flights within an airspace corridor, students within a school, patients within a hospital. Standard DML standard errors assume independence and are anti-conservative under within-cluster correlation. This cross-fits the AIPW (doubly-robust) score and computes a cluster-robust variance from per-cluster score sums (Liang & Zeger 1986, one-way; Cameron, Gelbach & Miller 2011, up to two-way).

Nuisances are cross-fitted (Chernozhukov et al. 2018): a logistic propensity and per-arm OLS outcome regressions, so the AIPW point estimate is Neyman-orthogonal. Only the SE is cluster-aware.

R parity: rmorie R/dml_clustered.R (morie_dml_clustered).

morie.dml_clustered.dml_clustered(data, treatment, outcome, covariates, cluster=None, n_folds=5, seed=123, eps=0.02, ps=None)[source]¶

Cluster-robust DML estimate of the ATE.

Parameters mirror morie_dml_clustered. cluster is a column name (one-way), a length-2 list (two-way), or None (i.i.d. SE). Returns a dict with ate, se, ci95, z, pval, n, n_clusters, se_kind.

Examples

>>> import random
>>> from morie.fn import _frame_core as pd
>>> r = random.Random(5)
>>> x = [r.gauss(0, 1) for _ in range(120)]
>>> t = [1 if xi + r.gauss(0, 1) > 0 else 0 for xi in x]
>>> y = [0.7 * ti + xi + r.gauss(0, 1) for ti, xi in zip(t, x)]
>>> df = pd.DataFrame({"y": y, "t": t, "x": x, "g": [i % 6 for i in range(120)]})
>>> out = dml_clustered(df, "t", "y", ["x"], cluster="g", seed=1)
>>> (round(out["ate"], 3), round(out["se"], 3))
(0.809, 0.153)

Parameters:

data (DataFrame)
treatment (str)
outcome (str)
covariates (list[str])
cluster (str | list[str] | None)
n_folds (int)
seed (int)
eps (float)
ps (marr | None)

Return type:

eBAC calculations. Ported from internal routines for computing continuous eBAC and legal threshold binaries.

morie.ebac.calculate_ebac(drinks, weight_lbs, hours, gender_constant)[source]¶

Compute the continuous estimated Blood Alcohol Concentration (eBAC) using the standard Widmark formula.

Parameters:

drinks (float) – The number of standard drinks consumed (1 drink = 14 grams of alcohol).
weight_lbs (float) – The weight of the individual in pounds.
hours (float) – Hours elapsed since drinking began.
gender_constant (float) – Widmark gender multiplier (standard is 0.73 for men, 0.66 for women).

Returns:

The estimated Blood Alcohol Concentration level (non-negative).

Return type:

morie.ebac.is_over_legal_limit(ebac, limit=0.08)[source]¶

Determine whether an eBAC exceeds the specified legal driving limit.

Parameters:

ebac (float) – The calculated eBAC value to evaluate.
limit (float, optional) – The legal blood alcohol concentration limit, defaults to 0.08.

Returns:

1 if the eBAC is greater than or equal to the limit, 0 otherwise.

Return type:

morie.fairness — group-disparity audit toolkit.

A subsystem for auditing risk-assessment, recidivism, and predictive-policing systems for racial (and other protected-attribute) disparities. morie does not deploy such systems — it measures whether an existing one encodes disparate treatment, so that researchers, oversight bodies, and the public can hold those systems accountable.

Phase A ships the disparity-metrics core (morie.fairness.metrics): classical group-fairness measures — disparate impact, demographic parity, equalised odds, average odds difference, and the Gini coefficient — each returning a rich, paragraph-level RichResult.

Later phases add the generalised predictive-policing audit module, the multi-city temporal analysis, the JAX simulation framework, and the explainability (XAI) layer.

Examples

>>> from morie.fairness import fairness_disparate_impact
>>> r = fairness_disparate_impact([1, 0, 1, 1, 0, 0], ["a", "a", "a", "b", "b", "b"])
>>> r["value"], r["privileged"], r["adverse_impact"]
(0.5, 'a', True)

morie.fairness.fairness_disparate_impact(y_pred, group, *, privileged=None, favorable=1)[source]¶

Disparate Impact Ratio — the EEOC four-fifths (80%) rule.

For each group, the disparate-impact ratio is its favourable-outcome rate divided by the privileged group’s rate. A value below 0.8 is the standard legal indicator of adverse impact.

Parameters:

y_pred (array-like) – The decision/assignment for each individual (e.g. flagged vs. not flagged, high-risk vs. not).
group (array-like) – Protected attribute (e.g. race), one value per individual.
privileged (optional) – The reference group. If omitted, the highest-rate group is used and a warning is emitted.
favorable (default 1) – The value of y_pred that counts as the favourable outcome.

Returns:

Headline value is the worst (smallest) ratio across groups.

Return type:

Examples

>>> import morie
>>> pred  = [1, 1, 1, 1, 1, 1, 1, 1, 0, 0]
>>> race  = ["A", "A", "A", "A", "A", "B", "B", "B", "B", "B"]
>>> res = morie.fairness_disparate_impact(pred, race, privileged="A")
>>> round(float(res), 2)  # group B rate 0.6 / group A rate 1.0
0.6

morie.fairness.fairness_demographic_parity(y_pred, group, *, privileged=None, favorable=1)[source]¶

Demographic Parity Gap — difference in favourable-outcome rates.

The gap is rate(group) - rate(privileged). Demographic parity holds when every group receives favourable outcomes at the same rate, i.e. all gaps are zero. Unlike the disparate-impact ratio, this is an additive difference, so it is well defined even when the privileged rate is zero.

Parameters:

y_pred (Any) – As in fairness_disparate_impact().
group (Any) – As in fairness_disparate_impact().
privileged (Any) – As in fairness_disparate_impact().
favorable (Any) – As in fairness_disparate_impact().

Returns:

Headline value is the largest absolute gap across groups.

Return type:

Examples

>>> import morie
>>> pred = [1, 1, 1, 1, 0, 0, 0, 1, 0, 0]
>>> race = ["A", "A", "A", "A", "A", "B", "B", "B", "B", "B"]
>>> res = morie.fairness_demographic_parity(pred, race, privileged="A")
>>> round(float(res), 2)  # group B rate 0.2 minus group A rate 0.8
-0.6

morie.fairness.fairness_equalized_odds(y_true, y_pred, group, *, privileged=None, favorable=1)[source]¶

Equalized Odds — true- and false-positive-rate gaps across groups.

Equalized odds holds when the true-positive rate (TPR) and false-positive rate (FPR) are equal across groups. This metric needs ground-truth labels, so it audits a system’s errors, not just its decision rates — a system can satisfy demographic parity yet still make far more false-positive errors against one group.

Parameters:

y_true (array-like) – Realised ground-truth outcome for each individual.
y_pred (array-like) – The system’s decision for each individual.
group (Any) – As in fairness_disparate_impact().
privileged (Any) – As in fairness_disparate_impact().
favorable (Any) – As in fairness_disparate_impact().

Returns:

Headline value is the largest absolute TPR-or-FPR gap.

Return type:

Examples

>>> import morie
>>> truth = [1, 0, 1, 0, 1, 0, 1, 0]
>>> pred  = [1, 0, 1, 0, 1, 1, 0, 1]
>>> race  = ["A", "A", "A", "A", "B", "B", "B", "B"]
>>> res = morie.fairness_equalized_odds(truth, pred, race, privileged="A")
>>> bool(res.payload["violation"])
True

morie.fairness.fairness_average_odds_difference(y_true, y_pred, group, *, privileged=None, favorable=1)[source]¶

Average Odds Difference — mean of the TPR and FPR gaps.

For each non-reference group, the average odds difference is 0.5 * ((FPR_group - FPR_ref) + (TPR_group - TPR_ref)). Zero means parity of errors; negative means the group is under-served by correct positives and/or subjected to fewer false positives. This is the single-number summary used in IBM AIF360 and in the COMPAS XAI Stories audit (which reduced it from 0.245 to 0.022 via the Prejudice Remover — a mitigation morie adds in a later phase).

Parameters:

y_true (Any) – As in fairness_equalized_odds().
y_pred (Any) – As in fairness_equalized_odds().
group (Any) – As in fairness_equalized_odds().
privileged (Any) – As in fairness_equalized_odds().
favorable (Any) – As in fairness_equalized_odds().

Returns:

Headline value is the group with the largest absolute average odds difference.

Return type:

Examples

>>> import morie
>>> truth = [1, 0, 1, 0, 1, 0, 1, 0]
>>> pred  = [1, 0, 1, 0, 1, 1, 0, 1]
>>> race  = ["A", "A", "A", "A", "B", "B", "B", "B"]
>>> res = morie.fairness_average_odds_difference(
...     truth, pred, race, privileged="A")
>>> abs(float(res)) > 0
True

morie.fairness.fairness_gini(values, *, group=None)[source]¶

Gini coefficient — concentration/inequality of a distribution.

The Gini coefficient ranges from 0 (every value identical, perfect equality) to nearly 1 (one unit holds everything). Applied to risk scores, patrol counts, or stop counts, it measures how unequally a predictive-policing system concentrates its attention. When a group is supplied, a per-group Gini is also reported, exposing whether the concentration is itself uneven across groups.

Parameters:

values (array-like) – Non-negative quantities (risk scores, counts, …).
group (array-like, optional) – Protected attribute; enables the per-group breakdown.

Returns:

Headline value is the overall Gini coefficient.

Return type:

Examples

>>> import morie
>>> round(float(morie.fairness_gini([5, 5, 5, 5])), 4)
0.0
>>> float(morie.fairness_gini([0, 0, 0, 100])) > 0.7
True

morie.fairness.fairness_bias_amplification(y_pred, group, *, privileged=None, favorable=1)[source]¶

Bias Amplification Score — composite of parity gap and inequality.

BAS = Δ_parity × G, where Δ_parity is the demographic parity gap of the worst-affected group and G is the Gini coefficient of the per-group favourable-outcome rates. The score is large only when a directional disparity (one group systematically favoured) coincides with high overall inequality across groups — it penalises systems that are both biased and unequal, and stays near zero if either component is absent.

Reimplemented from the definition in Barman & Barman, “Unmasking Algorithmic Bias in Predictive Policing: A GAN-Based Simulation Framework with Multi-City Temporal Analysis” (arXiv:2603.18987) — the composite Bias Amplification Score, BAS = Δ_parity × G.

Parameters:

y_pred (Any) – As in fairness_disparate_impact().
group (Any) – As in fairness_disparate_impact().
privileged (Any) – As in fairness_disparate_impact().
favorable (Any) – As in fairness_disparate_impact().

Returns:

Headline value is the Bias Amplification Score.

Return type:

Examples

>>> import morie
>>> pred = [1, 1, 1, 1, 0, 0, 0, 0]
>>> race = ["A", "A", "A", "A", "B", "B", "B", "B"]
>>> res = morie.fairness_bias_amplification(pred, race, privileged="A")
>>> round(float(res), 3)  # parity gap -1.0 x Gini 0.5
-0.5

morie.fairness.predpol_calibration_audit(areas, mean_risk, outcome_rate, group)[source]¶

Audit whether an algorithm’s area risk ranking matches realised outcomes.

For every area the audit forms two ranks — by predicted risk and by realised outcome rate (rank 1 = highest). Their difference, rank_gap = outcome_rank − risk_rank, is positive when the algorithm ranks an area more dangerous than its realised outcomes warrant (over-prediction) and negative when it under-predicts. Averaging the gap within each demographic group reveals whether the over-prediction is borne disproportionately by one group.

A Spearman rank correlation between predicted risk and realised outcome rate summarises overall calibration.

Parameters:

areas (array-like) – Area identifiers (one per area).
mean_risk (array-like) – Mean predicted risk for each area.
outcome_rate (array-like) – Realised outcome rate for each area.
group (array-like) – Majority/dominant protected-attribute label for each area.

Returns:

Headline value is the largest-magnitude per-group mean rank gap. Positive ⇒ that group’s areas are systematically over-predicted.

Return type:

Examples

>>> import morie
>>> areas   = ["d1", "d2", "d3", "d4"]
>>> risk    = [400, 300, 200, 100]   # algorithm's ranking
>>> outcome = [100, 200, 300, 400]   # realised — exactly reversed
>>> grp     = ["X", "X", "Y", "Y"]
>>> res = morie.predpol_calibration_audit(areas, risk, outcome, grp)
>>> res.payload["spearman"] < 0      # perfectly miscalibrated
True

morie.fairness.predpol_aggregate_areas(area, risk, outcome, *, group=None, population=None)[source]¶

Aggregate per-record predictive-policing data to one row per area.

Most real inputs are per-record: one risk score per individual, one incident per row. This helper rolls them up to the per-area arrays that predpol_calibration_audit() consumes.

Parameters:

area (array-like) – Area / district / precinct identifier for each record.
risk (array-like) – Predicted risk score for each record.
outcome (array-like) – Realised-outcome indicator/count for each record (e.g. 1 if the record is a shooting incident).
group (array-like, optional) – Protected attribute for each record; the per-area majority value becomes that area’s group label.
population (dict or array-like, optional) – Area population. As a dict it maps area -> population; as an array it is per-record (taken as constant within an area). When given, the outcome rate is reported per 10,000 inhabitants; otherwise it is the mean outcome per record.

Returns:

areas, mean_risk, outcome_rate, group (majority, or None), n_records — aligned, one entry per area.

Return type:

morie.fairness.predpol_score_disparity(score, group, *, reference=None)[source]¶

Descriptive disparity in a risk score across groups.

The first step of the SSL-style audit: before comparing predicted risk to realised outcomes, describe how the risk score itself differs across groups. Reports per-group n / mean / median / sd, a one-way ANOVA for whether group membership relates to the score, and each group’s mean-score gap from a reference group.

A statistically significant score gap is not by itself proof of bias — it may reflect genuine base-rate differences. That is exactly why morie pairs this descriptive step with predpol_calibration_audit(), which brings realised outcomes into the comparison.

Parameters:

score (array-like) – Continuous risk score, one per individual.
group (array-like) – Protected attribute, one per individual.
reference (optional) – Reference group for the mean-score gaps. Defaults to the lowest-scoring group, so gaps read as “extra points”.

Returns:

Headline value is the spread (max − min) of group mean scores.

Return type:

Examples

>>> import morie
>>> score = [9, 10, 11, 19, 20, 21]
>>> race  = ["A", "A", "A", "B", "B", "B"]
>>> res = morie.predpol_score_disparity(score, race)
>>> round(float(res), 1)   # group means 10 and 20
10.0

morie.fairness.predpol_temporal_audit(period, city, y_pred, group, *, privileged=None, favorable=1)[source]¶

Audit how disparity metrics move over time and across cities.

For every (city, period) cell the audit computes the four disparity metrics, then aggregates them per city — reporting, for each city, the mean of each metric, the count of periods with DIR > 1 (over-prediction months), and the DIR temporal range (max − min), which quantifies how unstable the metric is across the audited window.

Parameters:

period (array-like) – Time-period label for each record (e.g. "2019-03"). Sorted lexically for display, so ISO-style labels order correctly.
city (array-like) – City label for each record.
y_pred (array-like) – The decision/assignment for each record.
group (array-like) – Protected attribute for each record.
privileged (optional) – Reference group. If omitted it is inferred globally (from the pooled data) so every cell uses the same reference.
favorable (default 1) – The value of y_pred counted as the favourable outcome.

Returns:

Headline value is the largest per-city DIR temporal range — the worst temporal instability found.

Return type:

Examples

>>> import morie
>>> # one city, two periods: identical disparity each period
>>> period = ["p1"] * 10 + ["p2"] * 10
>>> city   = ["A"] * 20
>>> pred   = ([1, 1, 1, 1, 1, 1, 1, 1, 0, 0]) * 2
>>> grp    = (["X"] * 5 + ["Y"] * 5) * 2
>>> res = morie.predpol_temporal_audit(period, city, pred, grp,
...                                    privileged="X")
>>> round(res.payload["per_city"]["A"]["dir_range"], 6)  # stable
0.0

morie.fairness.noisy_or_detection(crime_xy, officer_xy, *, radius, p_detect=0.85, seed=None)[source]¶

Noisy-OR detection probability for each crime given patrol officers.

A crime is detected if any nearby officer detects it. With a per-officer detection probability p and k officers within radius:

P(detected | crime) = 1 − ∏ (1 − p) = 1 − (1 − p) ** k

Parameters:

crime_xy (array-like, shape (n, 2)) – Crime-event coordinates.
officer_xy (array-like, shape (m, 2)) – Patrol-officer coordinates.
radius (float) – Detection radius (same units as the coordinates).
p_detect (float, default 0.85) – Per-officer detection probability, in (0, 1].
seed (int, optional) – If given, a 0/1 detection outcome is also sampled.

Returns:

Headline value is the mean per-crime detection probability; payload carries probabilities, officers_in_range and, when seed is set, the sampled detected vector.

Return type:

Examples

>>> import morie
>>> crimes = [[0.0, 0.0], [10.0, 10.0]]
>>> officers = [[0.1, 0.1]]            # near the first crime only
>>> res = morie.noisy_or_detection(crimes, officers, radius=1.0)
>>> round(float(res.payload["probabilities"][0]), 2)
0.85
>>> float(res.payload["probabilities"][1])    # no officer in range
0.0

morie.fairness.simulate_biased_crime_data(*, n=2000, groups=('A', 'B'), group_props=None, n_areas=20, base_rate=0.3, bias=0.5, seed=0)[source]¶

Generate a synthetic predictive-policing dataset with known bias.

Every record carries an area (areas are group-segregated, as in Chicago), a group, a group-independent true_outcome, the system’s detected flag, and a continuous risk_score (0–500, SSL-style). The bias parameter is the ground truth the audits should recover:

bias = 0 — perfectly fair: every group is flagged at base_rate.
bias > 0 — non-reference groups are under-flagged: their detection rate is base_rate × (1 − bias), so the Disparate Impact Ratio is ≈ 1 − bias. Their mean risk_score is also shifted up by bias × 100 points.
bias < 0 — non-reference groups are over-flagged.

Because true_outcome is group-independent, any disparity the audits report is the injected bias and nothing else — which is what makes this a clean verification fixture.

Parameters:

n (int) – Number of records.
groups (sequence) – Group labels; groups[0] is the reference group.
group_props (sequence, optional) – Sampling proportions for the groups (default uniform).
n_areas (int) – Number of areas; must be ≥ the number of groups.
base_rate (float) – Reference-group favourable-outcome rate, in [0, 1].
bias (float) – Injected disparity, in [-1, 1].
seed (int) – Reproducibility seed.

Returns:

Columns: area, group, true_outcome, detected, risk_score — directly consumable by the morie.fairness audits.

Return type:

morie.fairness.xai_permutation_importance(predict_fn, X, *, feature_names=None, n_repeats=10, protected=None, seed=0)[source]¶

Permutation feature importance — model reliance on each feature.

Importance of a feature is the mean absolute change in the model’s predictions when that feature’s column is randomly shuffled (its information destroyed while the marginal distribution is kept). A feature the model ignores scores ≈ 0; a feature it leans on scores high.

Parameters:

predict_fn (callable) – Maps an (n, d) array to n predictions.
X (array-like, shape (n, d)) – The data to explain over.
feature_names (sequence of str, optional)
n_repeats (int) – Number of shuffles averaged per feature.
protected (sequence of str, optional) – Protected-attribute feature names. If any rank in the top third of features, a bias warning is raised — the COMPAS-audit fingerprint of a discriminatory model.
seed (int)

Returns:

Headline value is the largest importance; payload carries the per-feature importances and the protected-feature ranks.

Return type:

Examples

>>> import numpy as np, morie
>>> X = np.random.default_rng(0).normal(size=(200, 3))
>>> res = morie.xai_permutation_importance(
...     lambda A: A[:, 0], X, feature_names=["a", "b", "c"])
>>> res.payload["ranking"][0]          # feature 'a' dominates
'a'

morie.fairness.xai_partial_dependence(predict_fn, X, feature, *, feature_names=None, grid_size=20)[source]¶

Partial dependence of the model on one feature (Friedman).

Sweeps feature across a grid; at each grid value every row is set to that value and the predictions are averaged. The resulting curve is the model’s average response to the feature.

Returns:

payload carries the grid and pd (partial-dependence) arrays; headline value is the curve’s range.

Return type:

Parameters:

predict_fn (Callable)
X (Any)
feature (Any)
feature_names (Any)
grid_size (int)

morie.fairness.xai_ale(predict_fn, X, feature, *, feature_names=None, n_bins=10)[source]¶

First-order Accumulated Local Effects for one feature (Apley & Zhu).

ALE bins the feature, measures the local prediction change across each bin’s edges using only rows that fall in the bin, then accumulates and centres those local effects. Unlike partial dependence it stays unbiased when features are correlated.

Returns:

payload carries bin_centers and ale (centred effect).

Return type:

Parameters:

predict_fn (Callable)
X (Any)
feature (Any)
feature_names (Any)
n_bins (int)

morie.fairness.xai_ceteris_paribus(predict_fn, x, feature, *, X_ref, feature_names=None, grid_size=20)[source]¶

Ceteris-paribus profile — one instance, one feature varied.

Holds every feature of the instance x fixed except feature, which is swept across the range seen in X_ref. This is the instance-level analogue of partial dependence and the basis of the COMPAS audit’s “what if this defendant were older / a different race” probes.

Returns:

payload carries the grid and the profile predictions.

Return type:

Parameters:

predict_fn (Callable)
x (Any)
feature (Any)
X_ref (Any)
feature_names (Any)
grid_size (int)

morie.fairness.xai_shap_values(predict_fn, x, background, *, feature_names=None, n_samples=200, seed=0)[source]¶

Shapley feature attributions for one instance (sampling estimator).

Estimates SHAP values by the permutation-sampling method (Štrumbelj & Kononenko; the model-agnostic explainer of Lundberg & Lee): over many random feature orderings and random background rows, a feature’s value is “switched on” and its marginal effect on the prediction recorded. The values are additive — they sum to the instance’s prediction minus the mean background prediction.

Returns:

Headline value is the largest-magnitude SHAP value; payload carries the per-feature shap_values.

Return type:

Parameters:

predict_fn (Callable)
x (Any)
background (Any)
feature_names (Any)
n_samples (int)
seed (int)

class morie.fairness.CityProfile(name, area_col, risk_col=None, outcome_col=None, population_col=None, group_col=None, notes='')[source]¶

Bases: object

Column mapping for one city’s predictive-policing data.

Each *_col names the column, in that city’s own export, that carries the corresponding canonical field. risk_col and outcome_col may be None when a city only supplies one side (e.g. risk scores but no realised-outcome feed), in which case the missing side must be supplied separately to the audit.

Parameters:

name (str) – Identifier used with get_city() (e.g. "chicago").
area_col (str) – Column holding the area / district / precinct identifier.
risk_col (str or None) – Columns for the predicted risk score, realised-outcome count, area population, and protected attribute.
outcome_col (str or None) – Columns for the predicted risk score, realised-outcome count, area population, and protected attribute.
population_col (str or None) – Columns for the predicted risk score, realised-outcome count, area population, and protected attribute.
group_col (str or None) – Columns for the predicted risk score, realised-outcome count, area population, and protected attribute.
notes (str) – Free-text provenance / caveats, surfaced to the user.

name: str¶

area_col: str¶

risk_col: str | None = None¶

outcome_col: str | None = None¶

population_col: str | None = None¶

group_col: str | None = None¶

notes: str = ''¶

column_map()[source]¶

Return {source_column: canonical_field} for the columns this profile actually defines.

Return type:: dict[str, str]

morie.fairness.register_city(profile, *, overwrite=False)[source]¶

Add (or replace) a CityProfile in the registry.

Parameters:

profile (CityProfile)
overwrite (bool) – If False (default) registering an existing name raises ValueError; pass True to replace it.

Return type:

None

morie.fairness.get_city(name)[source]¶

Look up a registered CityProfile by name (case-insensitive).

Parameters:: name (str)
Return type:: CityProfile

morie.fairness.list_cities()[source]¶

Return the names of all registered city profiles, sorted.

Return type:: list[str]

morie.fairness.apply_profile(df, profile)[source]¶

Rename a city DataFrame onto the canonical audit schema.

Parameters:

df (pandas.DataFrame) – A city’s predictive-policing data in its native column names.
profile (CityProfile or str) – The profile (or the name of a registered profile) describing df’s columns.

Returns:

A copy of df with the profile’s columns renamed to the canonical names and only those canonical columns retained.

Return type:

Raises:

KeyError – If a column the profile names is absent from df.

Missing data analysis, diagnostics, and imputation for epidemiological datasets.

This module provides the complete analytical pipeline for handling missing data in observational health studies:

Pattern analysis – identify monotone, non-monotone, and systematic missingness; classify mechanisms (MCAR, MAR, MNAR).
Diagnostics – Little’s MCAR test, missingness pattern matrices, proportion summaries.
Single imputation – mean, median, mode, LOCF, NOCB, regression, hot deck, cold deck.
Multiple imputation – MICE (multivariate imputation by chained equations) with PMM, Bayesian linear regression, logistic regression for binary outcomes, polytomous regression for categorical outcomes, and random forest imputation.
Pooling – Rubin’s rules for combining multiply imputed estimates.
Convergence – trace-plot data, R-hat diagnostics.
Sensitivity analysis – tipping-point analysis, delta adjustment.

References

Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. Wiley. van Buuren, S. (2018). Flexible Imputation of Missing Data (2nd ed.).

CRC Press.

Little, R. J. A. (1988). A test of missing completely at random for: multivariate data with missing values. JASA, 83(404), 1198–1202.
White, I. R., Royston, P., & Wood, A. M. (2011). Multiple imputation using: chained equations: Issues and guidance for practice. Statistics in Medicine, 30(4), 377–399.

class morie.missing.MissingProfile(n_obs, n_vars, total_cells, n_missing, pct_missing, complete_cases, per_variable, pattern_matrix, is_monotone)[source]¶

Bases: object

Summary of missingness in a dataset.

Parameters:

n_obs (int) – Total number of observations.
n_vars (int) – Total number of variables.
total_cells (int) – n_obs * n_vars.
n_missing (int) – Total missing cells.
pct_missing (float) – Percentage of missing cells.
complete_cases (int) – Number of rows with zero missing values.
per_variable (DataFrame) – Per-variable missingness summary.
pattern_matrix (ndarray) – Binary matrix (1 = observed, 0 = missing).
is_monotone (bool) – Whether the missingness pattern is monotone.

n_obs: int¶

n_vars: int¶

total_cells: int¶

n_missing: int¶

pct_missing: float¶

complete_cases: int¶

per_variable: DataFrame¶

pattern_matrix: marr¶

is_monotone: bool¶

class morie.missing.MCARTestResult(test_statistic, p_value, df, n_patterns, method="Little's MCAR test")[source]¶

Bases: object

Result of Little’s MCAR test.

Parameters:

test_statistic (float)
p_value (float)
df (int)
n_patterns (int)
method (str)

test_statistic: float¶

p_value: float¶

df: int¶

n_patterns: int¶

method: str = "Little's MCAR test"¶

class morie.missing.ImputedDataset(data, imputation_id, method, n_imputed, variables_imputed)[source]¶

Bases: object

A single imputed dataset with metadata.

Parameters:

data (DataFrame)
imputation_id (int)
method (str)
n_imputed (int)
variables_imputed (list[str])

data: DataFrame¶

imputation_id: int¶

method: str¶

n_imputed: int¶

variables_imputed: list[str]¶

class morie.missing.MICEResult(datasets, n_imputations, n_iterations, convergence, variables_imputed)[source]¶

Bases: object

Result of multiple imputation by chained equations.

Parameters:

datasets (list[ImputedDataset]) – List of m imputed datasets.
n_imputations (int) – Number of imputations performed.
n_iterations (int) – Number of MICE iterations per imputation.
convergence (dict) – Convergence diagnostics (mean and variance traces).
variables_imputed (list[str]) – Variables that were imputed.

datasets: list[ImputedDataset]¶

n_imputations: int¶

n_iterations: int¶

convergence: dict¶

variables_imputed: list[str]¶

class morie.missing.PooledEstimate(estimate, within_variance, between_variance, total_variance, standard_error, df, ci_lower, ci_upper, fmi, relative_efficiency, lambda_hat, method="Rubin's rules")[source]¶

Bases: object

Pooled estimate from Rubin’s rules.

Parameters:

estimate (float)
within_variance (float)
between_variance (float)
total_variance (float)
standard_error (float)
df (float)
ci_lower (float)
ci_upper (float)
fmi (float)
relative_efficiency (float)
lambda_hat (float)
method (str)

estimate: float¶

within_variance: float¶

between_variance: float¶

total_variance: float¶

standard_error: float¶

df: float¶

ci_lower: float¶

ci_upper: float¶

fmi: float¶

relative_efficiency: float¶

lambda_hat: float¶

method: str = "Rubin's rules"¶

morie.missing.missing_profile(data)[source]¶

Comprehensive missingness profile for a DataFrame.

Parameters:: data (DataFrame) – Input dataset.
Return type:: MissingProfile

morie.missing.missing_pattern_table(data)[source]¶

Tabulate unique missingness patterns and their frequencies.

Parameters:: data (DataFrame)
Returns:: Each row is a unique pattern. Columns are variable names (1 = observed, 0 = missing) plus count and pct.
Return type:: DataFrame

morie.missing.missing_proportion_by_variable(data)[source]¶

Return per-variable missingness proportions.

Parameters:: data (DataFrame)
Returns:: Columns: variable, n_missing, n_observed, pct_missing.
Return type:: DataFrame

morie.missing.classify_missing_mechanism(data, target, predictors=None, alpha=0.05)[source]¶

Heuristic classification of the missing data mechanism for a target variable.

Strategy: - Run Little’s MCAR test (if possible) or a logistic regression of the

missingness indicator on other observed variables.

If the missingness is unrelated to observed variables (p > alpha), suggest MCAR.
If related to observed variables, suggest MAR.
MNAR cannot be tested from the data alone; a warning is issued.

Parameters:

data (DataFrame)
target (str) – Variable whose missingness is being classified.
predictors (list[str] or None) – Variables to use as predictors of missingness. If None, all other complete columns are used.
alpha (float, default 0.05)

Returns:

Keys: target, classification, p_value, details.

Return type:

morie.missing.littles_mcar_test(data)[source]¶

Little’s MCAR test for multivariate data.

Tests the null hypothesis that the data are missing completely at random by comparing observed group means (by missingness pattern) with the EM estimate of the overall mean.

Parameters:: data (DataFrame) – Numeric columns only.
Return type:: MCARTestResult

References

Little, R. J. A. (1988). A test of missing completely at random for multivariate data with missing values. JASA, 83(404), 1198–1202.

morie.missing.complete_case_analysis(data, outcome=None, covariates=None)[source]¶

Complete case (listwise deletion) analysis with diagnostics.

Parameters:

data (DataFrame)
outcome (str or None) – If provided, only these + covariate columns are checked.
covariates (list[str] or None)

Returns:

Keys: complete_data, n_original, n_complete, pct_dropped, variables_checked.

Return type:

morie.missing.impute_mean(data, columns=None)[source]¶

Replace missing values with the column mean.

Parameters:

data (DataFrame)
columns (list[str] or None) – Columns to impute. If None, all numeric columns.

Return type:

DataFrame

morie.missing.impute_median(data, columns=None)[source]¶

Replace missing values with the column median.

Parameters:

data (DataFrame)
columns (list[str] or None)

Return type:

DataFrame

morie.missing.impute_mode(data, columns=None)[source]¶

Replace missing values with the column mode.

Parameters:

data (DataFrame)
columns (list[str] or None)

Return type:

DataFrame

morie.missing.impute_locf(data, columns=None)[source]¶

Last observation carried forward (LOCF) imputation.

Parameters:

data (DataFrame) – Assumed to be sorted in temporal order.
columns (list[str] or None)

Return type:

DataFrame

morie.missing.impute_nocb(data, columns=None)[source]¶

Next observation carried backward (NOCB) imputation.

Parameters:

data (DataFrame) – Assumed to be sorted in temporal order.
columns (list[str] or None)

Return type:

DataFrame

morie.missing.impute_regression(data, target, predictors=None, add_noise=False)[source]¶

Regression imputation: predict missing values from a linear model.

Parameters:

data (DataFrame)
target (str) – Column to impute.
predictors (list[str] or None) – Columns to use as predictors. If None, all other complete numeric columns.
add_noise (bool, default False) – If True, add Gaussian noise proportional to the residual standard deviation (stochastic regression imputation).

Return type:

DataFrame

morie.missing.impute_hot_deck(data, target, match_vars=None, random_state=None)[source]¶

Hot-deck imputation: replace missing values with observed values from a randomly selected similar donor.

Parameters:

data (DataFrame)
target (str) – Column to impute.
match_vars (list[str] or None) – Variables to match on. If None, random selection from all observed.
random_state (int or None)

Return type:

DataFrame

morie.missing.impute_cold_deck(data, target, external_values)[source]¶

Cold-deck imputation: replace missing values with pre-specified external values.

Parameters:

data (DataFrame)
target (str)
external_values (dict) – Mapping from index (or group key) to replacement value. If a single value is desired, pass {None: value}.

Return type:

DataFrame

morie.missing.mice(data, m=5, max_iter=10, method='pmm', random_state=None, pmm_k=5)[source]¶

Multiple Imputation by Chained Equations (MICE).

Parameters:

data (DataFrame) – Dataset with missing values.
m (int, default 5) – Number of imputations.
max_iter (int, default 10) – Number of MICE iterations (cycles through all variables).
method (str or dict) – Imputation method per variable or a single method for all: "pmm" (predictive mean matching), "bayesian" (Bayesian linear), "logistic" (binary), "polytomous" (categorical), "rf" (random forest).
random_state (int or None) – Seed for reproducibility.
pmm_k (int, default 5) – Number of nearest neighbours for PMM.

Return type:

MICEResult

References

van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software, 45(3), 1–67.

morie.missing.rubins_rules(estimates, variances, confidence=0.95, dfcom=inf)[source]¶

Pool multiply imputed estimates using Rubin’s rules.

Parameters:

estimates (array-like) – Point estimates from each imputed dataset.
variances (array-like) – Variance estimates from each imputed dataset.
confidence (float, default 0.95)
dfcom (float, default inf) – Complete-data degrees of freedom for the Barnard–Rubin (1999) small-sample adjustment; inf gives Rubin’s large-sample df.

Returns:

As mice::pool.scalar: df is Barnard–Rubin’s, the FMI is (r + 2/(df + 3)) / (r + 1) and the relative efficiency 1 / (1 + fmi/m).

Return type:

PooledEstimate

References

Barnard, J., & Rubin, D. B. (1999). Small-sample degrees of freedom with multiple imputation. Biometrika, 86(4), 948–955.

Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. Wiley. Chapter 3.

morie.missing.pool_mice_estimates(mice_result, analysis_func, confidence=0.95)[source]¶

Apply an analysis function to each imputed dataset and pool with Rubin’s rules.

Parameters:

mice_result (MICEResult) – Output from mice().
analysis_func (callable) – Function that takes a DataFrame and returns (estimate, variance).
confidence (float, default 0.95)

Return type:

PooledEstimate

morie.missing.mice_convergence_data(mice_result)[source]¶

Extract convergence trace data from a MICE result for plotting.

Parameters:: mice_result (MICEResult)
Returns:: Columns: variable, iteration, mean, variance.
Return type:: DataFrame

morie.missing.fraction_missing_information(estimates, variances)[source]¶

Compute the fraction of missing information (FMI).

Parameters:

estimates (array-like) – Point estimates from each imputed dataset.
variances (array-like) – Within-imputation variances.

Returns:

FMI value in [0, 1].

Return type:

morie.missing.relative_efficiency(m, fmi)[source]¶

Compute the relative efficiency of m imputations.

$RE = 1 / (1 + \gamma / m)$ where $\gamma$ is the FMI.

Parameters:

m (int) – Number of imputations.
fmi (float) – Fraction of missing information.

Return type:

morie.missing.tipping_point_analysis(data, outcome, treatment, covariates, mice_result, delta_range=None, confidence=0.95)[source]¶

Tipping-point sensitivity analysis for MNAR.

Shifts imputed values of the outcome by a sequence of delta values and re-analyses to find the delta at which the treatment effect becomes non-significant.

Parameters:

data (DataFrame) – Original data with missing values.
outcome (str) – Outcome variable name.
treatment (str) – Treatment variable name.
covariates (list[str]) – Covariate names.
mice_result (MICEResult) – Fitted MICE result.
delta_range (sequence of float or None) – Deltas to test. If None, uses np.linspace(-2, 2, 21).
confidence (float, default 0.95)

Returns:

Columns: delta, estimate, se, ci_lower, ci_upper, p_value, significant.

Return type:

DataFrame

morie.missing.delta_adjustment(mice_result, variable, delta, group_col=None, group_value=None, original_data=None)[source]¶

Apply a delta adjustment to imputed values for sensitivity analysis.

Adds delta to the imputed values of variable for observations that were originally missing.

Parameters:

mice_result (MICEResult)
variable (str)
delta (float)
group_col (str or None) – If provided, only adjust within this group.
group_value (any) – Value of group_col to restrict adjustment to.
original_data (DataFrame or None) – Original data to identify which values were imputed.

Returns:

New MICEResult with adjusted datasets.

Return type:

MICEResult

morie.missing.select_auxiliary_variables(data, target, r_threshold=0.1, max_vars=20)[source]¶

Select auxiliary variables for imputation model.

Auxiliary variables are those that predict the target or its missingness indicator and should be included in the imputation model even if they are not in the analysis model.

Strategy: Compute the absolute correlation of each observed variable with (1) the target (among complete cases) and (2) the missingness indicator. Variables exceeding r_threshold in either are selected.

Parameters:

data (DataFrame)
target (str)
r_threshold (float, default 0.1) – Minimum absolute correlation to qualify.
max_vars (int, default 20) – Maximum number of auxiliary variables.

Return type:

morie.missing.passive_impute(data, derived_col, formula_func)[source]¶

Passive imputation for derived variables.

After imputing constituent variables, recompute the derived variable using the provided formula function.

Parameters:

data (DataFrame)
derived_col (str) – Name of the derived column.
formula_func (callable) – Function that takes the DataFrame and returns a Series of computed values for the derived column.

Return type:

DataFrame

morie.missing.imputation_diagnostics(original, imputed, variables=None)[source]¶

Compare distributions of observed and imputed values.

Parameters:

original (DataFrame) – Original dataset with missing values.
imputed (DataFrame) – Imputed dataset.
variables (list[str] or None) – Variables to check. If None, all variables that had missing data.

Returns:

Columns: variable, obs_mean, imp_mean, obs_sd, imp_sd, obs_min, imp_min, obs_max, imp_max, ks_statistic, ks_p_value.

Return type:

DataFrame

morie.missing.plot_imputation_density_data(original, imputed_datasets, variable)[source]¶

Generate data for overlaid density plots of observed vs. imputed values.

Parameters:

original (DataFrame)
imputed_datasets (list[DataFrame]) – Multiple imputed datasets.
variable (str)

Returns:

Columns: value, source, imputation_id.

Return type:

DataFrame

Meta-analysis composites: pool a set of estimates, and move between scales.

Parity with rmorie’s morie_meta_pool, morie_meta_effect_sizes and morie_meta_convert (R/ca_crim_native.R).

The arithmetic already lives in morie.fn._ca_crim as separate primitives – inverse-variance weights, Q, I-squared, the DerSimonian-Laird tau-squared, and the effect-size conversions. What was missing is the composite surface that does a whole pooling in one call, which is how the R side is used, so this module composes them rather than restating the formulas.

Why both a fixed and a random-effects answer come back together: the fixed-effect weights assume every study estimates the same quantity, and I-squared is the evidence about whether that holds. Reporting the pooled mean without the heterogeneity beside it is the standard way to overstate a meta-analysis.

morie.meta_analysis.meta_pool(ys, ses, z_cv=1.96, groups=None)[source]¶

Pool estimates by inverse variance, with the heterogeneity beside it.

Returns the fixed-effect mean and its interval, Cochran’s Q, I-squared, the DerSimonian-Laird tau-squared and the random-effects weights that follow from it. With groups, Q is split into within and between components, which is how a subgroup claim is actually tested.

Parameters:: z_cv (float)
Return type:: dict

morie.meta_analysis.meta_effect_sizes(m1=None, m2=None, s1=None, s2=None, n1=None, n2=None, t_value=None, a=None, b=None, c=None, d=None, r=None)[source]¶

Effect sizes from whatever a paper actually reported.

Primary studies report means and SDs, or a t statistic, or a 2x2 table, or a correlation. Each of those determines an effect size and its standard error, and only the ones the arguments support are returned – an absent input yields an absent key rather than a fabricated zero.

Return type:: dict

morie.meta_analysis.meta_convert(ln_or=None, se_ln_or=None, p1=None, p2=None, n1=None, n2=None, d=None, se_d=None, rr=None, or_value=None, r=None, se_r=None)[source]¶

Move an effect between the scales a synthesis has to mix.

A review rarely gets one scale. Log odds ratios, standardised mean differences, probit differences and correlations all convert, and the conversion constants are the ones the literature uses: the logistic SD (pi/sqrt(3)) for the logit route and Cox’s 1.65.

Return type:: dict

Machine Learning sensitivity diagnostics. Wraps SMOTE oversampling and random forest bounds calculations.

morie.ml.eval_robustness(X, y, test_X, test_y)[source]¶

Evaluate the robustness and predictive accuracy of a simple Random Forest classifier.

Parameters:

X (DataFrame) – Training features.
y (Series) – Training labels.
test_X (DataFrame) – Testing features.
test_y (Series) – Testing labels.

Returns:

A dictionary containing the classification report metrics.

Return type:

morie.ml.apply_smote(X, y, *, random_state=42, k_neighbors=None)[source]¶

Apply SMOTE oversampling to balance a binary outcome.

Returns the resampled (X, y) and a status dict with class counts before/after and the method used.

Falls back to random oversampling if imbalanced-learn is not installed or if the minority class has fewer samples than k_neighbors.

Parameters:

X (DataFrame)
y (Series)
random_state (int)
k_neighbors (int | None)

Return type:

tuple[DataFrame, Series, dict[str, Any]]

Survey weight calibration, replication, and diagnostics.

Provides a comprehensive toolkit for constructing, calibrating, trimming, and diagnosing survey weights used in complex survey designs. Supports design weights, post-stratification, raking (iterative proportional fitting), GREG calibration, replicate weight generation (jackknife, BRR, Fay’s BRR, bootstrap, SDR), and non-response adjustments.

All weight functions operate on numpy arrays or pandas Series for interoperability with the rest of the MORIE analysis pipeline.

References

Deville, J.-C. & Sarndal, C.-E. (1992). Calibration estimators in survey sampling. Journal of the American Statistical Association, 87(418), 376–382. https://doi.org/10.1080/01621459.1992.10475217

Kish, L. (1965). Survey Sampling. Wiley.

Lumley, T. (2010). Complex Surveys: A Guide to Analysis Using R. Wiley.

Wolter, K. M. (2007). Introduction to Variance Estimation (2nd ed.). Springer.

Fay, R. E. (1989). Theory and application of replicate weighting for variance calculations. Proceedings of the Survey Research Methods Section, ASA, 212–217.

Judkins, D. R. (1990). Fay’s method for variance estimation. Journal of Official Statistics, 6(3), 223–239.

class morie.weights.WeightDiagnostics(n, sum_weights, mean_weight, median_weight, std_weight, min_weight, max_weight, cv, effective_sample_size, design_effect, weight_range_ratio, n_zero, n_negative, percentiles=<factory>)[source]¶

Bases: object

Summary diagnostics for a set of survey weights.

Parameters:

n (int)
sum_weights (float)
mean_weight (float)
median_weight (float)
std_weight (float)
min_weight (float)
max_weight (float)
cv (float)
effective_sample_size (float)
design_effect (float)
weight_range_ratio (float)
n_zero (int)
n_negative (int)
percentiles (dict[str, float])

n: int¶

sum_weights: float¶

mean_weight: float¶

median_weight: float¶

std_weight: float¶

min_weight: float¶

max_weight: float¶

cv: float¶

effective_sample_size: float¶

design_effect: float¶

weight_range_ratio: float¶

n_zero: int¶

n_negative: int¶

percentiles: dict[str, float]¶

to_dict()[source]¶

Convert to dictionary.

Return type:: dict[str, Any]

class morie.weights.ExtremeWeightReport(n_extreme, threshold_lower, threshold_upper, extreme_indices, extreme_values, pct_extreme)[source]¶

Bases: object

Report on extreme weights.

Parameters:

n_extreme (int)
threshold_lower (float)
threshold_upper (float)
extreme_indices (list[int])
extreme_values (list[float])
pct_extreme (float)

n_extreme: int¶

threshold_lower: float¶

threshold_upper: float¶

extreme_indices: list[int]¶

extreme_values: list[float]¶

pct_extreme: float¶

class morie.weights.WeightInfluence(index, weight, influence, contribution_pct)[source]¶

Bases: object

Influence analysis for individual weights.

Parameters:

index (int)
weight (float)
influence (float)
contribution_pct (float)

index: int¶

weight: float¶

influence: float¶

contribution_pct: float¶

class morie.weights.CalibrationResult(weights, converged, iterations, max_adjustment, diagnostics)[source]¶

Bases: object

Result from a weight calibration procedure.

Parameters:

weights (marr)
converged (bool)
iterations (int)
max_adjustment (float)
diagnostics (WeightDiagnostics)

weights: marr¶

converged: bool¶

iterations: int¶

max_adjustment: float¶

diagnostics: WeightDiagnostics¶

morie.weights.compute_design_weights(selection_probs)[source]¶

Compute design weights as the inverse of selection probabilities.

\[w_i = \frac{1}{\pi_i}\]

where $\pi_i$ is the selection probability for unit i.

Parameters:: selection_probs (numpy.ndarray or pandas.Series) – Inclusion probabilities in (0, 1].
Returns:: Design weights.
Return type:: numpy.ndarray
Raises:: ValueError – If any selection probability is <= 0 or > 1.

Examples

>>> import numpy as np
>>> w = compute_design_weights(np.array([0.5, 0.25, 0.1]))
>>> w.tolist()
[2.0, 4.0, 10.0]

References

Horvitz, D. G. & Thompson, D. J. (1952). A generalization of sampling without replacement from a finite universe. JASA, 47(260), 663–685.

morie.weights.poststratify(weights, strata, population_totals)[source]¶

Post-stratification weight adjustment.

Adjusts weights so that the weighted sum within each stratum matches known population totals.

\[w_i^{\text{ps}} = w_i \cdot \frac{N_h}{\hat{N}_h}\]

where $\hat{N}_h = \sum_{i \in h} w_i$ is the weighted sample count in stratum h and $N_h$ is the population total.

Parameters:

weights (numpy.ndarray or pandas.Series) – Input weights.
strata (numpy.ndarray or pandas.Series) – Stratum membership for each unit.
population_totals (dict) – Known population totals per stratum.

Returns:

Post-stratified weights.

Return type:

Raises:

ValueError – If a stratum in the data has no population total.

Examples

>>> import numpy as np
>>> w = np.array([1.0, 1.0, 1.0, 1.0])
>>> s = np.array(["A", "A", "B", "B"])
>>> w_ps = poststratify(w, s, {"A": 100, "B": 200})
>>> w_ps.tolist()
[50.0, 50.0, 100.0, 100.0]

References

Holt, D. & Smith, T. M. F. (1979). Post stratification. JRSS Series A, 142(1), 33–46.

morie.weights.rake(weights, df, margins, *, max_iterations=100, tolerance=1e-06, bounds=None)[source]¶

Raking (iterative proportional fitting) calibration.

Adjusts weights to match known marginal population totals for multiple categorical variables simultaneously through iterative proportional fitting.

Parameters:

weights (numpy.ndarray or pandas.Series) – Initial weights.
df (pd.DataFrame) – Data frame containing the calibration variables.
margins (dict[str, dict[Any, float]]) – For each calibration variable (key), a dict mapping category values to target population totals.
max_iterations (int) – Maximum iterations (default 100).
tolerance (float) – Convergence tolerance (default 1e-6).
bounds (tuple[float, float] | None) – If provided, clip weight adjustment factors to (lower, upper).

Return type:

References

Deming, W. E. & Stephan, F. F. (1940). On a least squares adjustment of a sampled frequency table when the expected marginal totals are known. Annals of Mathematical Statistics, 11(4), 427–444.

morie.weights.greg_calibrate(weights, X, population_totals, *, max_iterations=50, tolerance=1e-08)[source]¶

Generalized regression (GREG) calibration estimator.

Finds calibration weights $w_i^*$ that minimize the chi-squared distance $\sum_i (w_i^* - w_i)^2 / w_i$ subject to the calibration constraint $\sum_i w_i^* \mathbf{x}_i = \mathbf{T}_x$.

The closed-form solution is:

\[w_i^* = w_i \left(1 + (\mathbf{T}_x - \hat{\mathbf{T}}_x)^\top \left(\sum_j w_j \mathbf{x}_j \mathbf{x}_j^\top\right)^{-1} \mathbf{x}_i\right)\]

Parameters:

weights (numpy.ndarray or pandas.Series) – Initial design weights.
X (numpy.ndarray or pandas.DataFrame) – Calibration variables (n x p matrix).
population_totals (numpy.ndarray) – Known population totals for each calibration variable (length p).
max_iterations (int) – Maximum iterations (default 50; single pass for linear calibration).
tolerance (float) – Convergence tolerance.

Return type:

References

Deville, J.-C. & Sarndal, C.-E. (1992). Calibration estimators in survey sampling. JASA, 87(418), 376–382.

morie.weights.calibrate_to_totals(weights, df, totals, *, method='raking', **kwargs)[source]¶

Calibrate weights to known population totals.

Convenience wrapper that selects the calibration method.

Parameters:

weights (numpy.ndarray or pandas.Series) – Initial weights.
df (pd.DataFrame) – Data frame.
totals (dict) – Column names mapped to a population total (numeric column) or, for raking, to a dict of category -> population count.
method (str) – raking or greg.
**kwargs – Additional arguments passed to the calibration function.

Return type:

morie.weights.trim_weights(weights, *, lower_percentile=1.0, upper_percentile=99.0, method='percentile')[source]¶

Trim extreme weights.

Parameters:

weights (numpy.ndarray or pandas.Series) – Input weights.
lower_percentile (float) – Lower trimming percentile (default 1).
upper_percentile (float) – Upper trimming percentile (default 99).
method (str) – percentile (clip at percentiles) or winsorize (replace extremes with boundary values).

Returns:

Trimmed weights.

Return type:

Examples

>>> import numpy as np
>>> w = np.array([0.1, 1.0, 1.0, 1.0, 100.0])
>>> w_trimmed = trim_weights(w, lower_percentile=5, upper_percentile=95)
>>> w_trimmed[-1] < 100.0
True

References

Potter, F. J. (1990). A study of procedures to identify and trim extreme sampling weights. Proceedings of the Survey Research Methods Section, ASA, 225–230.

morie.weights.smooth_weights(weights, *, method='linear_shrinkage', shrinkage_factor=0.5)[source]¶

Smooth survey weights to reduce variance.

Parameters:

weights (numpy.ndarray or pandas.Series) – Input weights.
method (str) – linear_shrinkage (shrink toward mean) or log_transform (compress via log scale).
shrinkage_factor (float) – Shrinkage intensity in [0, 1]. 0 = no shrinkage, 1 = all equal.

Returns:

Smoothed weights.

Return type:

References

Beaumont, J.-F. & Rivest, L.-P. (2009). Dealing with outliers in survey data. In Handbook of Statistics (Vol. 29A, pp. 247–279). Elsevier.

morie.weights.nonresponse_adjustment(weights, responded, *, adjustment_cells=None)[source]¶

Adjust weights for unit non-response within adjustment cells.

Within each cell, the adjustment factor is:

\[f_c = \frac{\sum_{i \in c} w_i}{\sum_{i \in c, r_i = 1} w_i}\]

where $r_i = 1$ for respondents. Non-respondent weights are set to zero.

Parameters:

weights (numpy.ndarray or pandas.Series) – Initial weights.
responded (numpy.ndarray or pandas.Series) – Boolean indicator (True = responded).
adjustment_cells (numpy.ndarray | pandas.Series | None) – Cell membership for adjustment. If None, uses a single cell.

Returns:

Adjusted weights (non-respondents get zero).

Return type:

References

Kalton, G. & Flores-Cervantes, I. (2003). Weighting methods. Journal of Official Statistics, 19(2), 81–97.

morie.weights.propensity_nonresponse_weights(weights, responded, X, *, model='logistic')[source]¶

Compute non-response adjusted weights using propensity modeling.

Fits a response propensity model and adjusts weights by the inverse of the predicted response probability.

Parameters:

weights (numpy.ndarray or pandas.Series) – Initial weights.
responded (numpy.ndarray or pandas.Series) – Boolean response indicator.
X (numpy.ndarray or pandas.DataFrame) – Covariates for the response propensity model.
model (str) – Model type: logistic (default).

Returns:

Adjusted weights (non-respondents get zero).

Return type:

References

Rosenbaum, P. R. & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.

morie.weights.compute_combined_weights(selection_probs, responded, *, adjustment_cells=None, calibration_strata=None, population_totals=None, trim_percentiles=None)[source]¶

Compute combined weights: design x non-response x calibration.

Applies the full weight construction pipeline:

Design weights (inverse selection probability).
Non-response adjustment (within cells).
Post-stratification calibration (to population totals).
Optional trimming of extreme weights.

Parameters:

selection_probs (numpy.ndarray or pandas.Series) – Selection probabilities.
responded (numpy.ndarray or pandas.Series) – Boolean response indicator.
adjustment_cells (numpy.ndarray | pandas.Series | None) – Non-response adjustment cells.
calibration_strata (numpy.ndarray | pandas.Series | None) – Strata for post-stratification.
population_totals (dict | None) – Population totals per stratum.
trim_percentiles (tuple[float, float] | None) – If provided, trim weights at these percentiles.

Returns:

Final combined weights.

Return type:

morie.weights.normalize_weights(weights, *, target='sample_size', population_size=None)[source]¶

Normalize weights to sum to a target.

Parameters:

weights (numpy.ndarray or pandas.Series) – Input weights.
target (str) – sample_size (sum to n) or population (sum to N).
population_size (float | None) – Required if target='population'.

Returns:

Normalized weights.

Return type:

Examples

>>> from morie.fn import _array_core as np
>>> w = np.array([2.0, 4.0, 6.0])
>>> w_norm = normalize_weights(w, target="sample_size")
>>> bool(np.isclose(w_norm.sum(), 3.0))
True

morie.weights.weight_diagnostics(weights)[source]¶

Compute comprehensive diagnostics for survey weights.

Parameters:: weights (numpy.ndarray or pandas.Series) – Survey weights.
Return type:: WeightDiagnostics

References

Kish, L. (1992). Weighting for unequal Pi. Journal of Official Statistics, 8(2), 183–200.

morie.weights.effective_sample_size(weights)[source]¶

Compute the Kish effective sample size.

\[n_{\text{eff}} = \frac{\left(\sum_i w_i\right)^2}{\sum_i w_i^2}\]

Parameters:: weights (numpy.ndarray or pandas.Series) – Survey weights.
Return type:: float

References

Kish, L. (1965). Survey Sampling, p. 162. Wiley.

morie.weights.design_effect(weights)[source]¶

Compute the design effect (DEFF) from weights.

\[\text{DEFF} = \frac{n}{n_{\text{eff}}}\]

Parameters:: weights (numpy.ndarray or pandas.Series)
Return type:: float

References

Kish, L. (1965). Survey Sampling, p. 162. Wiley.

morie.weights.detect_extreme_weights(weights, *, method='iqr', multiplier=3.0)[source]¶

Detect extreme weights using the IQR or percentile method.

Parameters:

weights (numpy.ndarray or pandas.Series) – Survey weights.
method (str) – iqr (IQR-based) or percentile (1st/99th percentile).
multiplier (float) – IQR multiplier for outlier detection (default 3.0).

Return type:

ExtremeWeightReport

References

Potter, F. J. (1990). A study of procedures to identify and trim extreme sampling weights. Proc. Survey Research Methods Section, ASA.

morie.weights.weight_influence(weights, values)[source]¶

Analyze how much each weight influences the weighted estimate.

For the weighted mean, the influence of unit i is:

\[\text{influence}_i = w_i \cdot (y_i - \bar{y}_w)\]

Parameters:

weights (numpy.ndarray or pandas.Series) – Survey weights.
values (numpy.ndarray or pandas.Series) – Variable of interest.

Returns:

Sorted by absolute influence (descending).

Return type:

list[WeightInfluence]

morie.weights.jackknife_replicate_weights(weights, strata, *, jk_type='JK1')[source]¶

Generate jackknife replicate weights.

For JK1 (unstratified delete-1): generates n replicate columns. For JKn (stratified): generates one replicate per PSU deletion within each stratum.

Parameters:

weights (numpy.ndarray or pandas.Series) – Full-sample weights.
strata (numpy.ndarray or pandas.Series) – Stratum membership (used as PSU grouping for JKn).
jk_type (str) – JK1 (delete-1) or JKn (stratified delete-n).

Returns:

Matrix of shape (n, n_replicates) containing replicate weights.

Return type:

References

Wolter, K. M. (2007). Introduction to Variance Estimation (2nd ed.), Chapter 4. Springer.

morie.weights.brr_replicate_weights(weights, strata, *, n_replicates=None, seed=42)[source]¶

Generate balanced repeated replication (BRR) weights.

For each replicate, within each stratum containing two PSUs, one PSU is given double weight and the other zero weight, according to a Hadamard matrix design.

Parameters:

weights (numpy.ndarray or pandas.Series) – Full-sample weights.
strata (numpy.ndarray or pandas.Series) – Stratum membership. Each stratum should contain exactly 2 PSUs.
n_replicates (int | None) – Number of replicates. If None, uses the smallest Hadamard order >= number of strata.
seed (int) – Random seed for Hadamard matrix permutation.

Returns:

Matrix of shape (n, n_replicates).

Return type:

References

McCarthy, P. J. (1969). Pseudo-replication: half samples. Review of the International Statistical Institute, 37(3), 239–264.

morie.weights.fay_brr_weights(weights, strata, *, fay_coefficient=0.5, n_replicates=None, seed=42)[source]¶

Generate Fay’s BRR replicate weights.

A generalization of BRR where units are perturbed rather than entirely removed, controlled by the Fay coefficient $\rho \in [0, 1)$. When $\rho = 0$, this reduces to standard BRR.

Parameters:

weights (numpy.ndarray or pandas.Series) – Full-sample weights.
strata (numpy.ndarray or pandas.Series) – Stratum membership.
fay_coefficient (float) – Perturbation coefficient in [0, 1). Default 0.5.
n_replicates (int | None) – Number of replicates.
seed (int) – Random seed.

Returns:

Matrix of shape (n, n_replicates).

Return type:

References

Fay, R. E. (1989). Theory and application of replicate weighting for variance calculations. Proc. Survey Research Methods Section, ASA.

morie.weights.bootstrap_replicate_weights(weights, *, n_replicates=200, strata=None, seed=42)[source]¶

Generate bootstrap replicate weights.

For each replicate, resample units with replacement within strata and compute bootstrap weights.

Parameters:

weights (numpy.ndarray or pandas.Series) – Full-sample weights.
n_replicates (int) – Number of bootstrap replicates (default 200).
strata (numpy.ndarray | pandas.Series | None) – If provided, bootstrap within strata.
seed (int) – Random seed.

Returns:

Matrix of shape (n, n_replicates).

Return type:

References

Rao, J. N. K. & Wu, C. F. J. (1988). Resampling inference with complex survey data. JASA, 83(401), 231–241.

morie.weights.sdr_replicate_weights(weights, *, n_replicates=100, seed=42)[source]¶

Generate successive difference replication (SDR) weights.

SDR is designed for systematic samples where units are ordered and variance is estimated from successive differences.

Parameters:

weights (numpy.ndarray or pandas.Series) – Full-sample weights (assumed in sort order).
n_replicates (int) – Number of replicates.
seed (int) – Random seed.

Returns:

Matrix of shape (n, n_replicates).

Return type:

References

Fay, R. E. & Train, G. F. (1995). Aspects of survey and model-based postcensal estimation of income and poverty characteristics. JASA, 90(432), 973–985.

morie.weights.replicate_variance(full_estimate, replicate_estimates, *, method='JK1', fay_coefficient=0.0)[source]¶

Estimate variance using replicate weight estimates.

Parameters:

full_estimate (float) – Estimate from full-sample weights.
replicate_estimates (numpy.ndarray) – Array of estimates from each replicate (length R).
method (str) – Replication method: JK1, JKn, BRR, Fay, bootstrap, SDR.
fay_coefficient (float) – For Fay’s BRR method.

Returns:

Keys: variance, se, ci_lower, ci_upper.

Return type:

dict[str, float]

References

Wolter, K. M. (2007). Introduction to Variance Estimation, Ch. 4-5.

morie.weights.multiframe_weights(weights_a, weights_b, overlap_a, overlap_b, *, method='hartley', theta=0.5)[source]¶

Compute multi-frame survey weights for dual-frame designs.

Parameters:

weights_a (numpy.ndarray or pandas.Series) – Weights from frame A.
weights_b (numpy.ndarray or pandas.Series) – Weights from frame B.
overlap_a (numpy.ndarray or pandas.Series) – Boolean: True if unit from frame A is also in frame B.
overlap_b (numpy.ndarray or pandas.Series) – Boolean: True if unit from frame B is also in frame A.
method (str) – hartley (fixed compositing) or optimal (variance-minimizing).
theta (float) – Compositing factor for overlap domain (default 0.5 = equal).

Returns:

Adjusted (weights_a, weights_b).

Return type:

tuple[numpy.ndarray, numpy.ndarray]

References

Hartley, H. O. (1962). Multiple frame surveys. Proc. Social Statistics Section, ASA, 203–206.

morie.weights.composite_weights(weight_components, *, method='product')[source]¶

Combine multiple weight components into a single composite weight.

Parameters:

weight_components (list[numpy.ndarray]) – List of weight arrays (all same length).
method (str) – product (multiply all) or geometric_mean.

Returns:

Composite weights.

Return type:

morie.weights.weight_diagnostics_report(weights, *, name='Survey Weights', output_path=None)[source]¶

Generate a comprehensive weight diagnostics report.

Parameters:

weights (numpy.ndarray or pandas.Series) – Survey weights.
name (str) – Report title.
output_path (str | Path | None) – If provided, save to this path.

Returns:

Markdown report text.

Return type:

morie.weights.render_diagnostics(diag)[source]¶

Display weight diagnostics with rich formatting.

Parameters:: diag (WeightDiagnostics)
Return type:: None

Inference module Wraps frequentist tests and bayesian inference algorithms.

This module provides:

R-compatible distribution functions (d/p/q/r prefix pattern) for continuous and discrete univariate distributions, all backed by scipy.stats.
Hypothesis testing: t-tests, chi-square, Fisher exact, ANOVA, nonparametric tests, normality tests, and variance-equality tests.
Confidence intervals for proportions, rates, ratios, and differences.
Effect-size estimators: Cohen’s d, Hedges’ g, eta-squared, omega-squared, Cramér’s V, phi coefficient, point-biserial r, Kendall’s tau, Spearman’s rho.
Power analysis: t-test, proportion test, ANOVA, and logistic-regression sample size, all using statsmodels power classes.

References

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.).: Lawrence Erlbaum Associates.
Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap.: Chapman & Hall/CRC.

Agresti, A. (2013). Categorical Data Analysis (3rd ed.). Wiley. Newcombe, R. G. (1998). Interval estimation for the difference between independent

proportions. Statistics in Medicine, 17, 873-890.

Wilson, E. B. (1927). Probable inference, the law of succession, and statistical: inference. Journal of the American Statistical Association, 22, 209-212.

morie.inference.calculate_interaction_power(sample_size, alpha=0.05, effect_size=0.2)[source]¶

Calculate statistical power for an ANOVA F-test, used as an approximation for interaction-term power in the CPADS study design.

The formula 1 - exp(-(n * f) / 50) that appeared in earlier versions of this function is not a valid power formula – it is a heuristic approximation with no grounding in probability theory and produces values that systematically diverge from the correct power at moderate sample sizes. It has been replaced by the exact normal-approximation F-test power computation from statsmodels.stats.power.

The implemented estimator solves:

\[\text{power} = 1 - F_{df_1, df_2, \lambda}(F_{\text{crit}})\]

where $F_{\text{crit}}$ is the critical value under the central F-distribution at level alpha, $\lambda = n \cdot f^2$ is the non-centrality parameter, $f$ is Cohen’s f effect size, $df_1$ defaults to 1 (single-df test), and $df_2 = n - df_1 - 1$.

Parameters:

sample_size (int) – Total number of observations.
alpha (float, optional) – Type I error probability limit (significance level), defaults to 0.05.
effect_size (float, optional) – Cohen’s f effect size for the interaction term, defaults to 0.2 (conventionally “small”).

Returns:

The estimated statistical power (1 - beta), clipped to [0, 1].

Return type:

References

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates.

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146

morie.inference.bootstrap_ci(estimation_func, data, n_iterations=1000, alpha=0.05, seed=42)[source]¶

Compute non-parametric percentile bootstrap confidence intervals.

A fixed random seed is used so that results are reproducible across calls with the same inputs. The default seed (42) is arbitrary; pass a different seed value if you need independent bootstrap draws in a simulation study.

The percentile bootstrap is used rather than the BCa or studentised bootstrap because it requires only a point-estimate function and makes no symmetry assumptions. For highly skewed sampling distributions, consider the BCa bootstrap (Efron & Tibshirani, 1993, Chapter 14).

Parameters:

estimation_func (Callable) – A callable that takes a DataFrame sample and returns a numeric point estimate.
data (pandas.DataFrame) – The pandas DataFrame representing the sample population.
n_iterations (int, optional) – Number of bootstrap iterations to perform, defaults to 1000.
alpha (float, optional) – The significance level for the interval (e.g. 0.05 for a 95% CI), defaults to 0.05.
seed (int, optional) – Random seed for reproducibility. Default 42.

Returns:

A tuple (lower, upper) containing the percentile bootstrap confidence interval bounds.

Return type:

References

Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC. https://doi.org/10.1201/9780429246593

Examples

>>> from morie.fn import _array_core as np
>>> from morie.fn import _frame_core as pd
>>> df = pd.DataFrame({"x": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0]})
>>> lo, hi = bootstrap_ci(lambda d: float(np.mean(d["x"])), df, n_iterations=200, seed=1)
>>> (round(lo, 3), round(hi, 3))
(3.0, 6.128)

morie.inference.dnorm(x, mean=0.0, sd=1.0, log=False)[source]¶

Normal distribution probability density function.

Computes the density of $X \sim \mathcal{N}(\mu, \sigma^2)$:

\[f(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\!\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)\]

Mirrors R’s dnorm(x, mean, sd, log).

Parameters:

x (float | marr) – Quantile(s) at which to evaluate the density.
mean (float) – Mean of the distribution (mu). Default 0.0.
sd (float) – Standard deviation (sigma > 0). Default 1.0.
log (bool) – If True return log-density (for numerical stability). Default False.

Returns:

Density value(s).

Raises:

ValueError – If sd <= 0.

Return type:

float | marr

References

R Core Team (2024). dnorm {stats}. R documentation.

morie.inference.pnorm(x, mean=0.0, sd=1.0, lower_tail=True, log=False)[source]¶

Normal distribution cumulative distribution function.

Mirrors R’s pnorm(x, mean, sd, lower.tail, log.p).

Parameters:

x (float | marr) – Quantile(s).
mean (float) – Mean. Default 0.0.
sd (float) – Standard deviation (> 0). Default 1.0.
lower_tail (bool) – If True (default) compute P(X <= x); else P(X > x).
log (bool) – If True return log-probability. Default False.

Returns:

CDF value(s).

Raises:

ValueError – If sd <= 0.

Return type:

float | marr

References

R Core Team (2024). pnorm {stats}. R documentation.

morie.inference.qnorm(p, mean=0.0, sd=1.0, lower_tail=True, log=False)[source]¶

Normal distribution quantile function (inverse CDF).

Mirrors R’s qnorm(p, mean, sd, lower.tail, log.p).

Parameters:

p (float | marr) – Probability value(s) in (0, 1); or log-probabilities if log=True.
mean (float) – Mean. Default 0.0.
sd (float) – Standard deviation (> 0). Default 1.0.
lower_tail (bool) – If True (default), p = P(X <= x); else p = P(X > x).
log (bool) – If True, p is treated as log(probability). Default False.

Returns:

Quantile(s).

Raises:

ValueError – If sd <= 0.

Return type:

float | marr

References

R Core Team (2024). qnorm {stats}. R documentation.

morie.inference.rnorm(n, mean=0.0, sd=1.0, seed=None)[source]¶

Draw a random sample from a normal distribution.

Mirrors R’s rnorm(n, mean, sd).

Parameters:

n (int) – Number of observations to draw (> 0).
mean (float) – Mean. Default 0.0.
sd (float) – Standard deviation (> 0). Default 1.0.
seed (int | None) – Random seed for reproducibility. Default None.

Returns:

1-D array of length n.

Raises:

ValueError – If n <= 0 or sd <= 0.

Return type:

marr

References

R Core Team (2024). rnorm {stats}. R documentation.

morie.inference.dt(x, df, ncp=None, log=False)[source]¶

Student’s t-distribution probability density function.

Mirrors R’s dt(x, df, ncp, log).

Parameters:

x (float | marr) – Quantile(s).
df (float) – Degrees of freedom (> 0).
ncp (float | None) – Non-centrality parameter. If None, central t is used.
log (bool) – If True return log-density. Default False.

Returns:

Density value(s).

Raises:

ValueError – If df <= 0.

Return type:

float | marr

References

R Core Team (2024). dt {stats}. R documentation.

morie.inference.pt(x, df, lower_tail=True, log=False)[source]¶

Student’s t-distribution CDF.

Mirrors R’s pt(x, df, lower.tail, log.p).

Parameters:

x (float | marr) – Quantile(s).
df (float) – Degrees of freedom (> 0).
lower_tail (bool) – If True compute P(T <= x). Default True.
log (bool) – If True return log-probability. Default False.

Returns:

CDF value(s).

Raises:

ValueError – If df <= 0.

Return type:

float | marr

References

R Core Team (2024). pt {stats}. R documentation.

morie.inference.qt(p, df, lower_tail=True, log=False)[source]¶

Student’s t-distribution quantile function.

Mirrors R’s qt(p, df, lower.tail, log.p).

Parameters:

p (float | marr) – Probability value(s) in (0, 1).
df (float) – Degrees of freedom (> 0).
lower_tail (bool) – If True p = P(T <= x). Default True.
log (bool) – If True, p is log-probability. Default False.

Returns:

Quantile(s).

Raises:

ValueError – If df <= 0.

Return type:

float | marr

References

R Core Team (2024). qt {stats}. R documentation.

morie.inference.dchisq(x, df, log=False)[source]¶

Chi-squared distribution PDF.

Mirrors R’s dchisq(x, df, log).

Parameters:

x (float | marr) – Quantile(s) (>= 0).
df (float) – Degrees of freedom (> 0).
log (bool) – If True return log-density. Default False.

Returns:

Density value(s).

Raises:

ValueError – If df <= 0.

Return type:

float | marr

References

R Core Team (2024). dchisq {stats}. R documentation.

morie.inference.pchisq(x, df, lower_tail=True, log=False)[source]¶

Chi-squared distribution CDF.

Mirrors R’s pchisq(x, df, lower.tail, log.p).

Parameters:

x (float | marr) – Quantile(s).
df (float) – Degrees of freedom (> 0).
lower_tail (bool) – If True compute P(X <= x). Default True.
log (bool) – If True return log-probability. Default False.

Returns:

CDF value(s).

Raises:

ValueError – If df <= 0.

Return type:

float | marr

References

R Core Team (2024). pchisq {stats}. R documentation.

morie.inference.qchisq(p, df, lower_tail=True, log=False)[source]¶

Chi-squared distribution quantile function.

Mirrors R’s qchisq(p, df, lower.tail, log.p).

Parameters:

p (float | marr) – Probability value(s) in (0, 1).
df (float) – Degrees of freedom (> 0).
lower_tail (bool) – If True p = P(X <= x). Default True.
log (bool) – If True p is log-probability. Default False.

Returns:

Quantile(s).

Raises:

ValueError – If df <= 0.

Return type:

float | marr

References

R Core Team (2024). qchisq {stats}. R documentation.

morie.inference.dbinom(x, size, prob, log=False)[source]¶

Binomial distribution probability mass function.

Computes $P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}$.

Mirrors R’s dbinom(x, size, prob, log).

Parameters:

x (int | marr) – Number of successes (integer >= 0).
size (int) – Number of trials (n >= 0).
prob (float) – Success probability per trial (in [0, 1]).
log (bool) – If True return log-PMF. Default False.

Returns:

PMF value(s).

Raises:

ValueError – If size < 0 or prob not in [0, 1].

Return type:

float | marr

References

R Core Team (2024). dbinom {stats}. R documentation.

morie.inference.pbinom(x, size, prob, lower_tail=True, log=False)[source]¶

Binomial distribution CDF.

Mirrors R’s pbinom(x, size, prob, lower.tail, log.p).

Parameters:

x (int | marr) – Quantile(s).
size (int) – Number of trials (>= 0).
prob (float) – Success probability per trial (in [0, 1]).
lower_tail (bool) – If True compute P(X <= x). Default True.
log (bool) – If True return log-probability. Default False.

Returns:

CDF value(s).

Raises:

ValueError – If size < 0 or prob not in [0, 1].

Return type:

float | marr

References

R Core Team (2024). pbinom {stats}. R documentation.

morie.inference.qbinom(p, size, prob, lower_tail=True, log=False)[source]¶

Binomial distribution quantile function.

Returns the smallest integer k such that P(X <= k) >= p. Mirrors R’s qbinom(p, size, prob, lower.tail, log.p).

Parameters:

p (float | marr) – Probability value(s) in (0, 1).
size (int) – Number of trials (>= 0).
prob (float) – Success probability per trial (in [0, 1]).
lower_tail (bool) – If True p = P(X <= x). Default True.
log (bool) – If True p is log-probability. Default False.

Returns:

Quantile(s) as integer(s).

Raises:

ValueError – If size < 0 or prob not in [0, 1].

Return type:

int | marr

References

R Core Team (2024). qbinom {stats}. R documentation.

morie.inference.rbinom(n, size, prob, seed=None)[source]¶

Draw a random sample from a binomial distribution.

Mirrors R’s rbinom(n, size, prob).

Parameters:

n (int) – Number of observations to draw (> 0).
size (int) – Number of trials per observation (>= 0).
prob (float) – Success probability per trial (in [0, 1]).
seed (int | None) – Random seed for reproducibility. Default None.

Returns:

1-D integer array of length n.

Raises:

ValueError – If n <= 0, size < 0, or prob not in [0, 1].

Return type:

marr

References

R Core Team (2024). rbinom {stats}. R documentation.

morie.inference.dpois(x, lambda_, log=False)[source]¶

Poisson distribution probability mass function.

Computes $P(X = k) = e^{-\lambda} \lambda^k / k!$.

Mirrors R’s dpois(x, lambda, log).

Parameters:

x (int | marr) – Non-negative integer value(s).
lambda – Rate parameter (lambda > 0).
log (bool) – If True return log-PMF. Default False.
lambda_ (float)

Returns:

PMF value(s).

Raises:

ValueError – If lambda_ <= 0.

Return type:

float | marr

References

R Core Team (2024). dpois {stats}. R documentation.

morie.inference.ppois(x, lambda_, lower_tail=True, log=False)[source]¶

Poisson distribution CDF.

Mirrors R’s ppois(x, lambda, lower.tail, log.p).

Parameters:

x (int | marr) – Quantile(s).
lambda – Rate parameter (lambda > 0).
lower_tail (bool) – If True compute P(X <= x). Default True.
log (bool) – If True return log-probability. Default False.
lambda_ (float)

Returns:

CDF value(s).

Raises:

ValueError – If lambda_ <= 0.

Return type:

float | marr

References

R Core Team (2024). ppois {stats}. R documentation.

morie.inference.qpois(p, lambda_, lower_tail=True, log=False)[source]¶

Poisson distribution quantile function.

Mirrors R’s qpois(p, lambda, lower.tail, log.p).

Parameters:

p (float | marr) – Probability value(s) in (0, 1).
lambda – Rate parameter (lambda > 0).
lower_tail (bool) – If True p = P(X <= x). Default True.
log (bool) – If True p is log-probability. Default False.
lambda_ (float)

Returns:

Quantile(s).

Raises:

ValueError – If lambda_ <= 0.

Return type:

int | marr

References

R Core Team (2024). qpois {stats}. R documentation.

morie.inference.rpois(n, lambda_, seed=None)[source]¶

Draw a random sample from a Poisson distribution.

Mirrors R’s rpois(n, lambda).

Parameters:

n (int) – Number of observations to draw (> 0).
lambda – Rate parameter (lambda > 0).
seed (int | None) – Random seed for reproducibility. Default None.
lambda_ (float)

Returns:

1-D integer array of length n.

Raises:

ValueError – If n <= 0 or lambda_ <= 0.

Return type:

marr

References

R Core Team (2024). rpois {stats}. R documentation.

morie.inference.dbeta(x, alpha, beta, log=False)[source]¶

Beta distribution probability density function.

Mirrors R’s dbeta(x, shape1, shape2, log).

Parameters:

x (float | marr) – Value(s) in [0, 1].
alpha (float) – First shape parameter (> 0).
beta (float) – Second shape parameter (> 0).
log (bool) – If True return log-density. Default False.

Returns:

Density value(s).

Raises:

ValueError – If alpha <= 0 or beta <= 0.

Return type:

float | marr

References

R Core Team (2024). dbeta {stats}. R documentation.

morie.inference.pbeta(x, alpha, beta, lower_tail=True, log=False)[source]¶

Beta distribution CDF.

Mirrors R’s pbeta(x, shape1, shape2, lower.tail, log.p).

Parameters:

x (float | marr) – Value(s) in [0, 1].
alpha (float) – First shape parameter (> 0).
beta (float) – Second shape parameter (> 0).
lower_tail (bool) – If True compute P(X <= x). Default True.
log (bool) – If True return log-probability. Default False.

Returns:

CDF value(s).

Raises:

ValueError – If alpha <= 0 or beta <= 0.

Return type:

float | marr

References

R Core Team (2024). pbeta {stats}. R documentation.

morie.inference.qbeta(p, alpha, beta, lower_tail=True, log=False)[source]¶

Beta distribution quantile function.

Mirrors R’s qbeta(p, shape1, shape2, lower.tail, log.p).

Parameters:

p (float | marr) – Probability value(s) in (0, 1).
alpha (float) – First shape parameter (> 0).
beta (float) – Second shape parameter (> 0).
lower_tail (bool) – If True p = P(X <= x). Default True.
log (bool) – If True p is log-probability. Default False.

Returns:

Quantile(s).

Raises:

ValueError – If alpha <= 0 or beta <= 0.

Return type:

float | marr

References

R Core Team (2024). qbeta {stats}. R documentation.

morie.inference.dgamma(x, shape, rate=1.0, scale=None, log=False)[source]¶

Gamma distribution probability density function.

Supports both rate and scale parameterisation (rate = 1/scale). When scale is provided it takes precedence over rate. Mirrors R’s dgamma(x, shape, rate, scale, log).

Parameters:

x (float | marr) – Quantile(s) (>= 0).
shape (float) – Shape parameter alpha (> 0).
rate (float) – Rate parameter beta = 1/scale (> 0). Default 1.0.
scale (float | None) – Scale parameter theta = 1/rate. Overrides rate if provided.
log (bool) – If True return log-density. Default False.

Returns:

Density value(s).

Raises:

ValueError – If shape <= 0, or both rate and scale are inconsistent.

Return type:

float | marr

References

R Core Team (2024). dgamma {stats}. R documentation.

morie.inference.pgamma(x, shape, rate=1.0, scale=None, lower_tail=True, log=False)[source]¶

Gamma distribution CDF.

Mirrors R’s pgamma(x, shape, rate, scale, lower.tail, log.p).

Parameters:

x (float | marr) – Quantile(s).
shape (float) – Shape parameter (> 0).
rate (float) – Rate parameter (> 0). Default 1.0.
scale (float | None) – Scale parameter (overrides rate if provided).
lower_tail (bool) – If True compute P(X <= x). Default True.
log (bool) – If True return log-probability. Default False.

Returns:

CDF value(s).

Raises:

ValueError – If shape <= 0.

Return type:

float | marr

References

R Core Team (2024). pgamma {stats}. R documentation.

morie.inference.dunif(x, min=0.0, max=1.0, log=False)[source]¶

Uniform distribution probability density function.

Density is 1/(max - min) for x in [min, max], 0 otherwise. Mirrors R’s dunif(x, min, max, log).

Parameters:

x (float | marr) – Value(s).
min (float) – Lower bound of the support. Default 0.0.
max (float) – Upper bound of the support. Default 1.0.
log (bool) – If True return log-density. Default False.

Returns:

Density value(s).

Raises:

ValueError – If min >= max.

Return type:

float | marr

References

R Core Team (2024). dunif {stats}. R documentation.

morie.inference.punif(x, min=0.0, max=1.0, lower_tail=True, log=False)[source]¶

Uniform distribution CDF.

Mirrors R’s punif(x, min, max, lower.tail, log.p).

Parameters:

x (float | marr) – Value(s).
min (float) – Lower bound. Default 0.0.
max (float) – Upper bound. Default 1.0.
lower_tail (bool) – If True compute P(X <= x). Default True.
log (bool) – If True return log-probability. Default False.

Returns:

CDF value(s).

Raises:

ValueError – If min >= max.

Return type:

float | marr

References

R Core Team (2024). punif {stats}. R documentation.

morie.inference.runif(n, min=0.0, max=1.0, seed=None)[source]¶

Draw a random sample from a uniform distribution.

Mirrors R’s runif(n, min, max).

Parameters:

n (int) – Number of observations to draw (> 0).
min (float) – Lower bound. Default 0.0.
max (float) – Upper bound. Default 1.0.
seed (int | None) – Random seed for reproducibility. Default None.

Returns:

1-D array of length n.

Raises:

ValueError – If n <= 0 or min >= max.

Return type:

marr

References

R Core Team (2024). runif {stats}. R documentation.

morie.inference.two_sample_t_test(x1, x2, *, equal_var=False, alternative='two-sided')[source]¶

Two-sample t-test (Welch or Student).

Default is Welch’s t-test (equal_var=False), which does not assume equal population variances and is generally safer (Ruxton, 2006).

Parameters:

x1 (list | marr) – First sample (1-D array-like).
x2 (list | marr) – Second sample (1-D array-like).
equal_var (bool) – If True use Student’s t-test; if False (default) use Welch’s.
alternative (str) – "two-sided", "greater", or "less".

Returns:

dict with keys t, df, p_value, ci_diff_lower, ci_diff_upper, mean_diff, method.

Raises:

ValueError – If x1 or x2 is empty or alternative is invalid.

Return type:

References

Welch, B. L. (1947). The generalization of Student’s problem when several: different population variances are involved. Biometrika, 34(1-2), 28-35.
Ruxton, G. D. (2006). The unequal variance t-test is an underused alternative: to Student’s t-test and the Mann-Whitney U test. Behavioral Ecology, 17(4).

morie.inference.one_sample_t_test(x, mu0=0.0, *, alternative='two-sided')[source]¶

One-sample t-test against a specified null mean.

Tests H0: mu = mu0 vs Ha depending on alternative.

Parameters:

x (list | marr) – Sample data (1-D array-like).
mu0 (float) – Null hypothesis mean. Default 0.0.
alternative (str) – "two-sided", "greater", or "less".

Returns:

dict with keys t, df, p_value, mean, se, ci_lower, ci_upper.

Raises:

ValueError – If x has fewer than 2 observations.

Return type:

References

Student (1908). The probable error of a mean. Biometrika, 6(1), 1-25.

morie.inference.paired_t_test(x1, x2, *, alternative='two-sided')[source]¶

Paired samples t-test.

Tests H0: mean(x1 - x2) = 0. Requires equal-length samples.

Parameters:

x1 (list | marr) – First sample (1-D array-like).
x2 (list | marr) – Second sample (same length as x1).
alternative (str) – "two-sided", "greater", or "less".

Returns:

dict with keys t, df, p_value, mean_diff, se_diff, ci_lower, ci_upper.

Raises:

ValueError – If samples have different lengths or fewer than 2 pairs.

Return type:

References

Student (1908). The probable error of a mean. Biometrika, 6(1), 1-25.

morie.inference.chi_square_test(observed, *, expected=None)[source]¶

Chi-square goodness-of-fit (or independence) test.

For a 1-D observed array, performs goodness-of-fit against expected (uniform if None). For a 2-D array, performs a chi-square test of independence (expected is ignored).

Parameters:

observed (list | marr) – Observed frequencies. 1-D or 2-D array-like.
expected (list | marr | None) – Expected frequencies for 1-D case. Must sum to same total as observed. Uniform if None.

Returns:

dict with keys chi2, df, p_value, expected.

Raises:

ValueError – If observed contains negative values.

Return type:

References

Agresti, A. (2013). Categorical Data Analysis (3rd ed.). Wiley. (Chapter 1.)

morie.inference.fisher_exact_test(table_2x2)[source]¶

Fisher’s exact test for a 2x2 contingency table.

Appropriate when expected cell counts are small (< 5) and the chi-square approximation is unreliable.

Parameters:: table_2x2 (list | marr) – A 2x2 array-like [[a, b], [c, d]].
Returns:: dict with keys odds_ratio (conditional maximum-likelihood estimate, as R’s fisher.test), ci_lower/ci_upper (exact conditional 95% interval), sample_odds_ratio (ad/bc) and p_value.
Raises:: ValueError – If the table is not 2x2 or contains negative values.
Return type:: dict

References

Fisher, R. A. (1935). The logic of inductive inference. Journal of the Royal: Statistical Society, 98(1), 39-82.

morie.inference.anova_one_way(*groups)[source]¶

One-way ANOVA F-test.

Tests H0: mu_1 = mu_2 = … = mu_k. Assumes independent observations, normality within each group, and homoscedasticity.

Parameters:: groups – Two or more 1-D array-like group samples.
Returns:: dict with keys F, df_between, df_within, p_value, eta_squared.
Raises:: ValueError – If fewer than 2 groups are provided.
Return type:: dict

References

Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver & Boyd. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.).

morie.inference.kruskal_wallis_test(*groups)[source]¶

Kruskal-Wallis H-test (non-parametric one-way ANOVA).

A rank-based alternative to one-way ANOVA that does not assume normality.

Parameters:: groups – Two or more 1-D array-like group samples.
Returns:: dict with keys H, df, p_value.
Raises:: ValueError – If fewer than 2 groups are provided.
Return type:: dict

References

Kruskal, W. H., & Wallis, W. A. (1952). Use of ranks in one-criterion: variance analysis. Journal of the American Statistical Association, 47.

morie.inference.mann_whitney_test(x1, x2, *, alternative='two-sided')[source]¶

Mann-Whitney U test (Wilcoxon rank-sum test).

Non-parametric alternative to the two-sample t-test for ordinal or non-normally distributed data.

Parameters:

x1 (list | marr) – First sample (1-D array-like).
x2 (list | marr) – Second sample (1-D array-like).
alternative (str) – "two-sided", "greater", or "less".

Returns:

dict with keys U, p_value.

Raises:

ValueError – If x1 or x2 is empty or alternative is invalid.

Return type:

References

Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random: variables is stochastically larger than the other. Annals of Mathematical Statistics, 18(1), 50-60.

morie.inference.wilcoxon_signed_rank_test(x1, x2, *, alternative='two-sided')[source]¶

Wilcoxon signed-rank test for paired samples.

Non-parametric alternative to the paired t-test.

Parameters:

x1 (list | marr) – First sample (1-D array-like).
x2 (list | marr) – Second sample (same length as x1).
alternative (str) – "two-sided", "greater", or "less".

Returns:

dict with keys statistic, p_value.

Raises:

ValueError – If samples have different lengths or fewer than 2 pairs.

Return type:

References

Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics: Bulletin, 1(6), 80-83.

morie.inference.shapiro_wilk_test(x)[source]¶

Shapiro-Wilk test for normality.

Tests H0: the sample comes from a normal distribution. Recommended for n <= 2000; use Kolmogorov-Smirnov for larger samples.

Parameters:: x (list | marr) – Sample data (1-D array-like, 3 <= n <= 5000).
Returns:: dict with keys W, p_value, is_normal (True if p_value > 0.05).
Raises:: ValueError – If x has fewer than 3 observations.
Return type:: dict

References

Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for: normality. Biometrika, 52(3-4), 591-611.

morie.inference.levene_test(*groups)[source]¶

Levene’s test for equality of variances.

Tests H0: sigma_1^2 = sigma_2^2 = … = sigma_k^2. Uses the median (Brown-Forsythe variant) for robustness to non-normality.

Parameters:: groups – Two or more 1-D array-like group samples.
Returns:: dict with keys W, df_between, df_within, p_value.
Raises:: ValueError – If fewer than 2 groups are provided.
Return type:: dict

References

Levene, H. (1960). Robust tests for equality of variances. In I. Olkin (Ed.),: Contributions to Probability and Statistics. Stanford University Press.
Brown, M. B., & Forsythe, A. B. (1974). Robust tests for equality of: variances. Journal of the American Statistical Association, 69, 364-367.

morie.inference.proportion_ci(successes, n, *, method='wilson', alpha=0.05)[source]¶

Confidence interval for a single proportion.

Three methods are supported: - "wilson" (default): Wilson score interval – recommended for small n

and proportions near 0 or 1 (Brown, Cai & DasGupta, 2001).

"clopper-pearson": Exact Clopper-Pearson interval – conservative.
"agresti-coull": Agresti-Coull interval – good coverage for moderate n.

Parameters:

successes (int) – Number of successes (0 <= successes <= n).
n (int) – Total number of trials (> 0).
method (str) – CI method. Default "wilson".
alpha (float) – Significance level (default 0.05 -> 95% CI).

Returns:

Tuple (lower, upper).

Raises:

ValueError – If successes < 0, n <= 0, or alpha not in (0, 1).

Return type:

References

Wilson, E. B. (1927). Probable inference, the law of succession, and: statistical inference. JASA, 22, 209-212.
Clopper, C. J., & Pearson, E. S. (1934). The use of confidence or fiducial: limits illustrated in the case of the binomial. Biometrika, 26, 404-413.
Brown, L. D., Cai, T. T., & DasGupta, A. (2001). Interval estimation for a: binomial proportion. Statistical Science, 16(2), 101-133.

morie.inference.rate_ratio_ci(n1, t1, n2, t2, *, alpha=0.05)[source]¶

Incidence rate ratio (IRR) with asymptotic log-normal confidence interval.

IRR = (n1/t1) / (n2/t2)

The CI uses the delta method on the log scale: SE(log IRR) = sqrt(1/n1 + 1/n2) (assuming Poisson counts).

Parameters:

n1 (int) – Event count in group 1 (>= 0).
t1 (float) – Person-time in group 1 (> 0).
n2 (int) – Event count in group 2 (>= 0).
t2 (float) – Person-time in group 2 (> 0).
alpha (float) – Significance level. Default 0.05.

Returns:

dict with keys irr, log_irr, se_log_irr, ci_lower, ci_upper, p_value.

Raises:

ValueError – If t1 <= 0, t2 <= 0, or n1/n2 < 0.

Return type:

References

Rothman, K. J., Greenland, S., & Lash, T. L. (2008). Modern Epidemiology: (3rd ed.). Lippincott Williams & Wilkins. (Chapter 14.)

morie.inference.odds_ratio_ci(table_2x2, *, alpha=0.05, method='exact')[source]¶

Odds ratio with an exact conditional or a Woolf confidence interval.

method="exact" (default): the exact conditional interval of R’s fisher.test (Cornfield 1956), from the noncentral hypergeometric distribution. method="woolf": exp(log OR -/+ z sqrt(1/a + 1/b + 1/c + 1/d)), with 0.5 added to every cell when one is zero (Haldane-Anscombe).

For a 2x2 table [[a, b], [c, d]]: OR = (a * d) / (b * c)

Parameters:

table_2x2 (list | marr) – 2x2 array [[exposed_case, exposed_control], [unexposed_case, unexposed_control]].
alpha (float) – Significance level. Default 0.05.
method (str)

Returns:

dict with keys odds_ratio, ci_lower, ci_upper, p_value.

Raises:

ValueError – If table is not 2x2 or contains negatives.

Return type:

References

Baptista, J., & Pike, M. C. (1977). Exact two-sided confidence limits for the: odds ratio in a 2 × 2 table. Applied Statistics, 26(2), 214-220.

morie.inference.risk_ratio_ci(table_2x2, *, alpha=0.05)[source]¶

Risk ratio (relative risk) with log-normal Wald confidence interval.

For a 2x2 table [[a, b], [c, d]] where rows are exposure and columns are outcome: RR = [a/(a+b)] / [c/(c+d)]

Parameters:

table_2x2 (list | marr) – 2x2 array [[exposed_outcome, exposed_no_outcome], [unexposed_outcome, unexposed_no_outcome]].
alpha (float) – Significance level. Default 0.05.

Returns:

dict with keys risk_ratio, ci_lower, ci_upper, p_value.

Raises:

ValueError – If table is not 2x2 or contains negatives.

Return type:

References

Rothman, K. J., Greenland, S., & Lash, T. L. (2008). Modern Epidemiology: (3rd ed.). Lippincott Williams & Wilkins. (Chapter 4.)

morie.inference.risk_difference_ci(table_2x2, *, alpha=0.05)[source]¶

Risk difference (absolute risk difference, ARD) with Newcombe’s CI.

RD = p1 - p2 = a/(a+b) - c/(c+d)

Uses the Newcombe (1998) score-based method, which has better coverage than the standard Wald interval, especially near the boundary.

Parameters:

table_2x2 (list | marr) – 2x2 array [[a, b], [c, d]].
alpha (float) – Significance level. Default 0.05.

Returns:

dict with keys risk_difference, ci_lower, ci_upper, p_value.

Raises:

ValueError – If table is not 2x2, contains negatives, or row totals are 0.

Return type:

References

Newcombe, R. G. (1998). Interval estimation for the difference between: independent proportions: comparison of eleven methods. Statistics in Medicine, 17, 873-890.

morie.inference.cohens_d(x1, x2, *, pooled=True)[source]¶

Cohen’s d effect size for two independent groups.

Using pooled SD (default): d = (mean_1 - mean_2) / s_pooled

where s_pooled = sqrt([(n1-1)*s1^2 + (n2-1)*s2^2] / (n1+n2-2)).

Using the control-group SD (pooled=False): d = (mean_1 - mean_2) / s_2

Conventional benchmarks: |d| = 0.2 small, 0.5 medium, 0.8 large (Cohen, 1988).

Parameters:

x1 (list | marr) – First group sample.
x2 (list | marr) – Second group sample (reference/control when pooled=False).
pooled (bool) – If True use pooled SD denominator. Default True.

Returns:

Cohen’s d (signed float).

Raises:

ValueError – If either sample has fewer than 2 observations.

Return type:

References

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences: (2nd ed.). Lawrence Erlbaum Associates.

morie.inference.hedges_g(x1, x2)[source]¶

Hedges’ g effect size (bias-corrected Cohen’s d).

Applies Hedges’ correction factor J(m) to reduce small-sample bias: g = d * J(m), where m = n1 + n2 - 2 J(m) = Gamma(m/2) / (sqrt(m/2) Gamma((m-1)/2)) (exact; 1 - 3/(4m-1) for large m)

Parameters:

x1 (list | marr) – First group sample.
x2 (list | marr) – Second group sample.

Returns:

Hedges’ g (signed float).

Raises:

ValueError – If either sample has fewer than 2 observations.

Return type:

References

Hedges, L. V. (1981). Distribution theory for Glass’s estimator of effect: size and related estimators. Journal of Educational Statistics, 6(2), 107-128.

morie.inference.eta_squared(f_stat, df_between, df_within)[source]¶

Eta-squared (eta^2) from ANOVA F-statistic.

eta^2 = SS_between / SS_total = (df_between * F) / (df_between * F + df_within)

Conventional benchmarks: 0.01 small, 0.06 medium, 0.14 large (Cohen, 1988).

Parameters:

f_stat (float) – F-statistic from one-way ANOVA (>= 0).
df_between (int) – Between-groups degrees of freedom (k - 1).
df_within (int) – Within-groups degrees of freedom (N - k).

Returns:

Eta-squared in [0, 1].

Raises:

ValueError – If f_stat < 0 or df values are non-positive.

Return type:

References

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.).

morie.inference.omega_squared(f_stat, df_between, df_within, n)[source]¶

Omega-squared (omega^2) from ANOVA – less biased than eta-squared.

omega^2 = (SS_between - df_between * MS_within) / (SS_total + MS_within): = (df_between * (F - 1)) / (df_between * F + df_within + 1)

Parameters:

f_stat (float) – F-statistic from one-way ANOVA.
df_between (int) – Between-groups df (k - 1).
df_within (int) – Within-groups df (N - k).
n (int) – Total sample size N.

Returns:

Omega-squared; clipped to [0, 1] as negative values are set to 0.

Raises:

ValueError – If f_stat < 0 or df values are non-positive.

Return type:

References

Hays, W. L. (1963). Statistics for Psychologists. Holt, Rinehart and Winston. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.).

morie.inference.cramers_v(contingency_table)[source]¶

Cramér’s V for categorical association (r x c contingency table).

V = sqrt(chi^2 / (n * min(r-1, c-1)))

Ranges from 0 (no association) to 1 (perfect association). Conventional benchmarks for df_min = 1: 0.10 small, 0.30 medium, 0.50 large.

Parameters:: contingency_table (list | marr) – r x c contingency table (non-negative integers).
Returns:: Cramér’s V in [0, 1].
Raises:: ValueError – If table has fewer than 2 rows or columns, or negative entries.
Return type:: float

References

Cramér, H. (1946). Mathematical Methods of Statistics. Princeton University Press. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.).

morie.inference.phi_coefficient(table_2x2)[source]¶

Phi coefficient for association in a 2x2 contingency table.

phi = (ad - bc) / sqrt((a+b)(c+d)(a+c)(b+d))

Equivalent to the Pearson correlation between two binary variables. Ranges from -1 to +1.

Parameters:: table_2x2 (list | marr) – 2x2 array [[a, b], [c, d]].
Returns:: Phi coefficient.
Raises:: ValueError – If table is not 2x2 or marginal totals are zero.
Return type:: float

References

Pearson, K. (1900). Mathematical contributions to the theory of evolution.: Philosophical Transactions of the Royal Society A, 195, 1-47.

morie.inference.point_biserial_r(binary_var, continuous_var)[source]¶

Point-biserial correlation between a binary and a continuous variable.

r_pb = (M_1 - M_0) / s_total * sqrt(n_1 * n_0 / n^2)

Equivalent to the Pearson correlation when one variable is dichotomous.

Parameters:

binary_var (list | marr) – Binary variable (1-D array-like with values 0 and 1).
continuous_var (list | marr) – Continuous variable (same length as binary_var).

Returns:

dict with keys r, p_value.

Raises:

ValueError – If inputs have different lengths or binary_var is not binary.

Return type:

References

Glass, G. V., & Hopkins, K. D. (1996). Statistical Methods in Education and: Psychology (3rd ed.). Allyn & Bacon.

morie.inference.kendall_tau(x, y)[source]¶

Kendall’s tau-b rank correlation coefficient.

A non-parametric measure of ordinal association that is less sensitive to outliers than Spearman’s rho.

Parameters:

x (list | marr) – First variable (1-D array-like).
y (list | marr) – Second variable (same length as x).

Returns:

dict with keys tau, p_value.

Raises:

ValueError – If x and y have different lengths or fewer than 3 observations.

Return type:

References

Kendall, M. G. (1938). A new measure of rank correlation. Biometrika, 30(1-2), 81-93.

morie.inference.spearman_rho(x, y)[source]¶

Spearman rank correlation coefficient (rho).

Pearson correlation applied to ranks; captures monotone (not just linear) associations and is robust to outliers.

Parameters:

x (list | marr) – First variable (1-D array-like).
y (list | marr) – Second variable (same length as x).

Returns:

dict with keys rho, p_value.

Raises:

ValueError – If x and y have different lengths or fewer than 3 observations.

Return type:

References

Spearman, C. (1904). The proof and measurement of association between two things.: American Journal of Psychology, 15(1), 72-101.

morie.inference.power_t_test(n=None, delta=None, sd=1.0, alpha=0.05, power=None, *, alternative='two-sided', type='two-sample')[source]¶

Solve for the one missing quantity (n, delta or power) of a t-test power calculation, exactly as R’s power.t.test (strict = FALSE): power = P(T’ > t_{1-alpha/s, nu}) for the noncentral t with nu = (n - 1) * k and ncp = sqrt(n / k) * delta / sd, where k = 2 for a two-sample test (n per group) and 1 for one-sample or paired, and s = 2 for a two-sided test. Roots are solved to machine precision (R’s uniroot stops at about 1e-4).

References

R Core Team. power.t.test {stats}. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.).

Parameters:

n (float | None)
delta (float | None)
sd (float)
alpha (float)
power (float | None)
alternative (str)
type (str)

Return type:

morie.inference.power_prop_test(n=None, p1=None, p2=None, alpha=0.05, power=None, *, alternative='two-sided')[source]¶

Power of the two-proportion test, exactly as R’s power.prop.test (strict = FALSE), solving for the one missing quantity (n per group, p2 given p1, or power): power = Phi( (sqrt(n) |p1 - p2| - z_{1-alpha/s} sqrt((p1 + p2)(1 - (p1 + p2)/2)))

/ sqrt(p1 (1 - p1) + p2 (1 - p2)) ).

References

R Core Team. power.prop.test {stats}. Fleiss, J. L., Levin, B. & Paik, M. C. (2003). Statistical Methods for Rates and Proportions, ch. 4.

Parameters:

n (float | None)
p1 (float | None)
p2 (float | None)
alpha (float)
power (float | None)
alternative (str)

Return type:

morie.inference.power_anova(n=None, k=None, f=None, alpha=0.05, power=None)[source]¶

Power for one-way ANOVA. Solves for one missing parameter.

Mirrors R’s power.anova.test().

Parameters:

n (float | None) – Number of observations per group.
k (int | None) – Number of groups.
f (float | None) – Cohen’s f effect size (f = sigma_between / sigma_within). Conventional benchmarks: 0.10 small, 0.25 medium, 0.40 large.
alpha (float) – Type I error rate. Default 0.05.
power (float | None) – Desired power.

Returns:

The value of the missing parameter.

Raises:

ValueError – If more than one parameter is None or k < 2.

Return type:

References

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.).: Section 8.

R Core Team (2024). power.anova.test {stats}. R documentation.

morie.inference.sample_size_logistic(*args, **kwargs)[source]¶

Hsieh, Bloch & Larsen (1998) total sample size for logistic regression; see morie.fn.n_logit.sample_size_logistic(), the single implementation.

Return type:: int

Model diagnostics and assumption checking for statistical models.

Provides comprehensive diagnostic tools for regression models (linear, logistic, Cox), including residual analysis, influence diagnostics, collinearity assessment, specification tests, and goodness-of-fit measures.

All diagnostic functions return structured results compatible with the MORIE visualization module for publication-ready diagnostic plots.

class morie.diagnostics.ResidualDiagnostics(raw_residuals, standardized_residuals, studentized_residuals, deviance_residuals, pearson_residuals, fitted_values, normality_test, heteroskedasticity_test, autocorrelation_test, n_outliers, outlier_indices)[source]¶

Bases: object

Comprehensive residual analysis results.

Parameters:

raw_residuals (marr)
standardized_residuals (marr)
studentized_residuals (marr)
deviance_residuals (marr | None)
pearson_residuals (marr | None)
fitted_values (marr)
normality_test (dict[str, float])
heteroskedasticity_test (dict[str, float])
autocorrelation_test (dict[str, float])
n_outliers (int)
outlier_indices (marr)

raw_residuals: marr¶

standardized_residuals: marr¶

studentized_residuals: marr¶

deviance_residuals: marr | None¶

pearson_residuals: marr | None¶

fitted_values: marr¶

normality_test: dict[str, float]¶

heteroskedasticity_test: dict[str, float]¶

autocorrelation_test: dict[str, float]¶

n_outliers: int¶

outlier_indices: marr¶

class morie.diagnostics.InfluenceDiagnostics(hat_values, cooks_distance, dffits, dfbetas, covratio, n_influential, influential_indices, high_leverage_indices, high_cooksd_indices)[source]¶

Bases: object

Influence and leverage diagnostics.

Parameters:

hat_values (marr)
cooks_distance (marr)
dffits (marr)
dfbetas (marr)
covratio (marr)
n_influential (int)
influential_indices (marr)
high_leverage_indices (marr)
high_cooksd_indices (marr)

hat_values: marr¶

cooks_distance: marr¶

dffits: marr¶

dfbetas: marr¶

covratio: marr¶

n_influential: int¶

influential_indices: marr¶

high_leverage_indices: marr¶

high_cooksd_indices: marr¶

class morie.diagnostics.CollinearityDiagnostics(vif, condition_number, condition_indices, variance_decomposition, eigenvalues, n_collinear, collinear_pairs)[source]¶

Bases: object

Multicollinearity assessment results.

Parameters:

vif (dict[str, float])
condition_number (float)
condition_indices (marr)
variance_decomposition (DataFrame)
eigenvalues (marr)
n_collinear (int)
collinear_pairs (list[tuple[str, str, float]])

vif: dict[str, float]¶

condition_number: float¶

condition_indices: marr¶

variance_decomposition: DataFrame¶

eigenvalues: marr¶

n_collinear: int¶

collinear_pairs: list[tuple[str, str, float]]¶

class morie.diagnostics.SpecificationTest(name, statistic, p_value, df, conclusion)[source]¶

Bases: object

Specification/functional form test results.

Parameters:

name (str)
statistic (float)
p_value (float)
df (int | None)
conclusion (str)

name: str¶

statistic: float¶

p_value: float¶

df: int | None¶

conclusion: str¶

class morie.diagnostics.GoodnessOfFit(r_squared, adj_r_squared, pseudo_r_squared, aic, bic, log_likelihood, deviance, pearson_chi2, df_model, df_residual, f_statistic, f_pvalue, n_obs)[source]¶

Bases: object

Model goodness-of-fit summary.

Parameters:

r_squared (float | None)
adj_r_squared (float | None)
pseudo_r_squared (float | None)
aic (float)
bic (float)
log_likelihood (float)
deviance (float | None)
pearson_chi2 (float | None)
df_model (int)
df_residual (int)
f_statistic (float | None)
f_pvalue (float | None)
n_obs (int)

r_squared: float | None¶

adj_r_squared: float | None¶

pseudo_r_squared: float | None¶

aic: float¶

bic: float¶

log_likelihood: float¶

deviance: float | None¶

pearson_chi2: float | None¶

df_model: int¶

df_residual: int¶

f_statistic: float | None¶

f_pvalue: float | None¶

n_obs: int¶

class morie.diagnostics.DiagnosticReport(residuals, influence, collinearity, goodness_of_fit, specification_tests, overall_assessment)[source]¶

Bases: object

Complete diagnostic report for a fitted model.

Parameters:

residuals (ResidualDiagnostics)
influence (InfluenceDiagnostics)
collinearity (CollinearityDiagnostics)
goodness_of_fit (GoodnessOfFit)
specification_tests (list[SpecificationTest])
overall_assessment (str)

residuals: ResidualDiagnostics¶

influence: InfluenceDiagnostics¶

collinearity: CollinearityDiagnostics¶

goodness_of_fit: GoodnessOfFit¶

specification_tests: list[SpecificationTest]¶

overall_assessment: str¶

morie.diagnostics.compute_residuals(y, y_hat, X, model_type='linear')[source]¶

Compute comprehensive residual diagnostics.

Parameters:

y (array-like) – Observed response.
y_hat (array-like) – Fitted/predicted values.
X (array-like) – Design matrix.
model_type (str) – ‘linear’, ‘logistic’, or ‘poisson’.

Return type:

ResidualDiagnostics

morie.diagnostics.compute_influence(y, X, y_hat=None)[source]¶

Compute influence and leverage diagnostics.

Parameters:

y (array-like) – Observed response.
X (array-like) – Design matrix.
y_hat (array-like, optional) – Fitted values. Computed via OLS if not provided.

Return type:

InfluenceDiagnostics

morie.diagnostics.compute_vif(X, column_names=None)[source]¶

Compute Variance Inflation Factors.

Parameters:

X (array-like, shape (n, p)) – Design matrix (without intercept).
column_names (list[str], optional) – Column names.

Returns:

VIF for each predictor.

Return type:

dict[str, float]

morie.diagnostics.collinearity_diagnostics(X, column_names=None)[source]¶

Comprehensive collinearity diagnostics.

Parameters:

X (array-like, shape (n, p)) – Design matrix (with or without intercept).
column_names (list[str], optional) – Column names.

Return type:

CollinearityDiagnostics

morie.diagnostics.ramsey_reset_test(y, X, powers=(2, 3))[source]¶

Ramsey RESET test for functional form misspecification.

Parameters:

y (array-like) – Response variable.
X (array-like) – Design matrix (with intercept).
powers (tuple) – Powers of fitted values to include.

Return type:

morie.diagnostics.link_test(y, X, model_type='linear')[source]¶

Pregibon link test for GLM specification.

Parameters:

y (array-like) – Response.
X (array-like) – Design matrix.
model_type (str) – ‘linear’ or ‘logistic’.

Return type:

morie.diagnostics.hosmer_lemeshow_test(y, y_prob, n_groups=10)[source]¶

Hosmer-Lemeshow goodness-of-fit test for logistic regression.

Parameters:

y (array-like) – Binary response.
y_prob (array-like) – Predicted probabilities.
n_groups (int) – Number of decile groups.

Return type:

morie.diagnostics.compute_goodness_of_fit(y, y_hat, X, model_type='linear', log_likelihood=None)[source]¶

Compute comprehensive goodness-of-fit statistics.

Parameters:

y (array-like) – Observed response.
y_hat (array-like) – Fitted values.
X (array-like) – Design matrix.
model_type (str) – ‘linear’, ‘logistic’, ‘poisson’.
log_likelihood (float, optional) – Log-likelihood of the model (computed if not provided).

Return type:

GoodnessOfFit

morie.diagnostics.ph_assumption_test(survival_times, event_indicator, covariates, covariate_names=None)[source]¶

Test the proportional hazards assumption using Schoenfeld residuals.

Parameters:

survival_times (array-like) – Event or censoring times.
event_indicator (array-like) – 1 if event, 0 if censored.
covariates (array-like) – Covariate matrix.
covariate_names (list[str], optional) – Names for each covariate.

Returns:

One test result per covariate.

Return type:

list[SpecificationTest]

morie.diagnostics.likelihood_ratio_test(ll_restricted, ll_full, df_diff)[source]¶

Likelihood ratio test for nested models.

Parameters:

ll_restricted (float) – Log-likelihood of the restricted model.
ll_full (float) – Log-likelihood of the full model.
df_diff (int) – Difference in degrees of freedom.

Return type:

morie.diagnostics.wald_test(estimates, vcov, R=None, r=None)[source]¶

Wald test for linear restrictions on parameters.

Tests H0: R @ beta = r.

Parameters:

estimates (array-like) – Parameter estimates.
vcov (array-like) – Variance-covariance matrix.
R (array-like, optional) – Restriction matrix. Default: identity (test all params = 0).
r (array-like, optional) – Restriction vector. Default: zeros.

Return type:

morie.diagnostics.score_test(score_vector, information_matrix)[source]¶

Score (Lagrange multiplier) test.

Parameters:

score_vector (array-like) – Score vector evaluated under H0.
information_matrix (array-like) – Expected or observed information matrix under H0.

Return type:

morie.diagnostics.full_diagnostics(y, X, y_hat=None, model_type='linear', column_names=None)[source]¶

Run all diagnostic tests and return a comprehensive report.

Parameters:

y (array-like) – Observed response.
X (array-like) – Design matrix.
y_hat (array-like, optional) – Fitted values.
model_type (str) – ‘linear’, ‘logistic’, ‘poisson’.
column_names (list[str], optional) – Column names for X.

Return type:

DiagnosticReport

morie.signal – Biomedical signal processing facade.

Re-exports all signal processing, fractal complexity, ECG/HRV, PCG, and cepstral analysis functions from morie.fn.*.

from morie.signal import buttlp, ecgdet, hfd, pcgmur

morie.signal.buttlp(x, fs, cutoff, *, order=4)¶

Zero-phase Butterworth lowpass filter.

Parameters:

x (marr) – 1-D input signal.
fs (float) – Sampling frequency in Hz.
cutoff (float) – Cutoff frequency in Hz.
order (int) – Filter order (default 4).

Returns:

SignalResult with filtered signal.

Return type:

morie.signal.butthp(x, fs, cutoff, *, order=4)¶

Zero-phase Butterworth highpass filter.

Parameters:

x (marr) – 1-D input signal.
fs (float) – Sampling frequency in Hz.
cutoff (float) – Cutoff frequency in Hz.
order (int) – Filter order (default 4).

Returns:

SignalResult with filtered signal.

Return type:

morie.signal.buttbp(x, fs, low, high, *, order=4)¶

Zero-phase Butterworth bandpass filter.

Parameters:

x (marr) – 1-D input signal.
fs (float) – Sampling frequency in Hz.
low (float) – Lower cutoff frequency in Hz.
high (float) – Upper cutoff frequency in Hz.
order (int) – Filter order (default 4).

Returns:

SignalResult with filtered signal.

Return type:

morie.signal.buttbs(x, fs, low, high, *, order=4)¶

Zero-phase Butterworth bandstop filter.

Parameters:

x (marr) – 1-D input signal.
fs (float) – Sampling frequency in Hz.
low (float) – Lower cutoff frequency in Hz.
high (float) – Upper cutoff frequency in Hz.
order (int) – Filter order (default 4).

Returns:

SignalResult with filtered signal.

Return type:

morie.signal.sgolay(x, *, window=11, polyorder=3)¶

Savitzky-Golay polynomial smoothing.

Parameters:

x (marr) – 1-D input signal.
window (int) – Window length (must be odd, default 11).
polyorder (int) – Polynomial order (default 3).

Returns:

SignalResult with smoothed signal.

Return type:

morie.signal.welch(x, fs, *, nperseg=256)¶

Welch’s method for power spectral density estimation.

Parameters:

x (marr) – 1-D input signal.
fs (float) – Sampling frequency in Hz.
nperseg (int) – Length of each segment (default 256).

Returns:

SignalResult with PSD in filtered and frequencies in extra["freqs"].

Return type:

morie.signal.pburg(x, fs, *, order=16, nfft=256)¶

Burg AR PSD estimation (parametric, suited for short HRV windows).

Parameters:

x (marr) – 1-D input signal.
fs (float) – Sampling frequency in Hz.
order (int) – AR model order (default 16).
nfft (int) – FFT length for PSD evaluation (default 256).

Returns:

SignalResult with PSD in filtered and frequencies in extra["freqs"].

Return type:

morie.signal.hfd(x, *, kmax=10)¶

Higuchi fractal dimension of a 1-D signal.

Estimates fractal dimension by computing curve length L(k) for k = 1..kmax and fitting log(L) vs log(1/k). The slope is the fractal dimension D (1 <= D <= 2).

Parameters:

x (marr) – 1-D input signal (length >= 2 * kmax).
kmax (int) – Maximum interval (default 10). Auto-reduced if too large.

Returns:

DescriptiveResult with D in value.

Return type:

References

Higuchi, T. (1988). Approach to an irregular time series on the basis of the fractal theory. Physica D, 31(2), 277-283.

morie.signal.kfd(x)¶

Katz fractal dimension of a 1-D signal.

D = log10(L) / log10(d), where L is the total path length and d is the diameter (max distance from first point).

Parameters:: x (marr) – 1-D input signal.
Returns:: DescriptiveResult with D in value.
Return type:: DescriptiveResult

morie.signal.pfd(x)¶

Petrosian fractal dimension of a 1-D signal.

D = log10(N) / (log10(N) + log10(N / (N + 0.4 * N_delta))), where N_delta is the number of sign changes in the first difference.

Parameters:: x (marr) – 1-D input signal.
Returns:: DescriptiveResult with D in value.
Return type:: DescriptiveResult

morie.signal.dfa(x, *, scales=None)¶

Detrended fluctuation analysis (DFA) alpha exponent.

White noise: alpha ~ 0.5. Brownian motion: alpha ~ 1.5.

Parameters:

x (marr) – 1-D input signal (length >= 16).
scales (marr | None) – Array of window sizes. Auto-generated if None.

Returns:

DescriptiveResult with alpha in value.

Return type:

morie.signal.sampen(x, *, m=2, r=0.2)¶

Sample entropy of a 1-D signal.

Parameters:

x (marr) – 1-D input signal.
m (int) – Embedding dimension (default 2).
r (float) – Tolerance as fraction of std (default 0.2).

Returns:

DescriptiveResult with SampEn in value.

Return type:

morie.signal.hurst(series)[source]¶

Hurst exponent via rescaled range (R/S) analysis.

Estimates long-range dependence by computing $(R/S)_n$ over sub-series of increasing length and fitting:

\[\begin{split}\\log(R/S)_n = H \\log n + c\end{split}\]

Parameters:: series (marr) – 1-D array of time series values (length >= 20).
Returns:: dict with H (Hurst exponent, 0 to 1), interpretation ("persistent", "anti-persistent", or "random"), log_n, log_rs (for diagnostic plot).
Raises:: ValueError – If series has fewer than 20 observations.
Return type:: dict

References

Hurst, H. E. (1951). Long-term storage capacity of reservoirs. Transactions of the American Society of Civil Engineers, 116, 770-799.

Mandelbrot, B. B. & Wallis, J. R. (1969). Robustness of the rescaled range R/S in the measurement of noncyclic long run statistical dependence. Water Resources Research, 5(5), 967-988.

morie.signal.ecgdet(ecg, fs)¶

Pan-Tompkins QRS detection algorithm.

Returns R-peak sample indices in extra["r_peaks"].

Parameters:

ecg (marr) – 1-D ECG signal.
fs (float) – Sampling frequency in Hz.

Returns:

SignalResult with R-peak indices.

Return type:

morie.signal.rrint(r_peaks, fs)¶

Compute RR interval series from R-peak sample indices.

Parameters:

r_peaks (marr) – 1-D array of R-peak sample indices.
fs (float) – Sampling frequency in Hz.

Returns:

DescriptiveResult with RR intervals (ms) in extra["rr_ms"].

Return type:

morie.signal.hrvtd(rr)¶

HRV time-domain analysis (SDNN, RMSSD, pNN50).

Parameters:: rr (marr) – 1-D array of RR intervals in milliseconds.
Returns:: DescriptiveResult with metrics in extra.
Return type:: DescriptiveResult

morie.signal.hrvfd(rr, *, fs_interp=4.0)¶

HRV frequency-domain analysis (VLF, LF, HF, LF/HF ratio).

Parameters:

rr (marr) – 1-D array of RR intervals in milliseconds.
fs_interp (float) – Interpolation frequency for uniform resampling (default 4 Hz).

Returns:

DescriptiveResult with power bands in extra.

Return type:

morie.signal.hrvnl(rr)¶

HRV nonlinear analysis (Poincare SD1, SD2).

Parameters:: rr (marr) – 1-D array of RR intervals in milliseconds.
Returns:: DescriptiveResult with SD1, SD2 in extra.
Return type:: DescriptiveResult

morie.signal.pcgflt(pcg, fs, *, low=25.0, high=400.0, order=4)¶

Bandpass filter for phonocardiogram preprocessing.

Parameters:

pcg (marr) – 1-D PCG signal.
fs (float) – Sampling frequency in Hz.
low (float) – Lower cutoff in Hz (default 25).
high (float) – Upper cutoff in Hz (default 400).
order (int) – Filter order (default 4).

Returns:

SignalResult with filtered signal.

Return type:

morie.signal.pcgenv(pcg, fs)¶

Shannon energy envelope for phonocardiogram signals.

Parameters:

pcg (marr) – 1-D PCG signal.
fs (float) – Sampling frequency in Hz.

Returns:

SignalResult with envelope in filtered.

Return type:

morie.signal.pcgseg(envelope, fs=2000.0, *, min_gap_ms=100.0)¶

Segment a PCG envelope into S1 and S2 heart sounds.

Thresholds the envelope, finds contiguous above-threshold regions, merges close peaks, and labels alternating events as S1 (systolic) and S2 (diastolic).

Parameters:

envelope (marr) – 1-D Shannon-energy envelope.
fs (float) – Sampling frequency in Hz.
min_gap_ms (float) – Minimum gap between peaks in ms.

Returns:

DescriptiveResult with cycle count and indices.

Return type:

morie.signal.pcgmur(pcg, fs)¶

Murmur detection from a phonocardiogram signal.

Combines band energy, spectral entropy, and Higuchi fractal dimension to produce a murmur likelihood score (0-1).

Parameters:

pcg (marr) – 1-D PCG signal.
fs (float) – Sampling frequency in Hz.

Returns:

DescriptiveResult with score in value.

Return type:

morie.signal.cepst(x, *, n_fft=None)¶

Real cepstrum: c[n] = IFFT(log|FFT(x)|).

Parameters:

x (marr) – 1-D input signal.
n_fft (int | None) – FFT length (default: next power of 2 >= len(x)).

Returns:

SignalResult with cepstral coefficients in filtered and quefrency axis in extra["quefrency"].

Return type:

morie.signal.hcepst(x, *, n_fft=None)¶

Complex cepstrum with phase unwrapping.

The complex cepstrum is the inverse FFT of log(FFT(x)) where the log uses unwrapped phase.

Parameters:

x (marr) – 1-D input signal.
n_fft (int | None) – FFT length (default: next power of 2 >= len(x)).

Returns:

SignalResult with complex cepstrum in filtered.

Return type:

morie.signal.hdecon(x, *, cutoff, n_fft=None)¶

Homomorphic deconvolution separating minimum-phase and all-pass components.

Uses complex cepstrum + low-time liftering to extract the minimum-phase (impulse response) component.

Parameters:

x (marr) – 1-D input signal (assumed to be a convolution h * e).
cutoff (int) – Liftering cutoff (quefrency index). Cepstral coefficients above this index are zeroed to extract the slow-varying component.
n_fft (int | None) – FFT length (default: next power of 2 >= len(x)).

Returns:

SignalResult with minimum-phase component in filtered and excitation in extra["excitation"].

Return type:

Synchronised longitudinal simulation: SyncRNG + AR / VAR coefficient matrices + multivariate normal generation with structured covariance.

The techniques (synchronised pseudo-random streams, lagged / contemporaneous coefficient matrices, multivariate normal generation under positive-definite Toeplitz / compound-symmetric covariance structures) are standard methods from the longitudinal-modelling and ecological-time-series literature.

Functions:

sync_rng(seed): construct a synchronised RNG that emits the same: stream across the Python and R sides of a workflow (when paired with the R morie::sync_rng_R(seed) wrapper).
generate_ar_coefficients(): construct a p × p autoregressive: coefficient matrix with stationarity-preserving spectral constraints.
generate_var_coefficients(): construct a (p × p × lags) array of: VAR(L) coefficient matrices.
mvn_with_covariance(): draw multivariate normal samples under one: of {independent, AR(1), compound-symmetric, Toeplitz} covariance kernels.
simulate_longitudinal_panel(): top-level driver that ties RNG,: coefficient generation, MVN draws, and missing-data masking into a single tidy long-format DataFrame.

References

Hamilton, J. D. (1994). Time Series Analysis. Princeton University: Press. (VAR coefficient matrices, stationarity criteria.)
Diggle, P. J., Liang, K.-Y., & Zeger, S. L. (1994). Analysis of: Longitudinal Data. Oxford University Press.
L’Ecuyer, P. (1999). Good parameters and implementations for: combined multiple recursive random number generators. Operations Research, 47(1), 159–164.

morie.longitudinal_sim.sync_rng(seed)[source]¶

Return a numpy Generator initialised with seed.

Pairs with the R-side wrapper morie::sync_rng_R(seed) to give bit-identical streams across the two language sides of a cross-language simulation workflow. Both sides use the PCG64 family initialised from the same scalar integer, which is reproducible.

Parameters:: seed (int)
Return type:: _SplitMix64

morie.longitudinal_sim.generate_ar_coefficients(p, *, rng, spectral_radius=0.8, diagonal_bias=0.4)[source]¶

Generate a stationarity-preserving p × p AR coefficient matrix.

Parameters:

p (int) – dimension (number of variables in the panel).
rng (_SplitMix64) – a numpy Generator (see sync_rng).
spectral_radius (float) – target spectral radius < 1 to keep the AR process stationary. Recommended 0.5–0.9 for realistic longitudinal panels.
diagonal_bias (float) – how much weight to put on diagonal vs off- diagonal entries. 0 = pure cross-variable coupling, 1 = each variable purely autoregressive on itself.

Returns:

A p × p numpy array A such that the AR(1) process x_{t+1} = A x_t + ε_t is stationary.

Return type:

marr

morie.longitudinal_sim.generate_var_coefficients(p, lags, *, rng, spectral_radius=0.8, decay=0.6)[source]¶

Generate a p × p × lags VAR(L) coefficient array.

Each lag-l slice has its spectral radius scaled by decay**l so that the companion matrix of the VAR system is stationary.

Parameters:

p (int)
lags (int)
rng (_SplitMix64)
spectral_radius (float)
decay (float)

Return type:

marr

morie.longitudinal_sim.mvn_with_covariance(n, p, *, rng, kernel='ar1', rho=0.5, mean=None)[source]¶

Draw n samples from N_p(mean, Σ) with structured Σ.

Parameters:

n (int) – number of samples.
p (int) – dimension.
rng (_SplitMix64) – numpy Generator.
kernel (Literal['independent', 'ar1', 'compound', 'toeplitz']) – one of “independent”, “ar1”, “compound”, “toeplitz”. “ar1” and “toeplitz” are equivalent here.
rho (float) – correlation parameter (off-diagonal element for compound, decay rate for AR1/toeplitz).
mean (marr | None) – optional p-length mean vector (defaults to zero).

Returns:

An n × p array of samples.

Return type:

marr

morie.longitudinal_sim.simulate_longitudinal_panel(spec)[source]¶

Simulate a longitudinal panel and return as a tidy long-format frame.

Each row is (subject_id, t, variable, value). The simulation:

Seeds an RNG from spec.seed.
Builds a VAR(spec.ar_lags) coefficient array.
Draws subject-specific innovation streams from the chosen covariance kernel.
Iterates the VAR system forward in time.
Optionally masks missing_fraction entries to NaN.
Optionally amplifies outlier_fraction entries by outlier_scale.

Returns:: Tidy long-format DataFrame with columns subject_id, t, variable, value.
Parameters:: spec (LongitudinalSimSpec)
Return type:: DataFrame

Bridge between Go TIDE and Python morie statistical functions.

Three modes:: python -m morie.stat_bridge registry-json -> JSON dump for Go tree python -m morie.stat_bridge help -> formatted help text python -m morie.stat_bridge run “cmd args” -> run command, print result

morie.stat_bridge.registry_json()[source]¶

The morie.fn registry as a JSON list (short name, full name, category, description, quote).

Return type:: str

morie.stat_bridge.help_text()[source]¶

The morie.fn registry listed by category, with a total count.

Return type:: str

morie.stat_bridge.exec_command(cmd_str)[source]¶

Run one statistics command line (name arg ...) and return its output as text.

The name is looked up first among the stat commands, then in the morie.fn registry; errors come back as Error: ... text.

Examples

>>> exec_command("")
'Error: empty command'

Parameters:: cmd_str (str)
Return type:: str

morie.stat_bridge.main()[source]¶: Command-line entry: python -m morie.stat_bridge registry-json|help|run 'cmd'|load-dataset NAME.

ctypes bridge for MORIE semiparametric C kernels.

Loads semipar_kernels.dylib (macOS) or semipar_kernels.so (Linux) for hardware-accelerated kernel density estimation, Nadaraya-Watson regression, local linear regression, and bandwidth selection. Falls back to pure NumPy/SciPy if the shared library is not available.

These primitives underpin the nuisance estimation stage in TMLE, AIPW, and DML causal inference pipelines.

Compile the C library first:

# macOS
cc -O2 -march=native -shared -o semipar_kernels.dylib semipar_kernels.c -lm -framework Accelerate
# Linux
cc -O2 -march=native -shared -fPIC -o semipar_kernels.so semipar_kernels.c -lm

Usage¶

>>> from morie.semipar_bridge import nw_regression, kde, silverman_bandwidth
>>> import numpy as np
>>> rng = np.random.default_rng(42)
>>> x = rng.standard_normal(200)
>>> y = np.sin(x) + 0.3 * rng.standard_normal(200)
>>> h = silverman_bandwidth(x)
>>> x_grid = np.linspace(-3, 3, 100)
>>> y_hat = nw_regression(x, y, x_grid, h)
>>> density = kde(x, x_grid, h)

morie.semipar_bridge.is_available()[source]¶

Return True if the C kernel library is loaded.

Return type:: bool

morie.semipar_bridge.kernel_eval(u, kernel_type=0)[source]¶

Evaluate a kernel function at point u.

Parameters:

u (float) – Evaluation point (scaled by bandwidth).
kernel_type (int) – One of KERNEL_GAUSSIAN (0), KERNEL_EPANECHNIKOV (1), KERNEL_UNIFORM (2), KERNEL_TRIANGULAR (3), KERNEL_BIWEIGHT (4).

Returns:

Kernel value K(u).

Return type:

morie.semipar_bridge.nw_regression(x, y, x_eval, bandwidth)[source]¶

Nadaraya-Watson kernel regression.

\[\hat{m}_h(x) = \frac{\sum_{i=1}^n K_h(x - X_i) Y_i} {\sum_{i=1}^n K_h(x - X_i)}\]

Parameters:

x (array, shape (n,)) – Observed covariate values.
y (array, shape (n,)) – Observed outcomes.
x_eval (array, shape (n_eval,)) – Points at which to evaluate the regression.
bandwidth (float) – Kernel bandwidth h > 0.

Returns:

y_hat – Fitted values.

Return type:

array, shape (n_eval,)

References

Nadaraya, E. A. (1964). On Estimating Regression. Theory of Probability and Its Applications, 9(1), 141–142.

morie.semipar_bridge.local_linear(x, y, x_eval, bandwidth, return_slope=False)[source]¶

Local linear kernel regression (Fan & Gijbels, 1996).

Avoids boundary bias by fitting a local linear model at each point.

Parameters:

x (array, shape (n,)) – Observed covariate values.
y (array, shape (n,)) – Observed outcomes.
x_eval (array, shape (n_eval,)) – Evaluation points.
bandwidth (float) – Kernel bandwidth h > 0.
return_slope (bool) – If True, also return the local slope at each evaluation point.

Returns:

y_hat (array, shape (n_eval,)) – Fitted values.
beta_hat (array, shape (n_eval,), optional) – Local slopes (only if return_slope=True).

Return type:

marr | tuple[marr, marr]

References

Fan, J. & Gijbels, I. (1996). Local Polynomial Modelling and Its Applications. Chapman & Hall.

morie.semipar_bridge.kde(x, x_eval, bandwidth, kernel_type=0)[source]¶

Kernel density estimation.

\[\hat{f}_h(x) = \frac{1}{nh} \sum_{i=1}^n K\!\left(\frac{x - X_i}{h}\right)\]

Parameters:

x (array, shape (n,)) – Observed data.
x_eval (array, shape (n_eval,)) – Evaluation points.
bandwidth (float) – Kernel bandwidth h > 0.
kernel_type (int) – Kernel function (default: KERNEL_GAUSSIAN).

Returns:

density – Estimated density.

Return type:

array, shape (n_eval,)

References

Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis. Chapman & Hall.

morie.semipar_bridge.silverman_bandwidth(x)[source]¶

Silverman rule-of-thumb bandwidth.

\[h = 0.9 \cdot \min\!\left(\hat\sigma,\; \frac{\mathrm{IQR}}{1.34}\right) \cdot n^{-1/5}\]

Parameters:: x (array, shape (n,)) – Observed data.
Returns:: Estimated bandwidth.
Return type:: float

References

Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis, p. 48. Chapman & Hall.

morie.semipar_bridge.loocv_bandwidth(x, y, bw_min=None, bw_max=None, n_grid=30)[source]¶

Leave-one-out cross-validation bandwidth for NW regression.

Minimizes:

\[\mathrm{CV}(h) = \frac{1}{n}\sum_{i=1}^n \left[Y_i - \hat{m}_{h,-i}(X_i)\right]^2\]

Parameters:

x (array, shape (n,)) – Observed covariates.
y (array, shape (n,)) – Observed outcomes.
bw_min (float, optional) – Minimum bandwidth (default: 0.1 * silverman).
bw_max (float, optional) – Maximum bandwidth (default: 2.0 * silverman).
n_grid (int) – Number of grid points to search.

Returns:

Optimal bandwidth.

Return type:

References

Hardle, W. (1990). Applied Nonparametric Regression. Cambridge.

morie.semipar_bridge.kernel_cond_moments(x, y, x_eval, bandwidth, return_variance=True)[source]¶

Kernel-weighted conditional mean and variance.

Used by TMLE/AIPW for smoothing the conditional outcome model.

\[\hat{E}[Y \mid X=x] = \frac{\sum_i K_h(x - X_i)\, Y_i} {\sum_i K_h(x - X_i)}\]

Parameters:

x (array, shape (n,)) – Observed covariates.
y (array, shape (n,)) – Observed outcomes.
x_eval (array, shape (n_eval,)) – Evaluation points.
bandwidth (float) – Kernel bandwidth h > 0.
return_variance (bool) – If True, also return conditional variance.

Returns:

mean (array, shape (n_eval,)) – Conditional mean.
var (array, shape (n_eval,), optional) – Conditional variance (if return_variance=True).

Return type:

marr | tuple[marr, marr]

class morie.semipar_bridge.SemiparKernels[source]¶

Bases: object

Object-oriented interface to semiparametric C kernels.

Wraps the module-level functions and reports which backend is active.

Examples

>>> from morie.fn import _array_core as np
>>> from morie.semipar_bridge import SemiparKernels
>>> sk = SemiparKernels()
>>> sk.backend in ("c", "numpy")
True
>>> x = np.linspace(0.0, 1.0, 50)
>>> y = 2.0 * x
>>> bw = sk.silverman_bandwidth(x)
>>> y_hat = sk.nw_regression(x, y, np.array([0.5]), bw)
>>> round(float(y_hat[0]), 2)       # a straight line is smoothed to itself mid-range
1.0

property available: bool¶: True if the C library is loaded and ready.

property backend: str¶: Return ‘c’ or ‘numpy’ depending on which backend is active.

nw_regression(x, y, x_eval, bandwidth)[source]¶

Nadaraya-Watson kernel regression. See nw_regression().

Parameters:

x (marr)
y (marr)
x_eval (marr)
bandwidth (float)

Return type:

marr

local_linear(x, y, x_eval, bandwidth)[source]¶

Local linear regression. See local_linear().

Parameters:

x (marr)
y (marr)
x_eval (marr)
bandwidth (float)

Return type:

tuple[marr, marr]

kde(x, x_eval, bandwidth, kernel='gaussian')[source]¶

Kernel density estimation. See kde().

Parameters:

x (marr)
x_eval (marr)
bandwidth (float)
kernel (str)

Return type:

marr

loocv_bandwidth(x, y, bw_min=None, bw_max=None, n_grid=30)[source]¶

LOOCV bandwidth selection. See loocv_bandwidth().

Parameters:

x (marr)
y (marr)
bw_min (float | None)
bw_max (float | None)
n_grid (int)

Return type:

silverman_bandwidth(x)[source]¶

Silverman rule-of-thumb bandwidth. See silverman_bandwidth().

Parameters:: x (marr)
Return type:: float

morie.semipar_bridge.compile_semipar_lib(output_dir=None, optimize=True)[source]¶

Attempt to compile the semiparametric kernels C library.

Parameters:

output_dir (str or Path, optional) – Where to write the shared library. Defaults to the morie package dir.
optimize (bool) – Use -O2 -march=native (default True).

Returns:

Path to compiled library, or None on failure.

Return type:

Path or None

Environmental-health composition module for MORIE.

Composes primitives from morie.fn.* (IPW, AIPW, DML, PLR, etc.) into end-to-end pipelines for quantifying how pollution exposure translates into disease burden. Built on top of the M1-canonically-verified causal library (21/22 fns, see an internal coverage report).

The pipeline stages, top-down:

exposure (μg/m³) ─┐
├─-> concentration-response (RR, log-RR)

outcome rate ─┘ │
├─-> attributable fraction (PAF)

population ──────────────┤
├─-> mortality displaced ├─-> burden of pollution (DALYs) ├─-> equity analysis (concentration index) └─-> FSA-level stratification

Each function takes tidy inputs, returns a dataclass, and cites the specific paper whose formula it implements.

References (module-level):

Burnett, R. T. et al. (2014). “An integrated risk function for: estimating the global burden of disease attributable to ambient fine particulate matter exposure.” Environmental Health Perspectives, 122(4), 397-403.
Atkinson, R. W. et al. (2018). “Long-term concentration-response: functions for mortality and NO2.” Environmental Research, 161, 101-113.

WHO (2021). Global Air Quality Guidelines. US EPA (2018). BenMAP-CE User’s Manual Appendices. Rothman, K. J., Greenland, S., & Lash, T. L. (2008). Modern

Epidemiology, 3rd ed., Ch 5 (PAF derivation).

Wagstaff, A., Paci, P., & van Doorslaer, E. (1991). “On the: measurement of inequalities in health.” Social Science & Medicine, 33(5), 545-557.

class morie.envhealth.CRFResult(rr, log_rr, reference_conc, exposure_conc, pollutant, citation, extra=<factory>)[source]¶

Bases: object

Concentration-response function result for a single exposure level.

Parameters:

rr (float)
log_rr (float)
reference_conc (float)
exposure_conc (float)
pollutant (str)
citation (str)
extra (dict[str, Any])

rr: float¶

log_rr: float¶

reference_conc: float¶

exposure_conc: float¶

pollutant: str¶

citation: str¶

extra: dict[str, Any]¶

class morie.envhealth.BurdenResult(paf, attributable_cases, baseline_cases, population, baseline_rate, exposure_mean, reference_conc, pollutant, citation, extra=<factory>)[source]¶

Bases: object

Population-level pollution burden attribution result.

Parameters:

paf (float)
attributable_cases (float)
baseline_cases (float)
population (int)
baseline_rate (float)
exposure_mean (float)
reference_conc (float)
pollutant (str)
citation (str)
extra (dict[str, Any])

paf: float¶

attributable_cases: float¶

baseline_cases: float¶

population: int¶

baseline_rate: float¶

exposure_mean: float¶

reference_conc: float¶

pollutant: str¶

citation: str¶

extra: dict[str, Any]¶

class morie.envhealth.EquityResult(concentration_index, interpretation, n_quintiles, exposure_mean, citation, extra=<factory>)[source]¶

Bases: object

Concentration-index result for exposure × income.

Parameters:

concentration_index (float)
interpretation (str)
n_quintiles (int)
exposure_mean (float)
citation (str)
extra (dict[str, Any])

concentration_index: float¶

interpretation: str¶

n_quintiles: int¶

exposure_mean: float¶

citation: str¶

extra: dict[str, Any]¶

morie.envhealth.concentration_response_pm25(exposure, outcome='all_cause_mortality', reference_conc=5.8)[source]¶

Concentration-response function for PM2.5.

All-cause mortality is log-linear with RR 1.08 per 10 μg/m³, the pooled cohort estimate of the WHO 2021 guideline review (Chen & Hoek 2020):

RR(z) = exp(β · (z − z_cf) / 10), β = ln(1.08)

The cause-specific outcomes use the Integrated Exposure-Response (IER) curve (Burnett et al. 2014, Eq. 1), which was fit per cause and has no all-cause form:

RR(z) = 1 + α · (1 − exp(−γ · (z − z_cf)^δ)) if z > z_cf RR(z) = 1 otherwise

outcome selects the all-cause estimate or a published IER (α, γ, δ) triple.

Parameters:

exposure (float or 1-D array) – Ambient PM2.5 concentration in μg/m³.
outcome (str) – Outcome for which IER parameters are selected. Current options: "all_cause_mortality", "ihd", "stroke".
reference_conc (float) – Counterfactual concentration (z_cf), the level below which no excess risk is assumed. Default 5.8 μg/m³ (WHO 2021 interim target and GBD 2013 counterfactual).

Return type:

CRFResult

References

Chen, J. & Hoek, G. (2020). Environment International, 143, 105974. WHO (2021). Global Air Quality Guidelines. Burnett, R. T. et al. (2014). EHP, 122(4), 397-403.

morie.envhealth.concentration_response_no2(exposure, outcome='all_cause_mortality', reference_conc=10.0, beta_per_10=None)[source]¶

Log-linear concentration-response function for NO2.

Formula (Huangfu & Atkinson 2020, the WHO 2021 guideline review):

RR(z) = exp(β · (z − z_cf) / 10)

where β is the log-RR per 10 μg/m³ increment. Default outcome parameters come from the WHO 2021 Global Air Quality Guidelines evidence synthesis.

Parameters:

exposure (float or 1-D array) – Annual-mean NO2 concentration in μg/m³.
outcome (str) – "all_cause_mortality" (β=ln 1.02=0.0198: RR 1.02, 95% CI 1.01-1.04, per 10 μg/m³), "respiratory" (β=0.029), or "childhood_asthma" (β=0.039).
reference_conc (float) – Counterfactual concentration z_cf. Default 10 μg/m³ (WHO 2021 annual guideline).
beta_per_10 (float, optional) – Override β (log-RR per 10 μg/m³). If given, outcome is ignored for the coefficient (but still stamped on the result for provenance).

Return type:

CRFResult

References

Atkinson, R. W. et al. (2018). Env Research, 161, 101-113. WHO (2021). Global Air Quality Guidelines.

morie.envhealth.attributable_fraction(rr, exposure_prevalence)[source]¶

Population Attributable Fraction (PAF) from a risk ratio.

Formula (Levin 1953; Rothman et al. 2008, Ch 5):

PAF = p · (RR − 1) / (1 + p · (RR − 1))

where p is the prevalence of exposure in the population and RR is the relative risk at that exposure level.

Interpretation: the fraction of outcome cases in the population that would be avoided if the exposure were removed (assuming causation).

Parameters:

rr (float) – Relative risk at the observed exposure level (must be ≥ 1 for a harmful exposure; PAF < 0 for protective exposures).
exposure_prevalence (float) – Proportion of the population exposed, in [0, 1].

Returns:

PAF in (−∞, 1). For RR > 1 and p ∈ [0, 1], PAF ∈ [0, 1 − 1/RR].

Return type:

References

Levin, M. L. (1953). Acta Un Int Cancr, 9, 531-541. Rothman, Greenland & Lash (2008). Modern Epidemiology, 3e, Ch 5.

Examples

>>> round(attributable_fraction(rr=1.08, exposure_prevalence=0.9), 4)   # Levin's formula
0.0672

morie.envhealth.mortality_displaced(exposure_delta, population, baseline_rate, beta_per_unit)[source]¶

Deaths avoided under a counterfactual exposure reduction.

Formula (US EPA BenMAP-CE, Anenberg et al. 2010):

ΔY = y₀ · N · (1 − exp(−β · Δx))

where y₀ is baseline mortality rate, N is the at-risk population, β is the log-RR per unit exposure, and Δx is the counterfactual reduction (positive for reduction). Under small-β approximation:

ΔY ≈ y₀ · N · β · Δx

Parameters:

exposure_delta (float) – Counterfactual exposure reduction (μg/m³). Positive means the exposure goes DOWN; the function returns deaths avoided.
population (int) – At-risk population size.
baseline_rate (float) – Baseline mortality rate (deaths per person-year), e.g. 0.008 for Ontario adult all-cause.
beta_per_unit (float) – Log-RR per μg/m³ (note: per unit, NOT per 10 units). For NO2 all-cause with β_per_10 = ln(1.02) = 0.0198, pass 0.00198 here.

Returns:

Expected avoided deaths per person-year of exposure reduction.

Return type:

References

US EPA (2018). BenMAP-CE User’s Manual Appendices. Anenberg, S. C. et al. (2010). EHP, 118(9), 1189-1195.

morie.envhealth.burden_of_pollution(exposure_mean, exposure_prevalence, baseline_rate, population, pollutant='NO2', outcome='all_cause_mortality', reference_conc=None)[source]¶

End-to-end pollution burden: exposure -> RR -> PAF -> attributable cases.

Chains concentration_response_pm25() or concentration_response_no2(), then attributable_fraction(), to produce the annual attributable case count.

Parameters:

exposure_mean (float) – Population-mean exposure level (μg/m³).
exposure_prevalence (float) – Proportion of the population at the exposure_mean level (typically 1.0 for ambient air pollution – everyone breathes it).
baseline_rate (float) – Outcome rate (cases per person-year) in the counterfactual unexposed scenario.
population (int) – At-risk population.
pollutant (str) – "PM2.5" or "NO2".
outcome (str) – Outcome key passed through to the CRF selector.
reference_conc (float, optional) – Counterfactual concentration; defaults to the pollutant-specific WHO guideline.

Return type:

BurdenResult

References

GBD 2019 Risk Factors Collaborators (2020). Lancet 396:1223-1249.

morie.envhealth.exposure_response_sensitivity(data, *, outcome, exposure, confounders, n_bootstrap=100, random_state=42)[source]¶

Exposure-response DML (PLR) with bootstrap sensitivity.

Wraps morie.fn.plr.estimate_plr() (RidgeCV nuisances, continuous treatment) with non-parametric bootstrap resampling. Each bootstrap iteration refits the PLR pipeline on a resample, yielding a distribution of ATE estimates that captures nuisance-estimation uncertainty beyond the analytic SE.

Why PLR and not estimate_double_ml(): pollution exposure is a continuous variable, so the ml_m (treatment model) must be a regressor, not a classifier. estimate_double_ml uses a RandomForestClassifier for ml_m and raises if the treatment isn’t binary in {0, 1}. PLR uses RidgeCV for both nuisances and handles continuous treatment natively (Chernozhukov et al. 2018 §4.2 PLR model).

Parameters:

data (pd.DataFrame) – Tidy data with exposure, outcome, and confounders as columns.
outcome (str) – Outcome column name.
exposure (str) – Exposure column name (the “treatment” in DML parlance).
confounders (list[str]) – Covariate column names.
n_bootstrap (int) – Number of resamples. Default 100 (enough for a 95% percentile CI).
random_state (int) – Seed (used for bootstrap resampling; PLR’s internal seed is fixed per its implementation).

Returns:

dict with keys (ate, se_analytic, se_bootstrap,)
ci_lower_bs, ci_upper_bs, n_bootstrap, method.

Return type:

References

Chernozhukov et al. (2018). Econometrics Journal, 21(1), C1-C68 §4.2. Bach, P. et al. (2022). DoubleML. JMLR 23(53), 1-6. Efron & Tibshirani (1993). An Introduction to the Bootstrap.

morie.envhealth.pollution_equity_analysis(data, *, exposure, income)[source]¶

Concentration index for pollution exposure by income.

Formula (Wagstaff, Paci & van Doorslaer 1991; O’Donnell et al. 2008):

CI = (2 / μ) · cov(h_i, R_i)

where h_i is the health (or in our case, exposure) variable, R_i is the fractional rank of person i in the income distribution, and μ is the mean of h across the population.

Interpretation:

CI = 0 -> exposure is distributed equally across the income distribution.
CI < 0 -> lower-income individuals bear a disproportionately higher exposure burden (pro-poor pollution burden, adverse equity).
CI > 0 -> higher-income individuals bear disproportionately more exposure (less common for pollution).

Parameters:

data (pd.DataFrame)
exposure (str) – Column containing per-unit exposure (μg/m³).
income (str) – Column containing per-unit income or income proxy.

Return type:

EquityResult

References

Wagstaff, A., Paci, P., & van Doorslaer, E. (1991). Soc Sci Med: 33(5), 545-557.
O’Donnell, O. et al. (2008). Analyzing Health Equity Using Household: Survey Data. World Bank, Ch 8.

morie.envhealth.burden_by_fsa(fsa_table, *, fsa_col='fsa', exposure_col='exposure', population_col='population', baseline_rate_col='baseline_rate', pollutant='NO2', outcome='all_cause_mortality')[source]¶

Per-FSA burden stratification.

Applies burden_of_pollution() to each row of a per-FSA table. Designed for Ontario Forward Sortation Areas (M6H, M5V, etc.) but agnostic to the geographic key.

Parameters:

fsa_table (pd.DataFrame) – One row per FSA with exposure/population/baseline_rate columns.
fsa_col (str)
exposure_col (str)
population_col (str)
baseline_rate_col (str)
pollutant (str)
outcome (str)

Returns:

Input rows plus: rr, paf, attributable_cases, baseline_cases. Sorted descending by attributable_cases so the worst-affected FSAs appear first.

Return type:

pd.DataFrame

References

GBD 2019 Risk Factors Collaborators (2020). Lancet 396:1223-1249.

morie.envhealth.cheatsheet()[source]¶

One-line list of the environmental-health functions in this module.

Examples

>>> cheatsheet().startswith("morie.envhealth:")
True

Return type:: str

morie.earth – geospatial pollution-exposure data ingestion.

Pulls satellite + ground-station air-quality data into pandas DataFrames that plug into the rest of MORIE (causal-inference library, Canadian health datasets, burden-of-disease attribution).

Data sources wired here:

OpenAQ – global ground-station PM/NO2/O3/SO2/CO/BC
(no auth; hits api.openaq.org)

Environment Canada NAPS

– Canadian National Air Pollution Surveillance
(no auth; hits Open-Canada CKAN)

Google Earth Engine

– Sentinel-5P TROPOMI, MODIS, Landsat, Copernicus
(requires service account or user OAuth)

ArcGIS Online

– ESRI Living Atlas + feature services
(requires ArcGIS account or public service URL)

Each fetcher returns either:

a pandas DataFrame (tidy long-format) – station data
a dict of {region_id: numpy array} – raster reductions
a raw GeoDataFrame when geopandas is available; falls back to plain DataFrame

Design goals:

Zero-cloud fallback: OpenAQ + NAPS work without ANY credentials.
Auth gracefully: Earth Engine + ArcGIS raise a clear message if credentials aren’t configured, with exact steps to fix it.
Honest caching: every fetch can be persisted to parquet under data/cache/earth/ keyed by a content hash of the query.
MORIE-native outputs: everything is a DataFrame ready for morie.causal / morie.fn.dml / morie.fn.aipw / etc.

morie.earth.fetch_openaq(country='CA', city=None, pollutants=('pm25', 'no2', 'o3', 'so2', 'co'), date_from=None, date_to=None, limit=10000, use_cache=True)[source]¶

Fetch OpenAQ ground-station measurements as a tidy DataFrame.

Parameters:

Canada). (country ISO-2 country code (default)
"Toronto"). (city Optional city name (e.g.)
codes. (pollutants Iterable of pollutant)
bound. (date_to ISO date/timestamp for upper)
bound.
v2 (limit Max rows per page (API caps at 1000 in)
v3). (10k in)
present. (use_cache Reuse a cached pull under the given query key if)
country (str)
city (str | None)
pollutants (Sequence[str])
date_from (str | None)
date_to (str | None)
limit (int)
use_cache (bool)

Returns:

pandas.DataFrame columns:: – location latitude longitude parameter value unit datetime_utc
Uses public v3 API; no API key required. If you hit rate limits you
can set $OPENAQ_API_KEY and we’ll send it.

morie.earth.fetch_naps(year, pollutant='no2', province=None, use_cache=True)[source]¶

Fetch Environment Canada NAPS annual data via Open-Canada CKAN.

Parameters:

2023). (year Calendar year (e.g.)
``{"no2" (pollutant One of) –
"pm25"
"o3"
"so2"
"co"
"pm10"}``.
"ON"). (province Optional ISO 2-letter province code (e.g.)
year (int)
pollutant (str)
province (str | None)
use_cache (bool)

Returns:

pandas.DataFrame with columns:: – station_id station_name latitude longitude province datetime_local value unit
NAPS is the Canadian ground-truth counterpart to OpenAQ with better
station coverage in Ontario / Quebec / BC. Data lands as CSV under
the CKAN package; we stream the matching resource for the year +
pollutant and return a DataFrame.

morie.earth.fetch_earth_engine(dataset, region, date_from, date_to, band=None, scale=1000, reducer='mean')[source]¶

Pull an Earth Engine ImageCollection reduced over a region/time-window.

Parameters:

(e.g. (dataset EE image-collection asset ID) – "COPERNICUS/S5P/OFFL/L3_NO2" for TROPOMI NO2).
ee.Geometry (region)
dict (GeoJSON)
(lon_min (or)
lat_min
lon_max
tuple. (lat_max))
"2024-01-01"). (date_from ISO date (e.g.)
date. (date_to ISO)
the (band Band name to extract (dataset-specific; defaults to) – first band of the first image).
metres. (scale Pixel scale in)
``{"mean" (reducer One of) –
"max"
"min"
"sum"
"median"}``.
dataset (str)
date_from (str)
date_to (str)
band (str | None)
scale (int)
reducer (str)

Returns:

dict with keys:: – {‘mean’: float, ‘stdDev’: float, ‘count’: int, ‘unit’: str}
Example (Toronto tropospheric NO2 over 2024). Needs the
earthengine-api package and an authenticated Earth Engine account – >>> no2 = fetch_earth_engine( # doctest: +SKIP … dataset=”COPERNICUS/S5P/OFFL/L3_NO2”, … region=(-79.64, 43.58, -79.12, 43.86), # Toronto bbox … date_from=”2024-01-01”, … date_to=”2024-12-31”, … band=”tropospheric_NO2_column_number_density”, … scale=1113, … ) >>> print(no2[“mean”]) # doctest: +SKIP

morie.earth.fetch_arcgis(service_url, where='1=1', out_fields='*', max_records=2000)[source]¶

Fetch an ArcGIS FeatureLayer as a DataFrame (+ geometry if geopandas present).

Parameters:

URL (service_url Full layer) – https://services.arcgis.com/.../FeatureServer/0.
e.g. – https://services.arcgis.com/.../FeatureServer/0.
2023"). (where SQL WHERE clause (e.g. "YEAR =)
all). (out_fields Comma-separated list of attributes ("*" =)
returned. (max_records Max features)
installed (Returns a DataFrame. If geopandas is)
is (geometry)
for (included as a GeoSeries; otherwise lat/lon columns are derived)
layers. (point)
Atlas (This is the simplest entry point for ESRI / ESRI Living)
use (layers that expose feature services. For authed layers)
item (arcgis.gis.GIS directly and pass the authenticated content)
here.
service_url (str)
where (str)
out_fields (str)
max_records (int)

morie.earth.earth_cache_dir()[source]¶

Return (and create) the cache directory for earth-engine results.

Return type:: Path

exception morie.earth.MissingCredentialsError[source]¶

Bases: RuntimeError

Raised when a data source requires credentials that aren’t configured.

The error message always includes exact instructions for how to set the credentials up, so a future maintainer can unblock without reading the docs.

morie.entheo_dmt – loaders + analyses for the Timmermann 2023 DMT EEG-fMRI dataset (github.com/timmer500/DMT_Imaging).

Dataset summary¶

20 subjects, 15 motion-survived (1-3, 5-12, 14-15, 17-19, 20 in fMRI dir).
2 conditions per subject: DMT (drug) and PCB (placebo).
fMRI: BOLD signal parcellated into 112 AAL regions, 840 TRs each (file LongS{NN}{DMT,PCB}.mat, key BOLD_AAL, shape (112, 840)).
EEG: IRASA-decomposed regressors (interpolated + scrubbed) pooled to 5 cortical regions (Central, Frontal, Occipital, Parietal, Temporal). Each region’s .mat file (RegressorsInterpscrubbedIRASA_*) contains three (14, 840, 5) arrays:

regDMT – 14 subjects × 840 TRs × 5 spectral bands under DMT regPCB – 14 subjects × 840 TRs × 5 spectral bands under PCB regdiff – DMT − PCB difference

The 5 bands are the IRASA-fractal-aperiodic decomposition; per the associated paper they index the canonical δ, θ, α, β, γ ranges.

This module exposes two layers:

Layer 1 (loaders): load_fmri_subject(), load_eeg_region(), available_subjects().

Layer 2 (analyses, stubbed): spectral_band_power(), dynamic_functional_connectivity(), lz_complexity(). These are placeholders that raise NotImplementedError; they will delegate to the Rangayyan-style spectral / coherence / complexity primitives in morie.fn.* once those are wired up for EEG.

References

Timmermann, C., Roseman, L., Schartner, M., et al. (2023). Human brain effects of DMT assessed via EEG-fMRI. PNAS 120(13): e2218949120.

morie.entheo_dmt.available_subjects()[source]¶

List the integer IDs of subjects that survive in the fMRI dir.

Returns subject IDs sorted ascending (e.g. [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18, 19, 20] – the motion-corrected subset).

Return type:: list[int]

morie.entheo_dmt.load_fmri_subject(subject_id, condition='DMT')[source]¶

Load a subject’s BOLD AAL parcellation under one condition.

Return type:

np.ndarray of shape (112, 840) – 112 AAL ROIs × 840 TRs.

Parameters:

subject_id (int)
condition (Literal['DMT', 'PCB'])

morie.entheo_dmt.load_eeg_region(region)[source]¶

Load IRASA EEG regressors for one cortical region.

Returns:

dict with keys regDMT, regPCB, regdiff each of shape
(14 subjects, 840 TRs, 5 bands). Bands index canonical δ, θ, α,
β, γ in increasing-frequency order.

Parameters:

region (Literal['Central', 'Frontal', 'Occipital', 'Parietal', 'Temporal'])

Return type:

dict[str, marr]

morie.entheo_dmt.dataset_overview()[source]¶

One-call introspection of the bundled DMT_Imaging dataset.

Return type:: RichResult

morie.entheo_dmt.spectral_band_power(signal, *, fs=200.0, bands=(('delta', 0.5, 4.0), ('theta', 4.0, 8.0), ('alpha', 8.0, 13.0), ('beta', 13.0, 30.0), ('gamma', 30.0, 80.0)), nperseg=None)[source]¶

Welch-PSD band-power decomposition for one EEG channel.

Wraps morie.fn.psdwl.psdwl() to estimate the PSD, then integrates over each canonical band $(f_mathrm{lo}, f_mathrm{hi})$ via the trapezoidal rule. Output is the per-band power (absolute, in V²/Hz·Hz units of the input) plus the same expressed as a fraction of total broadband power.

Parameters:

signal (np.ndarray) – 1-D EEG time series.
fs (float) – Sampling frequency in Hz. Default 200 Hz matches the Timmermann 2023 acquisition.
bands (tuple of (name, f_lo, f_hi)) – Frequency bands. Default = canonical δ/θ/α/β/γ.
nperseg (int or None) – Welch segment length. Defaults to min(len(signal), 4*fs) – i.e. 4-second segments at the default fs.

Return type:

morie.entheo_dmt.dynamic_functional_connectivity(bold, *, window=30, step=5)[source]¶

Sliding-window inter-region BOLD correlation (dRSFC).

For an AAL-parcellated BOLD matrix of shape (n_regions, n_TRs), compute the upper-triangular Pearson correlation matrix in each sliding window of window TRs advanced by step TRs.

Mirrors the dRSFC.m Matlab script in DMT_Imaging/Scripts/. Returns per-window flattened upper-triangular correlation vectors plus the global mean / std across windows.

Parameters:

bold (marr)
window (int)
step (int)

Return type:

morie.entheo_dmt.lz_complexity(signal, *, threshold=None)[source]¶

Lempel-Ziv (LZ76) complexity of a binarised signal.

Wraps morie.fn.lzcmp.lempel_ziv_complexity(). The DMT-vs-PCB contrast on LZ complexity is one of Timmermann 2023’s headline findings: LZ rises under DMT, indicating increased neural-signal diversity.

Parameters:

signal (marr)
threshold (float | None)

Return type:

morie.entheo_dmt.analyze_subject(subject_id, *, conditions=('DMT', 'PCB'), window=30, step=5)[source]¶

Run all three Layer-2 analyses on one subject’s BOLD data under each condition and return a comparison RichResult.

For EEG analyses (band-power, LZ), use load_eeg_region() which returns multi-subject pooled data; this function focuses on the per-subject BOLD pipeline.

Parameters:

subject_id (int)
conditions (tuple)
window (int)
step (int)

Return type:

Forensic toxicology: matrix-aware quantitation and antemortem inference.

A thin domain layer for postmortem forensic toxicology, aimed at the hard case where peripheral blood is compromised (hemodilution after water submersion, or decompositional change) and the analyst must reason from alternative matrices (liver, vitreous humour, brain, gastric contents). Like morie.taphonomy, it adds no new statistics: it supplies typed, zero-row schemas and dispatches quantitation to least squares and the antemortem-vs-artefact question to MORIE’s existing likelihood-ratio machinery (taphonomy_likelihood_ratio()).

This module ships no data. Calibrator responses, case measurements, and matrix concentrations must be supplied by the caller from real analytical runs. tox_matrix_schema() returns a typed zero-row frame; it never fabricates rows.

What it can and cannot do. It fits an internal-standard calibration curve and inverse-predicts an unknown concentration with LOD/LOQ, flags postmortem redistribution from a central:peripheral ratio, and quantifies support for antemortem ingestion versus postmortem microbial production as a likelihood ratio. It reports bounded statistical support, never proof or a posterior probability, and it does not establish cause of death.

R parity: rmorie R/tox.R (morie_tox_*).

References

Dinis-Oliveira RJ, et al. (2010). Collection of biological samples in forensic toxicology. Toxicology Mechanisms and Methods 20(7), 363-414. Pounder DJ, Jones GR (1990). Post-mortem drug redistribution. Forensic Science International 45(3), 253-263. Kugelberg FC, Jones AW (2007). Interpreting results of ethanol analysis in postmortem specimens. Forensic Science International 165(1), 10-29.

morie.tox.tox_matrix_schema()[source]¶

Return an empty, typed DataFrame for a forensic-toxicology sample table.

No rows are fabricated. Each column’s role (identifier / matrix / measurement / quality) is stored in df.attrs['role']. R parity: morie_tox_matrix_schema.

Return type:: DataFrame

morie.tox.tox_calibration(conc, response, weights='1/x^2', response_unknown=None)[source]¶

Fit an internal-standard calibration curve and inverse-predict an unknown.

Weighted linear least squares of instrument response on known calibrator concentration, the basis for GC-MS / LC-MS/MS quantitation. LOD/LOQ follow the ICH Q2 residual-standard-error convention (LOD = 3.3 s/b, LOQ = 10 s/b). With response_unknown, inverse-predicts and flags against LOD/LOQ. R parity: morie_tox_calibration.

Parameters:

weights (str | marr)
response_unknown (float | None)

Return type:

morie.tox.tox_pmr_ratio(central, peripheral)[source]¶

Flag postmortem redistribution from a central:peripheral (C/P) ratio.

PMR inflates central-blood drug concentrations relative to peripheral after death. C/P near 1 is stable; larger ratios indicate redistribution and unreliable central quantitation. R parity: morie_tox_pmr_ratio.

Parameters:

central (float)
peripheral (float)

Return type:

morie.tox.tox_antemortem_lr(marker, antemortem, postmortem)[source]¶

Likelihood ratio for antemortem ingestion versus postmortem artefact.

The caller supplies an observed marker (typically a secondary metabolite a living liver produces, e.g. ethyl glucuronide, or a congener/matrix ratio) and Gaussian models {"mean":.., "sd":..} for the two hypotheses. Thin wrapper over taphonomy_preservation_lr(). R parity: morie_tox_antemortem_lr.

Parameters:

antemortem (dict[str, Any])
postmortem (dict[str, Any])

Return type:

morie.tox.tox_matrix_reliability(matrix=None, submersion_days=0.0, decomp_stage=0.0)[source]¶

Rank specimen matrices by reliability under submersion and decomposition.

Encodes the documented resistance-to-degradation ordering (vitreous humour and brain are protected; the liver concentrates; blood-based matrices dilute) as a transparent heuristic, penalised by days submerged and decomposition stage. A documented ordering, not an empirical prior. Returns a DataFrame of matrix / reliability (0-1) / rank. R parity: morie_tox_matrix_reliability.

Parameters:

matrix (list[str] | None)
submersion_days (float)
decomp_stage (float)

Return type:

DataFrame

morie.tox.tox_left_censor_impute(values, lod, method='half')[source]¶

Impute left-censored (below-LOD) toxicology values.

Applies a documented simple-substitution rule to entries below lod. A NaN is not measured, not below the detection limit: it is left as NaN, excluded from the censoring fraction, and counted in n_missing. For regression on censored data prefer morie.horowitz_censored_regression. R parity: morie_tox_left_censor_impute.

Parameters:

lod (float)
method (str)

Return type:

morie.tox.tox_ethanol_congeners(ethanol, n_propanol=0.0, n_butanol=0.0, etg=nan, congener_threshold=0.0)[source]¶

Adjudicate ethanol as antemortem ingestion versus postmortem production.

Higher alcohols (n-propanol, n-butanol) are fermentation by-products, so their presence points to postmortem microbial production; ethyl glucuronide (EtG) is produced only by living metabolism, so its presence points to antemortem intake (Kugelberg & Jones 2007). For a quantified LR, feed EtG to tox_antemortem_lr(). R parity: morie_tox_ethanol_congeners.

Parameters:

ethanol (float)
n_propanol (float)
n_butanol (float)
etg (float)
congener_threshold (float)

Return type:

MORIE post-quantum cryptography (research/educational).

WARNING: This is a pure-Python reference implementation of NIST-standardized post-quantum cryptographic primitives (ML-KEM-768, ChaCha20-Poly1305). It is NOT constant-time and NOT suitable for production secrets. For production-grade encryption, use hardware KMS or audited C implementations.

Pure-Python ML-KEM is ~100x slower than the reference C implementation.

Examples

>>> import os
>>> from morie.crypto import chacha20_poly1305_decrypt, chacha20_poly1305_encrypt, hkdf_sha256
>>> key, nonce = os.urandom(32), os.urandom(12)
>>> ct, tag = chacha20_poly1305_encrypt(key, nonce, b"attack at dawn")
>>> chacha20_poly1305_decrypt(key, nonce, ct, tag)
b'attack at dawn'
>>> hkdf_sha256(b"k" * 32, length=8, salt=b"s", info=b"i").hex()
'7a1153df5b0dbe89'

morie.crypto.mlkem768_keygen()[source]¶

Generate ML-KEM-768 key pair.

Returns:: (public_key, secret_key) as bytes.
Return type:: tuple[bytes, bytes]

morie.crypto.mlkem768_encaps(pk)[source]¶

Encapsulate a shared secret using the public key (FIPS 203 Algorithm 17).

Parameters:: pk (bytes) – Public key from mlkem768_keygen().
Returns:: (ciphertext, shared_secret_32_bytes).
Return type:: tuple[bytes, bytes]

morie.crypto.mlkem768_decaps(sk, ct)[source]¶

Decapsulate the shared secret using the secret key (FIPS 203 Algorithm 18).

Re-encrypts the recovered message and compares the ciphertexts; on a mismatch the implicit-rejection key J(z || ct) is returned instead of an error, so a tampered ciphertext yields a different secret rather than a signal.

Parameters:

sk (bytes) – Secret key from mlkem768_keygen().
ct (bytes) – Ciphertext from mlkem768_encaps().

Returns:

32-byte shared secret.

Return type:

morie.crypto.chacha20_poly1305_encrypt(key, nonce, plaintext, aad=b'')[source]¶

ChaCha20-Poly1305 AEAD encryption (RFC 8439).

Parameters:

key (bytes) – 32-byte key.
nonce (bytes) – 12-byte nonce.
plaintext (bytes) – Data to encrypt.
aad (bytes) – Additional authenticated data.

Returns:

(ciphertext, 16-byte tag).

Raises:

ValueError – If key or nonce has wrong length.

Return type:

tuple[bytes, bytes]

morie.crypto.chacha20_poly1305_decrypt(key, nonce, ciphertext, tag, aad=b'')[source]¶

ChaCha20-Poly1305 AEAD decryption (RFC 8439).

Parameters:

key (bytes) – 32-byte key.
nonce (bytes) – 12-byte nonce.
ciphertext (bytes) – Data to decrypt.
tag (bytes) – 16-byte authentication tag.
aad (bytes) – Additional authenticated data.

Returns:

Decrypted plaintext.

Raises:

ValueError – If authentication fails or parameters invalid.

Return type:

morie.crypto.hkdf_sha256(ikm, length=32, salt=b'', info=b'')[source]¶

HKDF-SHA256 key derivation function.

Parameters:

ikm (bytes) – Input keying material.
length (int) – Output length in bytes (max 8160).
salt (bytes) – Optional salt (if empty, uses zero-filled hash-length).
info (bytes) – Optional context/application info.

Returns:

Derived key material.

Return type:

morie.crypto.hybrid_encrypt(plaintext, recipient_pk)[source]¶

Encrypt data using hybrid ML-KEM-768 + ChaCha20-Poly1305.

Encapsulate with the recipient’s ML-KEM public key: a ciphertext and a shared secret.
Derive the wrapping key from HKDF(shared_secret || kem_ct || pk).
Generate a random 32-byte symmetric key, wrap it with ChaCha20-Poly1305.
Encrypt the plaintext with the symmetric key, behind the 1.4.0 container marker.

Parameters:

plaintext (bytes) – Data to encrypt (arbitrary length).
recipient_pk (bytes) – Recipient’s ML-KEM-768 public key.

Returns:

Serialized ciphertext container.

Return type:

morie.crypto.hybrid_decrypt(ciphertext, recipient_sk)[source]¶

Decrypt hybrid ML-KEM-768 + ChaCha20-Poly1305 ciphertext.

Parameters:

ciphertext (bytes) – Serialized container from hybrid_encrypt().
recipient_sk (bytes) – Recipient’s ML-KEM-768 secret key.

Returns:

Decrypted plaintext.

Raises:

ValueError – If the ciphertext is malformed or authentication fails.

Return type:

morie.crypto.hybrid_keygen()¶

Generate an ML-KEM-768 key pair for hybrid encryption.

Convenience wrapper around morie.crypto.mlkem768_keygen().

Returns:: (public_key, secret_key) as bytes.
Return type:: tuple[bytes, bytes]

Examples

>>> pk, sk = keygen()
>>> (len(pk), len(sk))
(1184, 2400)
>>> hybrid_decrypt(hybrid_encrypt(b"the report", pk), sk)
b'the report'

morie.crypto.create_keystore(password, path='~/.morie/keys/keystore.json')[source]¶

Create a new empty keystore.

Parameters:

password (str) – Password to protect the keystore.
path (str) – File path (default ~/.morie/keys/keystore.json).

Raises:

FileExistsError – If the keystore already exists.

Return type:

None

morie.crypto.store_keypair(name, pk, sk, password, path='~/.morie/keys/keystore.json')[source]¶

Store a key pair in the keystore.

The public key is stored in the clear. The secret key is encrypted with ChaCha20-Poly1305 using a password-derived key.

Parameters:

name (str) – Identifier for this key pair.
pk (bytes) – Public key bytes.
sk (bytes) – Secret key bytes.
password (str) – Keystore password.
path (str) – Keystore path.

Raises:

FileNotFoundError – If the keystore does not exist.

Return type:

None

morie.crypto.load_keypair(name, password, path='~/.morie/keys/keystore.json')[source]¶

Load a key pair from the keystore.

Parameters:

name (str) – Identifier of the key pair.
password (str) – Keystore password.
path (str) – Keystore path.

Returns:

(public_key, secret_key) as bytes.

Raises:

KeyError – If the name is not found.
ValueError – If the password is wrong (tag mismatch).

Return type:

tuple[bytes, bytes]

morie.crypto.list_keys(password, path='~/.morie/keys/keystore.json')[source]¶

List key names in the keystore.

Verifies the password by attempting to derive the key (the store must exist and the salt must be valid).

Parameters:

password (str) – Keystore password.
path (str) – Keystore path.

Returns:

List of key names.

Return type:

morie.crypto.mldsa_keygen(level=65, seed=None)[source]¶

Generate an ML-DSA key pair.

Parameters:

level (int) – 44, 65 (default) or 87 – ML-DSA-44/65/87.
seed (bytes | None) – Optional 32-byte seed; the same seed gives the same key pair (FIPS 204 KeyGen_internal).

Returns:

(public_key, secret_key) as bytes (1312/2560, 1952/4032 or 2592/4896 bytes).

Return type:

tuple[bytes, bytes]

>>> pk, sk = mldsa_keygen(44, seed=bytes(32))
>>> len(pk), len(sk)
(1312, 2560)

morie.crypto.mldsa_sign(message, sk_bytes, context=b'', deterministic=False)[source]¶

Sign a message with ML-DSA (pure ML-DSA, FIPS 204 Algorithm 2).

Parameters:

message – Message (bytes, or str as UTF-8).
sk_bytes (bytes) – Secret key from mldsa_keygen(); its length gives the parameter set.
context (bytes) – Context string, at most 255 bytes (empty by default).
deterministic (bool) – Use the deterministic variant (32 zero bytes of signing randomness) instead of the hedged default.

Returns:

The signature (2420, 3309 or 4627 bytes).

Return type:

>>> pk, sk = mldsa_keygen(44, seed=bytes(32))
>>> len(mldsa_sign(b"manifest", sk))
2420

morie.crypto.mldsa_verify(message, signature, pk_bytes, context=b'')[source]¶

Verify an ML-DSA signature; False for any signature, key or context that does not fit.

>>> pk, sk = mldsa_keygen(44, seed=bytes(32))
>>> sig = mldsa_sign(b"manifest", sk)
>>> mldsa_verify(b"manifest", sig, pk), mldsa_verify(b"other", sig, pk)
(True, False)

Parameters:

signature (bytes)
pk_bytes (bytes)
context (bytes)

Return type:

morie.crypto.ntru_keygen(n=167, q=128, p=3)[source]¶

Generate an NTRU key pair.

Parameters:

n (int) – Polynomial degree (should be prime).
q (int) – Large modulus (power of 2).
p (int) – Small modulus (typically 3).

Returns:

dict with pk (h polynomial), sk (f, fp polynomials), params.

Return type:

morie.crypto.ntru_encrypt(message, pk, n=167, q=128)[source]¶

Encrypt a message polynomial with NTRU.

Parameters:

message (list[int]) – Message polynomial (coefficients in {-1, 0, 1}).
pk (list[int]) – Public key polynomial h.
n (int) – Polynomial degree.
q (int) – Modulus.

Returns:

Ciphertext polynomial.

Return type:

morie.crypto.ntru_decrypt(ciphertext, sk, n=167, q=128, p=3)[source]¶

Decrypt an NTRU ciphertext.

Parameters:

ciphertext (list[int]) – Ciphertext polynomial.
sk (dict) – Secret key dict with f and fp.
n (int) – Polynomial degree.
q (int) – Large modulus.
p (int) – Small modulus.

Returns:

Recovered message polynomial.

Return type:

morie.crypto.mceliece_keygen(m=4, t=2)[source]¶

Generate a simplified McEliece key pair.

Parameters:

m (int) – GF(2^m) extension degree.
t (int) – Error-correcting capability.

Returns:

dict with pk (public matrix T), sk (decoder info), params.

Return type:

morie.crypto.mceliece_encaps(pk)[source]¶

Encapsulate a key using McEliece.

Parameters:: pk (marr) – Public parity-check matrix (permuted).
Returns:: (shared_secret, ciphertext_syndrome).
Return type:: tuple[bytes, marr]

morie.crypto.mceliece_decaps(syndrome, sk)[source]¶

Decapsulate a McEliece ciphertext.

Parameters:

syndrome (marr) – Syndrome from mceliece_encaps().
sk (dict) – Secret key from mceliece_keygen().

Returns:

32-byte shared secret.

Return type:

morie.crypto.ntt(f, q, n, zetas)[source]¶

Forward Number Theoretic Transform over Z_q.

Cooley-Tukey butterfly, decimation-in-time. When zetas are derived from a primitive 2n-th root of unity, this computes the NTT for the ring Z_q[x]/(x^n+1) and inv_ntt is its exact inverse. When zetas come from an n-th root (Kyber), this is the incomplete NTT.

Reference: Galbraith, “Mathematics of Public Key Cryptography”, Ch. 18.

Parameters:

f (list[int]) – polynomial coefficients (length n).
q (int) – prime modulus.
n (int) – polynomial degree (power of 2).
zetas (list[int]) – twiddle factors from build_zetas().

Returns:

NTT-domain representation (length n).

Return type:

morie.crypto.inv_ntt(f, q, n, zetas)[source]¶

Inverse Number Theoretic Transform over Z_q.

Gentleman-Sande butterfly, decimation-in-frequency. Exact inverse of ntt() when the twiddle factors come from a primitive 2n-th root.

Parameters:

f (list[int]) – NTT-domain representation (length n).
q (int) – prime modulus.
n (int) – polynomial degree (power of 2).
zetas (list[int]) – twiddle factors from build_zetas().

Returns:

polynomial coefficients (length n).

Return type:

morie.crypto.build_zetas(q, n, g)[source]¶

Build NTT twiddle factors from a primitive 2n-th root of unity g mod q.

The generator g must satisfy g^n ≡ -1 (mod q) and g^{2n} ≡ 1 (mod q). For a full NTT over Z_q[x]/(x^n+1), q must satisfy q ≡ 1 (mod 2n). The returned list contains bit-reversed powers of g used as butterfly factors in the Cooley-Tukey NTT.

For Kyber-style incomplete NTT (q ≡ 1 mod n but not mod 2n), pass an n-th root of unity instead. The NTT will produce n/2 degree-1 polynomial residues, suitable for pointwise basemul but not for coefficient-domain roundtrip.

Parameters:

q (int) – prime modulus.
n (int) – polynomial degree (power of 2).
g (int) – primitive root of unity mod q.

Returns:

list of n twiddle factors.

Return type:

morie.crypto.poly_add(a, b, q)[source]¶

Coefficient-wise polynomial addition mod q.

Parameters:

a (list[int]) – first polynomial (length n).
b (list[int]) – second polynomial (length n).
q (int) – modulus.

Returns:

(a + b) mod q, coefficient-wise.

Return type:

morie.crypto.poly_sub(a, b, q)[source]¶

Coefficient-wise polynomial subtraction mod q.

Parameters:

a (list[int]) – first polynomial (length n).
b (list[int]) – second polynomial (length n).
q (int) – modulus.

Returns:

(a - b) mod q, coefficient-wise.

Return type:

morie.crypto.poly_mul_ntt(a, b, q, zetas=None)[source]¶

Pointwise multiplication in NTT domain mod q.

Both a and b must already be in NTT representation. The zetas parameter is accepted for API uniformity but not used (NTT multiplication is pointwise in the evaluation domain).

Parameters:

a (list[int]) – first polynomial in NTT domain (length n).
b (list[int]) – second polynomial in NTT domain (length n).
q (int) – modulus.
zetas (list[int] | None) – unused, accepted for API consistency.

Returns:

pointwise product mod q (length n).

Return type:

morie.crypto.poly_ring_mul(a, b, q, mod_poly=None)[source]¶

Schoolbook polynomial multiply mod (x^n + 1) mod q.

O(n^2) but works for any q and n without NTT compatibility constraints. If mod_poly is an integer, it is treated as the degree n of the cyclotomic polynomial x^n + 1.

Reference: Hoffstein, Pipher, Silverman, “An Introduction to Mathematical Cryptography”, Ch. 7.

Parameters:

a (list[int]) – first polynomial coefficients (length <= n).
b (list[int]) – second polynomial coefficients (length <= n).
q (int) – modulus.
mod_poly (int | None) – degree n of x^n + 1 (default: len(a)).

Returns:

product polynomial mod (x^n + 1) mod q.

Return type:

morie.crypto.poly_ring_mul_xn_minus_1(a, b, q, n)[source]¶

Schoolbook polynomial multiply mod (x^n - 1) mod q.

Used by NTRU where the ring is Z_q[x]/(x^n - 1), not (x^n + 1). Wrapping is additive (convolution) rather than negacyclic.

Parameters:

a (list[int]) – first polynomial coefficients.
b (list[int]) – second polynomial coefficients.
q (int) – modulus.
n (int) – ring degree.

Returns:

product polynomial mod (x^n - 1) mod q.

Return type:

morie.crypto.find_irreducible(m)[source]¶

Find an irreducible polynomial of degree m over GF(2).

Returns the polynomial as an integer where bit k represents the coefficient of x^k. Uses a lookup table for m <= 13 (Conway polynomials) and brute-force search with Rabin’s irreducibility test for larger m.

Reference: Galbraith, “Mathematics of Public Key Cryptography”, Ch. 2.

Parameters:: m (int) – desired degree.
Returns:: irreducible polynomial as integer.
Raises:: ValueError – if no irreducible found (should not happen).
Return type:: int

morie.crypto.gf2m_add(a, b)[source]¶

Add two elements in GF(2^m).

Addition in characteristic 2 is bitwise XOR.

Parameters:

a (int) – field element as integer.
b (int) – field element as integer.

Returns:

a + b in GF(2^m).

Return type:

morie.crypto.gf2m_mul(a, b, mod_poly)[source]¶

Multiply two elements in GF(2^m) modulo an irreducible polynomial.

Carry-less multiplication followed by reduction mod the irreducible.

Reference: Galbraith, “Mathematics of Public Key Cryptography”, §2.9.

Parameters:

a (int) – field element as integer.
b (int) – field element as integer.
mod_poly (int) – irreducible polynomial (from find_irreducible).

Returns:

a * b in GF(2^m).

Return type:

morie.crypto.gf2m_inv(a, mod_poly)[source]¶

Multiplicative inverse in GF(2^m) via Fermat’s little theorem.

Computes a^{2^m - 2} mod p(x), which equals a^{-1} since the multiplicative group has order 2^m - 1.

Parameters:

a (int) – nonzero field element.
mod_poly (int) – irreducible polynomial.

Returns:

a^{-1} in GF(2^m).

Raises:

ValueError – if a is zero.

Return type:

morie.crypto.gf2m_pow(a, exp, mod_poly)[source]¶

Exponentiation in GF(2^m) via square-and-multiply.

Parameters:

a (int) – base element.
exp (int) – non-negative integer exponent.
mod_poly (int) – irreducible polynomial.

Returns:

a^exp in GF(2^m).

Return type:

morie.crypto.gf2_matrix_add(A, B)[source]¶

Matrix addition over GF(2).

Each element is 0 or 1. Addition is XOR.

Parameters:

A (list[list[int]]) – first matrix (list of rows).
B (list[list[int]]) – second matrix (list of rows).

Returns:

A + B over GF(2).

Return type:

list[list[int]]

morie.crypto.gf2_matrix_mul(A, B)[source]¶

Matrix multiplication over GF(2).

Uses XOR for addition and AND for multiplication, then reduces mod 2.

Parameters:

A (list[list[int]]) – m x n matrix (list of rows).
B (list[list[int]]) – n x p matrix (list of rows).

Returns:

m x p product matrix over GF(2).

Return type:

list[list[int]]

morie.crypto.gf2_matrix_inv(A)[source]¶

Matrix inverse over GF(2) via Gauss-Jordan elimination.

Reference: Galbraith, “Mathematics of Public Key Cryptography”, §2.3.

Parameters:: A (list[list[int]]) – n x n matrix (list of rows, entries 0 or 1).
Returns:: inverse matrix over GF(2), or None if singular.
Raises:: ValueError – if matrix is not square.
Return type:: list[list[int]] | None

morie.crypto.gram_schmidt(basis)[source]¶

Gram-Schmidt orthogonalization for a lattice basis.

Computes the orthogonal projection vectors B* and the Gram-Schmidt coefficients mu such that B[i] = B*[i] + sum_{j<i} mu[i,j] * B*[j].

Reference: Galbraith, “Mathematics of Public Key Cryptography”, Algorithm 17.1.

Parameters:: basis (marr) – n x m matrix (rows are basis vectors).
Returns:: (orthogonal_basis, mu_coefficients) both as numpy arrays.
Return type:: tuple[marr, marr]

morie.crypto.lll_reduce(basis, delta=0.75)[source]¶

LLL lattice basis reduction (Lenstra-Lenstra-Lovász, 1982).

Produces a delta-LLL-reduced basis satisfying: 1. Size-reduction: |mu[i,j]| <= 0.5 for all j < i. 2. Lovász condition: ||B*[k]||^2 >= (delta - mu[k,k-1]^2) * ||B*[k-1]||^2.

The first vector of the output has norm within 2^{(n-1)/2} of the shortest lattice vector.

Reference: Galbraith, “Mathematics of Public Key Cryptography”, Algorithm 17.2.

Parameters:

basis (marr) – n x m matrix (rows are basis vectors).
delta (float) – Lovász parameter (0.25 < delta <= 1, default 0.75).

Returns:

LLL-reduced basis as numpy array.

Return type:

marr

morie.crypto.bkz_reduce(basis, block_size=20, max_tours=10, delta=0.99)[source]¶

Simplified BKZ (Block Korkin-Zolotarev) lattice basis reduction.

Uses LLL as a subroutine applied to overlapping blocks. Real BKZ implementations use pruned enumeration for the SVP oracle; this educational version uses LLL on each block.

Reference: Galbraith, “Mathematics of Public Key Cryptography”, §17.2.

Parameters:

basis (marr) – n x m matrix (rows are basis vectors).
block_size (int) – BKZ block size (larger = better reduction, slower).
max_tours (int) – maximum number of BKZ tours.
delta (float) – LLL parameter for sub-reductions.

Returns:

BKZ-reduced basis.

Return type:

marr

morie.crypto.babai_nearest_plane(basis, target)[source]¶

Babai’s nearest plane algorithm for the Closest Vector Problem.

Produces an approximate solution to CVP by rounding Gram-Schmidt coefficients layer by layer from the top. Quality depends on the basis quality – apply LLL first for better results.

Reference: Galbraith, “Mathematics of Public Key Cryptography”, Algorithm 18.1.

Parameters:

basis (marr) – n x m matrix (rows are basis vectors).
target (marr) – target vector (length m).

Returns:

approximate closest lattice vector.

Return type:

marr

morie.crypto.svp_approx(basis, delta=0.75)[source]¶

Approximate Shortest Vector Problem via LLL reduction.

Returns the shortest vector in the LLL-reduced basis, which is within a factor of 2^{(n-1)/2} of the true shortest vector.

Parameters:

basis (marr) – n x m matrix (rows are basis vectors).
delta (float) – LLL parameter.

Returns:

approximately shortest lattice vector.

Return type:

marr

morie.crypto.lwe_sample(n=64, m=128, q=3329, sigma=3.2)[source]¶

Generate a Learning With Errors (LWE) instance.

Produces (A, b = A*s + e mod q) where s is the secret vector and e is a discrete Gaussian error vector. The LWE problem is to recover s given (A, b).

Reference: Hoffstein, Pipher, Silverman, “An Introduction to Mathematical Cryptography”, §7.8.

Parameters:

n (int) – secret dimension.
m (int) – number of samples.
q (int) – modulus.
sigma (float) – Gaussian noise standard deviation.

Returns:

dict with A, b, s, e (s and e are secret witnesses).

Return type:

morie.crypto.rlwe_sample(n=256, q=3329, sigma=3.2)[source]¶

Generate a Ring-LWE instance in Z_q[x]/(x^n+1).

The Ring-LWE problem is the ring analogue of LWE: given (a, b = a*s + e), recover the secret polynomial s. When q is NTT-friendly (q ≡ 1 mod 2n), multiplication uses NTT; otherwise falls back to schoolbook.

Reference: Hoffstein, Pipher, Silverman, “An Introduction to Mathematical Cryptography”, §7.10.

Parameters:

n (int) – polynomial degree (must be power of 2).
q (int) – prime modulus.
sigma (float) – Gaussian noise standard deviation.

Returns:

dict with a, b, s, e polynomials and parameters.

Return type:

morie.crypto.lwe_key_exchange(n=64, q=3329, sigma=3.2)[source]¶

Simulate a Diffie-Hellman-style LWE key exchange.

Alice and Bob each generate LWE instances with a shared matrix A, secrets and errors both drawn from the rounded Gaussian. Alice sends b_A = s_A^T * A + e_A, Bob sends b_B = A * s_B + e_B. Both derive approximately equal shared keys via rounding.

Reference: Hoffstein, Pipher, Silverman, “An Introduction to Mathematical Cryptography”, §7.11.

Parameters:

n (int) – dimension of the shared matrix A.
q (int) – modulus.
sigma (float) – Gaussian noise standard deviation.

Returns:

dict with alice_key, bob_key, match (bool), and intermediates.

Return type:

Reporting and output¶

Automated report generation for epidemiological analyses.

Produces publication-ready statistical reports from MORIE pipeline outputs. Supports APA-style statistical formatting, STROBE/CONSORT compliance checking, multi-format output (Markdown, LaTeX, HTML), and automated section generation for methods, results, limitations, and appendices.

The report generation system follows reporting guidelines endorsed by EQUATOR Network for observational studies (STROBE), clinical trials (CONSORT), and systematic reviews (PRISMA).

References

Vandenbroucke, J. P. et al. (2007). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and elaboration. Annals of Internal Medicine, 147(8), W163–W194. https://doi.org/10.7326/0003-4819-147-8-200710160-00010-w1

Schulz, K. F. et al. (2010). CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomized trials. BMJ, 340, c332. https://doi.org/10.1136/bmj.c332

American Psychological Association. (2020). Publication Manual of the American Psychological Association (7th ed.).

class morie.reporting.ReportSection(title, content, level=2, subsections=<factory>)[source]¶

Bases: object

A single section of a report.

Parameters:

title (str)
content (str)
level (int)
subsections (list[ReportSection])

title: str¶

content: str¶

level: int = 2¶

subsections: list[ReportSection]¶

to_markdown()[source]¶

Render section to Markdown.

Return type:: str

to_latex()[source]¶

Render section to LaTeX.

Return type:: str

to_html()[source]¶

Render section to HTML.

Return type:: str

class morie.reporting.ReportFigure(number, caption, path, label='')[source]¶

Bases: object

A figure reference in a report.

Parameters:

number (int)
caption (str)
path (str)
label (str)

number: int¶

caption: str¶

path: str¶

label: str = ''¶

to_markdown()[source]¶

Render figure reference in Markdown.

Return type:: str

to_html()[source]¶

Render figure in HTML.

Return type:: str

to_latex()[source]¶

Render figure in LaTeX.

Return type:: str

class morie.reporting.ReportTable(number, caption, df, label='')[source]¶

Bases: object

A table reference in a report.

Parameters:

number (int)
caption (str)
df (DataFrame)
label (str)

number: int¶

caption: str¶

df: DataFrame¶

label: str = ''¶

to_markdown()[source]¶

Render table in Markdown.

Return type:: str

to_latex()[source]¶

Render table in LaTeX.

Return type:: str

to_html()[source]¶

Render table in HTML.

Return type:: str

class morie.reporting.Report(title, authors, date, sections=<factory>, figures=<factory>, tables=<factory>, metadata=<factory>)[source]¶

Bases: object

A complete report with sections, figures, and tables.

Parameters:

title (str)
authors (list[str])
date (str)
sections (list[ReportSection])
figures (list[ReportFigure])
tables (list[ReportTable])
metadata (dict[str, Any])

title: str¶

authors: list[str]¶

date: str¶

sections: list[ReportSection]¶

figures: list[ReportFigure]¶

tables: list[ReportTable]¶

metadata: dict[str, Any]¶

add_section(section)[source]¶

Add a section to the report.

Parameters:: section (ReportSection)
Return type:: None

add_figure(caption, path, label='')[source]¶

Add a figure and return its reference.

Parameters:

caption (str)
path (str)
label (str)

Return type:

ReportFigure

add_table(caption, df, label='')[source]¶

Add a table and return its reference.

Parameters:

caption (str)
df (DataFrame)
label (str)

Return type:

ReportTable

class morie.reporting.STROBECheck(item_number, category, description, present, evidence='', recommendation='')[source]¶

Bases: object

A single STROBE checklist item.

Parameters:

item_number (int)
category (str)
description (str)
present (bool)
evidence (str)
recommendation (str)

item_number: int¶

category: str¶

description: str¶

present: bool¶

evidence: str = ''¶

recommendation: str = ''¶

class morie.reporting.AuditResult(file_path, checks, missing_elements, score)[source]¶

Bases: object

Result of a statistical reporting audit.

Parameters:

file_path (str)
checks (list[dict[str, Any]])
missing_elements (list[str])
score (float)

file_path: str¶

checks: list[dict[str, Any]]¶

missing_elements: list[str]¶

score: float¶

property complete: bool¶: True if all required reporting elements are present.

morie.reporting.format_p_value(p, *, threshold=0.001)[source]¶

Format a p-value in APA style.

Parameters:

p (float) – The p-value.
threshold (float) – Below this threshold, report as p < threshold.

Returns:

Formatted p-value string.

Return type:

Examples

>>> format_p_value(0.034)
'p = .034'
>>> format_p_value(0.0002)
'p < .001'
>>> format_p_value(0.5)
'p = .500'

morie.reporting.format_t_test(t, df, p, *, d=None)[source]¶

Format a t-test result in APA style.

Parameters:

t (float) – t-statistic.
df (float) – Degrees of freedom.
p (float) – p-value.
d (float | None) – Cohen’s d effect size.

Returns:

APA-formatted string, e.g. t(98) = 2.45, p = .016, d = 0.49.

Return type:

Examples

>>> format_t_test(2.45, 98, 0.016, d=0.49)
't(98) = 2.45, p = .016, d = 0.49'

morie.reporting.format_f_test(f, df1, df2, p, *, eta_sq=None, partial_eta_sq=None)[source]¶

Format an F-test result in APA style.

Parameters:

f (float) – F-statistic.
df1 (float) – Numerator degrees of freedom.
df2 (float) – Denominator degrees of freedom.
p (float) – p-value.
eta_sq (float | None) – Eta-squared effect size.
partial_eta_sq (float | None) – Partial eta-squared.

Returns:

APA-formatted F-test string.

Return type:

Examples

>>> format_f_test(4.56, 2, 97, 0.013, partial_eta_sq=0.086)
'F(2, 97) = 4.56, p = .013, partial eta-sq = .086'

morie.reporting.format_chi_square(chi2, df, p, *, n=None, cramers_v=None)[source]¶

Format a chi-square test result in APA style.

Parameters:

chi2 (float) – Chi-square statistic.
df (int) – Degrees of freedom.
p (float) – p-value.
n (int | None) – Sample size.
cramers_v (float | None) – Cramer’s V effect size.

Returns:

APA-formatted chi-square string.

Return type:

Examples

>>> format_chi_square(12.34, 3, 0.006, n=200, cramers_v=0.18)
'chi-sq(3, N = 200) = 12.34, p = .006, V = 0.18'

morie.reporting.format_correlation(r, p, *, n=None, method='Pearson')[source]¶

Format a correlation result in APA style.

Parameters:

r (float) – Correlation coefficient.
p (float) – p-value.
n (int | None) – Sample size.
method (str) – Correlation method name.

Returns:

APA-formatted correlation string.

Return type:

Examples

>>> format_correlation(0.45, 0.002, n=50)
'r(48) = .45, p = .002'

morie.reporting.format_odds_ratio(or_val, ci_lower, ci_upper, *, p=None)[source]¶

Format an odds ratio with confidence interval.

Parameters:

or_val (float) – Odds ratio.
ci_lower (float) – Lower 95% CI bound.
ci_upper (float) – Upper 95% CI bound.
p (float | None) – p-value.

Returns:

Formatted odds ratio string.

Return type:

Examples

>>> format_odds_ratio(2.15, 1.34, 3.45, p=0.001)
'OR = 2.15, 95% CI [1.34, 3.45], p = .001'

morie.reporting.format_hazard_ratio(hr, ci_lower, ci_upper, *, p=None)[source]¶

Format a hazard ratio with confidence interval.

Parameters:

hr (float) – Hazard ratio.
ci_lower (float) – Lower 95% CI bound.
ci_upper (float) – Upper 95% CI bound.
p (float | None) – p-value.

Return type:

Examples

>>> format_hazard_ratio(1.85, 1.12, 3.05, p=0.016)
'HR = 1.85, 95% CI [1.12, 3.05], p = .016'

morie.reporting.format_mean_sd(mean, sd, *, n=None)[source]¶

Format a mean and standard deviation.

Parameters:

mean (float) – Mean value.
sd (float) – Standard deviation.
n (int | None) – Sample size.

Returns:

Formatted string, e.g. M = 3.45, SD = 1.23.

Return type:

Examples

>>> format_mean_sd(3.45, 1.23, n=50)
'M = 3.45, SD = 1.23 (n = 50)'

morie.reporting.format_ci(estimate, ci_lower, ci_upper, *, level=0.95, label='')[source]¶

Format an estimate with confidence interval.

Parameters:

estimate (float) – Point estimate.
ci_lower (float) – Lower CI bound.
ci_upper (float) – Upper CI bound.
level (float) – Confidence level (default 0.95).
label (str) – Optional label for the estimate.

Return type:

Examples

>>> format_ci(0.35, 0.22, 0.48, label="ATE")
'ATE = 0.35, 95% CI [0.22, 0.48]'

morie.reporting.format_regression_coefficient(name, beta, se, p, *, ci_lower=None, ci_upper=None)[source]¶

Format a regression coefficient.

Parameters:

name (str) – Variable name.
beta (float) – Coefficient estimate.
se (float) – Standard error.
p (float) – p-value.
ci_lower (float | None) – Lower CI bound.
ci_upper (float | None) – Upper CI bound.

Return type:

Examples

>>> format_regression_coefficient("age", 0.05, 0.02, 0.012, ci_lower=0.01, ci_upper=0.09)
'age: B = 0.05, SE = 0.02, 95% CI [0.01, 0.09], p = .012'

morie.reporting.generate_introduction(*, topic, background='', objectives=None, hypotheses=None)[source]¶

Generate an introduction section.

Parameters:

topic (str) – Main research topic.
background (str) – Background text.
objectives (list[str] | None) – Study objectives.
hypotheses (list[str] | None) – Study hypotheses.

Return type:

morie.reporting.generate_methods(*, study_design='cross-sectional observational study', data_source='', sample_description='', variables=None, statistical_methods=None, software=None, alpha=0.05)[source]¶

Generate a methods section.

Automatically describes the statistical methods used based on the parameters provided.

Parameters:

study_design (str) – Study design description.
data_source (str) – Data source description.
sample_description (str) – Sample/population description.
variables (dict[str, str] | None) – Variable names mapped to descriptions (e.g. {"treatment": "cannabis use"}).
statistical_methods (list[str] | None) – Names of statistical methods used.
software (list[str] | None) – Software/packages used.
alpha (float) – Significance level.

Return type:

morie.reporting.generate_results(*, sample_size=None, descriptive_results=None, primary_results=None, secondary_results=None, sensitivity_results=None)[source]¶

Generate a results section.

Parameters:

sample_size (int | None) – Total sample size.
descriptive_results (str | None) – Text for descriptive statistics subsection.
primary_results (str | None) – Primary analysis results.
secondary_results (str | None) – Secondary analysis results.
sensitivity_results (str | None) – Sensitivity analysis results.

Return type:

morie.reporting.generate_discussion(*, key_findings=None, interpretation='', comparison_to_literature='', implications='', future_directions='')[source]¶

Generate a discussion section.

Parameters:

key_findings (list[str] | None) – Key findings to summarize.
interpretation (str) – Interpretation of results.
comparison_to_literature (str) – Comparison to existing literature.
implications (str) – Implications for practice or policy.
future_directions (str) – Future research directions.

Return type:

morie.reporting.generate_limitations(*, study_design='cross-sectional', custom_limitations=None)[source]¶

Auto-generate a limitations section based on study design.

Identifies common limitations for the given study design and appends any custom limitations provided.

Parameters:

study_design (str) – Study design type.
custom_limitations (list[str] | None) – Additional limitations.

Return type:

morie.reporting.generate_executive_summary(*, background, methods_summary, key_findings, conclusions)[source]¶

Generate an executive summary.

Parameters:

background (str) – Brief background.
methods_summary (str) – Brief methods description.
key_findings (list[str]) – Bullet points of key findings.
conclusions (str) – Main conclusions.

Return type:

morie.reporting.generate_appendix(*, supplementary_tables=None, sensitivity_analyses=None, additional_methods=None)[source]¶

Generate an appendix section.

Parameters:

supplementary_tables (list[tuple[str, pd.DataFrame]] | None) – List of (caption, DataFrame) tuples.
sensitivity_analyses (list[tuple[str, str]] | None) – List of (title, content) tuples.
additional_methods (str | None) – Additional methodological details.

Return type:

morie.reporting.generate_consort_narrative(*, assessed_for_eligibility, excluded, exclusion_reasons=None, randomized=None, allocated_treatment=None, allocated_control=None, lost_to_followup_treatment=0, lost_to_followup_control=0, analyzed_treatment=None, analyzed_control=None)[source]¶

Generate a CONSORT-style flow diagram narrative.

Parameters:

assessed_for_eligibility (int) – Number assessed for eligibility.
excluded (int) – Number excluded.
exclusion_reasons (dict[str, int] | None) – Reasons for exclusion with counts.
randomized (int | None) – Number randomized (or included for observational).
allocated_treatment (int | None) – Number in treatment group.
allocated_control (int | None) – Number in control group.
lost_to_followup_treatment (int) – Lost to follow-up in treatment.
lost_to_followup_control (int) – Lost to follow-up in control.
analyzed_treatment (int | None) – Number analyzed in treatment.
analyzed_control (int | None) – Number analyzed in control.

Return type:

morie.reporting.check_strobe_compliance(report_text)[source]¶

Check a report for STROBE checklist compliance.

Uses keyword matching to detect the presence of required reporting elements. This is a heuristic check, not a substitute for expert review.

Parameters:: report_text (str) – Full report text.
Returns:: One check per STROBE item.
Return type:: list[STROBECheck]

morie.reporting.audit_statistical_reporting(path)[source]¶

Audit a CSV of statistical results for completeness.

Checks whether required reporting elements are present: point estimates, standard errors, confidence intervals, p-values, sample sizes, and effect sizes.

Parameters:: path (str | Path) – Path to a CSV file with statistical results.
Return type:: AuditResult

morie.reporting.generate_reproducibility_statement(*, software=None, seed=None, data_available=False, code_available=True, repository_url='')[source]¶

Generate a reproducibility statement.

Parameters:

software (list[str] | None) – Software and versions used.
seed (int | None) – Random seed used.
data_available (bool) – Whether data are publicly available.
code_available (bool) – Whether analysis code is available.
repository_url (str) – URL to code repository.

Return type:

morie.reporting.generate_data_availability_statement(*, data_public=False, repository='', access_conditions='', contact='')[source]¶

Generate a data availability statement.

Parameters:

data_public (bool) – Whether data are publicly available.
repository (str) – Data repository URL.
access_conditions (str) – Conditions for data access.
contact (str) – Contact for data requests.

Return type:

morie.reporting.generate_conflict_of_interest()[source]¶

Generate a conflict of interest statement template.

Return type:: str

morie.reporting.generate_funding_acknowledgment(*, funders=None)[source]¶

Generate a funding acknowledgment section.

Parameters:: funders (list[dict[str, str]] | None) – List of dicts with keys name, grant_number.
Return type:: str

morie.reporting.generate_author_contributions(contributions)[source]¶

Generate an author contribution statement using CRediT taxonomy.

Parameters:: contributions (dict[str, list[str]]) – Mapping of author names to CRediT roles. Valid roles: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing - original draft, Writing - review & editing, Visualization, Supervision, Project administration, Funding acquisition.
Return type:: str

Examples

>>> print(generate_author_contributions({
...     "Smith, J.": ["Conceptualization", "Methodology", "Writing - original draft"],
...     "Doe, A.": ["Formal analysis", "Software", "Writing - review & editing"],
... }))
**Author Contributions**:

Smith, J.: Conceptualization, Methodology, Writing - original draft.
Doe, A.: Formal analysis, Software, Writing - review & editing.

morie.reporting.compile_report(report, *, output_format='markdown')[source]¶

Compile a Report object into a single document string.

Parameters:

report (Report) – Report to compile.
output_format (str) – markdown, latex, or html.

Returns:

Compiled report text.

Return type:

morie.reporting.save_report(report, path, *, output_format=None)[source]¶

Compile and save a report to a file.

Parameters:

report (Report) – Report to save.
path (str | Path) – Output file path.
output_format (str | None) – If None, inferred from file extension.

Returns:

Path to the saved file.

Return type:

Path

morie.reporting.compare_reports(path_a, path_b)[source]¶

Compare two report versions.

Parameters:

path_a (str | Path) – First report.
path_b (str | Path) – Second report.

Returns:

Comparison with keys lines_added, lines_removed, sections_changed.

Return type:

dict[str, Any]

morie.reporting.generate_full_report(results_dir, *, title='MORIE Analysis Report', authors=None, study_design='cross-sectional observational study', output_path=None, output_format='markdown')[source]¶

Generate a complete report from pipeline output CSVs.

Scans a results directory, detects statistical outputs, formats them into report sections, and compiles the full document.

Parameters:

results_dir (str | Path) – Directory containing CSV outputs from MORIE modules.
title (str) – Report title.
authors (list[str] | None) – Author names.
study_design (str) – Study design for methods section.
output_path (str | Path | None) – If provided, save the report to this path.
output_format (str) – Output format (markdown, latex, html).

Returns:

The compiled report object.

Return type:

Report

morie.reporting.render_strobe_checklist(checks)[source]¶

Display STROBE compliance results with rich formatting.

Parameters:: checks (list[STROBECheck]) – STROBE check results.
Return type:: None

morie.reporting.render_audit_result(result)[source]¶

Display statistical reporting audit with rich formatting.

Parameters:: result (AuditResult) – Audit result to display.
Return type:: None

Publication-ready table generation for epidemiological analysis.

Generates Table 1 (baseline characteristics), regression tables, odds ratio tables, hazard ratio tables, correlation matrices, model comparison tables, and ANOVA tables. Supports output in multiple formats: pandas DataFrame, LaTeX (booktabs), HTML, Markdown, plain text, and CSV.

class morie.tables_pub.FootnoteRegistry(_notes=<factory>, _symbols='abcdefghijklmnopqrstuvwxyz')[source]¶

Bases: object

Accumulates footnotes and renders them in order.

Parameters:

_notes (list[str])
_symbols (str)

add(text)[source]¶

Add a footnote, return the symbol.

Parameters:: text (str)
Return type:: str

render(fmt='text')[source]¶

Render all footnotes.

Parameters:: fmt (Literal['dataframe', 'latex', 'html', 'markdown', 'text', 'csv'])
Return type:: str

morie.tables_pub.table1(data, group_col=None, continuous_vars=None, categorical_vars=None, *, continuous_summary='mean_sd', show_p=True, show_smd=True, show_missing=True, weights=None, digits=2, apa=False, output_format='dataframe', title='Table 1. Baseline Characteristics')[source]¶

Generate a Table 1 of baseline characteristics stratified by group.

Parameters:

data (DataFrame) – Input data with all variables.
group_col (str or None) – Column name defining groups. If None, a single-column summary is produced.
continuous_vars (list of str or None) – Continuous variables. If None, auto-detected as numeric non-group columns.
categorical_vars (list of str or None) – Categorical variables. If None, auto-detected as object/ category columns.
continuous_summary (str) – Summary format for continuous variables: "mean_sd", "median_iqr", or "mean_ci".
show_p (bool) – Include p-value column.
show_smd (bool) – Include standardised mean difference column (requires exactly two groups).
show_missing (bool) – Include a missing count column.
weights (str or None) – Column name for survey weights.
digits (int) – Number of decimal places.
apa (bool) – Use APA-style formatting (leading-zero suppression for p-values).
output_format (FormatTarget) – Output format.
title (str) – Table title.

Returns:

Table in the requested format.

Return type:

DataFrame or str

References

Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399-424. https://doi.org/10.1080/00273171.2011.568786

morie.tables_pub.regression_table(models, *, exponentiate=False, show_ci=True, show_stars=True, confidence=0.95, digits=3, model_stats=None, apa=False, output_format='dataframe', title='Regression Results', spanning_headers=None)[source]¶

Multi-model regression table (side-by-side).

Accepts fitted statsmodels results objects. Supports linear, logistic, Poisson, and other GLM models.

Parameters:

models (dict) – {model_name: fitted_model} where fitted_model is a statsmodels results object with params, bse, pvalues, conf_int(), and model summary attributes.
exponentiate (bool) – Exponentiate coefficients (e.g. for odds ratios or hazard ratios).
show_ci (bool) – Include confidence intervals.
show_stars (bool) – Include significance stars next to coefficients.
confidence (float) – Confidence level for intervals.
digits (int) – Decimal places.
model_stats (list of str or None) – Model-level statistics to include. Defaults to ["nobs", "rsquared", "aic", "bic", "llf"].
apa (bool) – APA formatting.
output_format (FormatTarget)
title (str)
spanning_headers (dict or None) – Grouped column headers {span_label: [model_names]}.

Return type:

DataFrame or str

Notes

Each model column shows: coefficient (with stars), SE in parentheses, and optionally the CI on a separate line.

morie.tables_pub.odds_ratio_table(model, *, confidence=0.95, digits=3, apa=False, output_format='dataframe', title='Odds Ratios')[source]¶

Odds ratio table from a fitted logistic regression model.

Parameters:

model (statsmodels results) – A fitted logistic regression (Logit or GLM with binomial family).
confidence (float)
digits (int)
apa (bool)
output_format (FormatTarget)
title (str)

Return type:

DataFrame or str

morie.tables_pub.hazard_ratio_table(params, se, pvalues, *, confidence=0.95, digits=3, apa=False, output_format='dataframe', title='Hazard Ratios')[source]¶

Hazard ratio table from Cox model parameters.

Parameters:

params (Series) – Log-hazard ratio coefficients (beta).
se (Series) – Standard errors of coefficients.
pvalues (Series) – P-values.
confidence (float)
digits (int)
apa (bool)
output_format (FormatTarget)
title (str)

Return type:

DataFrame or str

morie.tables_pub.correlation_table(data, *, method='pearson', show_stars=True, mask_diagonal=True, digits=3, output_format='dataframe', title='Correlation Matrix')[source]¶

Correlation matrix with significance stars.

Parameters:

data (DataFrame) – Numeric columns to correlate.
method (str) – "pearson", "spearman", or "kendall".
show_stars (bool) – Annotate with significance stars.
mask_diagonal (bool) – Replace diagonal with "-".
digits (int)
output_format (FormatTarget)
title (str)

Return type:

DataFrame or str

morie.tables_pub.model_comparison_table(models, *, nested=False, digits=3, output_format='dataframe', title='Model Comparison')[source]¶

Compare multiple fitted models on key fit statistics.

Parameters:

models (dict) – {model_name: fitted_model}. Models should be statsmodels result objects.
nested (bool) – If True, perform likelihood-ratio tests between successive models (assumes they are nested).
digits (int)
output_format (FormatTarget)
title (str)

Return type:

DataFrame or str

morie.tables_pub.anova_table(model, *, typ=2, digits=3, output_format='dataframe', title='ANOVA Table')[source]¶

ANOVA table from a fitted OLS model.

Parameters:

model (statsmodels OLS results) – A fitted OLS model.
typ (int) – ANOVA type (1, 2, or 3).
digits (int)
output_format (FormatTarget)
title (str)

Return type:

DataFrame or str

morie.tables_pub.format_number(x, *, style='fixed', digits=2, apa=False)[source]¶

Format a single number according to style conventions.

Parameters:

x (float) – The number to format.
style (str) – "fixed", "scientific", "percent", or "integer".
digits (int) – Significant digits or decimal places.
apa (bool) – APA-style (suppress leading zeros for correlations/p-values).

Return type:

morie.tables_pub.format_dataframe(df, *, numeric_fmt='.2f', pval_cols=None, bold_cols=None, output_format='dataframe', title='')[source]¶

Apply uniform formatting to a DataFrame for publication.

Parameters:

df (DataFrame)
numeric_fmt (str) – Format string for numeric columns.
pval_cols (list of str or None) – Columns to format as p-values.
bold_cols (list of str or None) – Columns where values meeting significance should be bolded (HTML/LaTeX).
output_format (FormatTarget)
title (str)

Return type:

DataFrame or str

morie.tables_pub.summary_statistics_table(data, variables=None, *, stats=None, digits=2, output_format='dataframe', title='Summary Statistics')[source]¶

Descriptive statistics table for a set of variables.

Parameters:

data (DataFrame)
variables (list of str or None) – Variables to summarise. Defaults to all numeric columns.
stats (list of str or None) – Statistics to compute. Defaults to ["n", "mean", "sd", "median", "min", "max", "missing"].
digits (int)
output_format (FormatTarget)
title (str)

Return type:

DataFrame or str

morie.tables_pub.treatment_effect_table(estimators, *, digits=3, output_format='dataframe', title='Treatment Effect Estimates')[source]¶

Summary table of causal effect estimates from multiple estimators.

Parameters:

estimators (dict) – {estimator_name: {"estimate": float, "se": float, "ci_lower": float, "ci_upper": float, "p_value": float}}.
digits (int)
output_format (FormatTarget)
title (str)

Return type:

DataFrame or str

Multi-format export engine for epidemiological analysis results.

Supports LaTeX, HTML, Markdown, Word (DOCX), Excel, figure export, bundled results archives, analysis reports, reporting checklists (CONSORT, STROBE), citation generation, and reproducibility manifests.

morie.export.df_to_latex(df, *, caption='', label='', booktabs=True, longtable=False, sideways=False, column_format=None, escape=True, bold_header=True, footnotes=None, float_format='%.3f')[source]¶

Export a DataFrame to a LaTeX table string.

Parameters:

df (DataFrame) – Data to export.
caption (str) – Table caption.
label (str) – LaTeX label for cross-referencing.
booktabs (bool) – Use booktabs rules (\toprule, \midrule, \bottomrule).
longtable (bool) – Use the longtable environment for multi-page tables.
sideways (bool) – Wrap in sidewaystable for landscape orientation.
column_format (str or None) – Column alignment string (e.g. "lccc"). Auto-generated if None.
escape (bool) – Escape special LaTeX characters in cell values.
bold_header (bool) – Bold column headers.
footnotes (list of str or None) – Footnotes to append below the table.
float_format (str) – Format string for floats.

Returns:

Complete LaTeX table source.

Return type:

morie.export.df_to_html(df, *, title='', css_classes=None, responsive=True, striped=True, hover=True, border=0, float_format='%.3f')[source]¶

Export a DataFrame to an HTML table with styling.

Parameters:

df (DataFrame)
title (str) – Table caption / title.
css_classes (list of str or None) – CSS classes to apply to the <table> element.
responsive (bool) – Wrap in a responsive <div>.
striped (bool) – Apply striped-row styling.
hover (bool) – Apply hover-row styling.
border (int) – HTML border attribute.
float_format (str) – Float formatting.

Returns:

Complete HTML table source.

Return type:

morie.export.df_to_markdown(df, *, title='', float_format='%.3f', index=True)[source]¶

Export a DataFrame to GitHub-Flavored Markdown.

Parameters:

df (DataFrame)
title (str)
float_format (str)
index (bool)

Return type:

morie.export.df_to_docx(df, path, *, title='', style='Table Grid')[source]¶

Export a DataFrame to a Word DOCX table.

Requires python-docx. This is an optional dependency; an ImportError is raised with a helpful message if not installed.

Parameters:

df (DataFrame)
path (str) – Output file path.
title (str) – Document / table title.
style (str) – Word table style name.

Returns:

Path to the saved file.

Return type:

Raises:

ImportError – If python-docx is not installed.

morie.export.df_to_excel(df, path, *, sheet_name='Results', title='', bold_header=True, alternating_rows=True, number_format='0.000', freeze_panes=(1, 0))[source]¶

Export a DataFrame to a formatted Excel file.

Parameters:

df (DataFrame)
path (str) – Output .xlsx file path.
sheet_name (str)
title (str) – Optional title written above the table.
bold_header (bool)
alternating_rows (bool)
number_format (str)
freeze_panes (tuple or None) – Row/column to freeze (default: freeze header row).

Returns:

Path to the saved file.

Return type:

morie.export.export_figure(fig, base_path, *, formats=None, dpi=300, transparent=False, close=True)[source]¶

Save a matplotlib figure to multiple formats simultaneously.

Parameters:

fig (Figure)
base_path (str) – Base filename without extension (e.g. "output/figure1").
formats (list of str or None) – File extensions. Default: ["pdf", "png", "svg"].
dpi (int) – Resolution for raster formats.
transparent (bool) – Transparent background.
close (bool) – Close the figure after saving.

Returns:

Paths to all saved files.

Return type:

list of str

morie.export.export_results_bundle(tables, figures=None, metadata=None, *, output_path='results_bundle.zip', table_formats=None, figure_formats=None, dpi=300)[source]¶

Export all tables, figures, and metadata into a single ZIP archive.

Parameters:

tables (dict) – {name: DataFrame} mapping.
figures (dict or None) – {name: Figure} mapping.
metadata (dict or None) – Metadata dictionary (serialised as JSON).
output_path (str) – Output zip file path.
table_formats (list of str or None) – Formats for tables (default: ["csv", "latex"]).
figure_formats (list of str or None) – Formats for figures (default: ["pdf", "png"]).
dpi (int)

Returns:

Path to the ZIP archive.

Return type:

morie.export.generate_report(title, sections, *, output_path='report.md', author='', date=None)[source]¶

Generate a Markdown analysis report from sections.

Parameters:

title (str) – Report title.
sections (list of dict) – Each dict has keys: "heading" (str), "text" (str), and optionally "table" (DataFrame) and "figure" (path to image).
output_path (str) – Output file path.
author (str)
date (str or None) – Date string. Defaults to today.

Returns:

Path to the saved report.

Return type:

class morie.export.ConsortNode(label, n, excluded=0, exclusion_reasons=None)[source]¶

Bases: object

A node in a CONSORT flow diagram.

Parameters:

label (str)
n (int)
excluded (int)
exclusion_reasons (list[tuple[str, int]] | None)

label: str¶

n: int¶

excluded: int = 0¶

exclusion_reasons: list[tuple[str, int]] | None = None¶

class morie.export.ConsortDiagram(assessed, randomised, arms)[source]¶

Bases: object

CONSORT-style flow diagram data.

Parameters:

assessed (ConsortNode)
randomised (ConsortNode)
arms (list[dict[str, ConsortNode]])

assessed: ConsortNode¶

randomised: ConsortNode¶

arms: list[dict[str, ConsortNode]]¶

to_dict()[source]¶

Serialise to a dictionary.

Return type:: dict[str, Any]

to_text()[source]¶

Render as a text-based flow diagram.

Return type:: str

morie.export.create_consort_diagram(assessed_n, excluded_n, randomised_n, arms, *, exclusion_reasons=None)[source]¶

Create a CONSORT flow diagram data structure.

Parameters:

assessed_n (int) – Number assessed for eligibility.
excluded_n (int) – Number excluded.
randomised_n (int) – Number randomised.
arms (list of dict) – Each dict: {"label": str, "allocated": int, "analysed": int, "lost": int, ...}.
exclusion_reasons (list of (reason, count) or None)

Return type:

ConsortDiagram

morie.export.strobe_checklist(study_type='cross-sectional', items_addressed=None)[source]¶

Generate a STROBE checklist for observational studies.

Parameters:

study_type (str) – "cross-sectional", "cohort", or "case-control".
items_addressed (dict or None) – {item_number: True/False} indicating which items have been addressed.

Returns:

Checklist with item numbers, descriptions, and status.

Return type:

DataFrame

References

von Elm, E., et al. (2007). The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement. The Lancet, 370(9596), 1453-1457. https://doi.org/10.1016/S0140-6736(07)61602-X

morie.export.generate_citation(title, authors, year, *, software_version='0.1.0', doi='', url='https://github.com/rootcoder007/morie', output_format='bibtex')[source]¶

Generate a citation entry for the analysis or the MORIE package.

Parameters:

title (str)
authors (list of str) – Author names.
year (int)
software_version (str)
doi (str)
url (str)
output_format (str) – "bibtex", "apa", or "ris".

Return type:

morie.export.create_reproducibility_manifest(data=None, *, parameters=None, seeds=None, output_path=None)[source]¶

Create a reproducibility manifest capturing environment and parameters.

Parameters:

data (DataFrame or None) – If provided, a SHA-256 checksum is computed.
parameters (dict or None) – Analysis parameters.
seeds (dict or None) – Random seeds.
output_path (str or None) – If provided, save manifest as JSON.

Return type:

morie.export.df_to_json(df, path=None, *, orient='records', date_format='iso')[source]¶

Export a DataFrame to JSON for web visualisation.

Parameters:

df (DataFrame)
path (str or None) – If provided, save to file.
orient (str) – JSON orientation (default "records").
date_format (str)

Returns:

JSON string.

Return type:

morie.export.df_to_csv_with_meta(df, path, *, description='', source='', variables=None)[source]¶

Export a DataFrame to CSV with a .csvmeta sidecar file.

Parameters:

df (DataFrame)
path (str) – Output CSV path.
description (str) – Dataset description.
source (str) – Data source.
variables (dict or None) – {column_name: description} variable dictionary.

Return type:

tuple of (csv_path, meta_path)

morie.export.to_clipboard(df, *, fmt='text', sep='\t')[source]¶

Copy a DataFrame to the system clipboard for pasting.

Parameters:

df (DataFrame)
fmt (str) – "text" (tab-separated) or "html".
sep (str) – Separator for text format.

Return type:

None

morie.export.latex_preamble(*, booktabs=True, longtable=False, sideways=False, amsmath=True, hyperref=True, natbib=True, threeparttable=True)[source]¶

Generate a LaTeX preamble with required packages for MORIE tables.

Parameters:

booktabs (bool)
longtable (bool)
sideways (bool)
amsmath (bool)
hyperref (bool)
natbib (bool)
threeparttable (bool)

Return type:

morie.export.batch_export(results, output_dir, *, table_formats=None, figure_formats=None, dpi=300, manifest=True)[source]¶

Batch export all tables and figures from a pipeline run.

Parameters:

results (dict) – {name: DataFrame_or_Figure} mapping.
output_dir (str) – Output directory.
table_formats (list of str or None) – Default: ["csv", "latex"].
figure_formats (list of str or None) – Default: ["pdf", "png"].
dpi (int)
manifest (bool) – Write a JSON manifest of exported files.

Returns:

Paths to all exported files.

Return type:

list of str

morie.explain — human-readable descriptions of module-output CSVs.

Backs morie explain <filename> and morie cheatsheet.

The explanations here are written for someone who has just run morie run-module <name> for the first time and is staring at a folder of 10-15 CSVs not knowing where to start. They describe:

what question the file answers,

how to read each column,

which row / cell to look at first.

Adding a new module’s output files means adding entries here.

morie.explain.describe(filename)[source]¶

Return the human-readable description of an output CSV, or a fallback.

Parameters:: filename (str)
Return type:: str

morie.explain.print_cheatsheet()[source]¶

Print the morie explain cheatsheet: the output-file glossary in brief.

Return type:: None

Publication-ready statistical visualizations for epidemiological analysis.

Provides forest plots, funnel plots, survival curves, ROC curves, DAGs, diagnostic plots, and many more – all with a consistent theme system suitable for journal submission.

morie.viz.set_theme(name='publication')[source]¶

Apply a predefined MORIE matplotlib theme globally.

Parameters:: name (str) – One of "publication", "presentation", or "minimal".
Raises:: ValueError – If name is not a recognised theme.
Return type:: None

morie.viz.get_theme(name='publication')[source]¶

Return a copy of the named theme dictionary.

Parameters:: name (str)
Return type:: dict[str, Any]

morie.viz.forest_plot(labels, estimates, ci_lower, ci_upper, *, null_value=0.0, xlabel='Effect size', title='', color='#2c7bb6', diamond_summary=None, weights=None, theme='publication', ax=None)[source]¶

Forest plot for meta-analysis, subgroup effects, or regression coefficients.

Parameters:

labels (sequence of str) – Row labels (study names, subgroups, or variable names).
estimates (sequence of float) – Point estimates.
ci_lower (sequence of float) – Lower and upper bounds of the confidence interval.
ci_upper (sequence of float) – Lower and upper bounds of the confidence interval.
null_value (float) – The null/reference value; a vertical dashed line is drawn here.
xlabel (str) – Label for the x-axis (e.g. "Odds Ratio" or "Beta").
title (str) – Plot title.
color (str) – Colour for the point estimates and CI lines.
diamond_summary (tuple of (estimate, lower, upper) or None) – If provided, a diamond summarising the pooled estimate is drawn at the bottom.
weights (sequence of float or None) – Optional study weights used to scale marker sizes.
theme (str or None) – MORIE theme name. Pass None to use current rc params.
ax (matplotlib Axes or None) – Axes to draw on. A new figure is created if None.

Return type:

matplotlib.figure.Figure

References

Lewis, S., & Clarke, M. (2001). Forest plots: trying to see the wood and the trees. BMJ, 322(7300), 1479-1480. https://doi.org/10.1136/bmj.322.7300.1479

morie.viz.funnel_plot(effects, se, *, null_value=0.0, egger_line=True, fill_contours=True, xlabel='Effect size', title='Funnel Plot', theme='publication', ax=None)[source]¶

Funnel plot for publication-bias assessment in meta-analysis.

Parameters:

effects (sequence of float) – Study-level effect sizes.
se (sequence of float) – Standard errors of the effect sizes.
null_value (float) – Expected effect under the null hypothesis.
egger_line (bool) – If True, overlay the Egger’s regression test line.
fill_contours (bool) – If True, shade pseudo-95% confidence contours.
xlabel (str) – Axis label and title.
title (str) – Axis label and title.
theme (str or None) – MORIE theme name.
ax (matplotlib Axes or None)

Return type:

matplotlib.figure.Figure

References

Sterne, J. A. C., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis. Journal of Clinical Epidemiology, 54(10), 1046-1055. https://doi.org/10.1016/S0895-4356(01)00377-8

morie.viz.kaplan_meier_plot(time, event, *, group=None, group_labels=None, confidence=0.95, show_ci=True, show_censoring=True, show_risk_table=True, xlabel='Time', ylabel='Survival Probability', title='', colors=None, theme='publication', ax=None)[source]¶

Kaplan-Meier survival curves with confidence bands, censoring marks, and optional risk table.

Parameters:

time (array-like) – Observed times (follow-up duration or event time).
event (array-like) – Event indicator (1 = event, 0 = censored).
group (array-like or None) – Grouping variable for stratified curves.
group_labels (dict or None) – Mapping from group values to display labels.
confidence (float) – Confidence level for the bands (default 0.95).
show_ci (bool) – Whether to display confidence bands.
show_censoring (bool) – Whether to mark censored observations with ticks.
show_risk_table (bool) – Whether to display the number-at-risk table below the curve.
xlabel (str) – Axis labels and title.
ylabel (str) – Axis labels and title.
title (str) – Axis labels and title.
colors (list of str or None) – Colours per group. Uses default colour cycle if None.
theme (str or None) – MORIE theme name.
ax (matplotlib Axes or None)

Return type:

matplotlib.figure.Figure

References

Kaplan, E. L., & Meier, P. (1958). Nonparametric estimation from incomplete observations. Journal of the American Statistical Association, 53(282), 457-481. https://doi.org/10.1080/01621459.1958.10501452

morie.viz.roc_curve_plot(y_true, y_score, *, show_optimal=True, show_ci=True, n_bootstrap=1000, confidence=0.95, label='Model', title='ROC Curve', theme='publication', ax=None)[source]¶

ROC curve with AUC, optimal cutoff (Youden’s J), and bootstrap CI.

Parameters:

y_true (array-like) – True binary labels.
y_score (array-like) – Predicted probabilities or decision scores.
show_optimal (bool) – Mark the optimal threshold (max Youden’s J).
show_ci (bool) – Display bootstrap confidence band for the ROC.
n_bootstrap (int) – Number of bootstrap replicates for the CI.
confidence (float) – Confidence level.
label (str) – Curve label in the legend.
title (str) – Plot title.
theme (str or None) – MORIE theme name.
ax (matplotlib Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.precision_recall_plot(y_true, y_score, *, label='Model', title='Precision-Recall Curve', theme='publication', ax=None)[source]¶

Precision-recall curve with average precision (AP).

Parameters:

y_true (array-like) – True binary labels.
y_score (array-like) – Predicted probabilities.
label (str) – Legend label and title.
title (str) – Legend label and title.
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.calibration_plot(y_true, y_pred, *, n_bins=10, strategy='quantile', show_histogram=True, title='Calibration Plot', theme='publication', ax=None)[source]¶

Calibration plot (observed vs predicted probability).

Parameters:

y_true (array-like) – True binary labels.
y_pred (array-like) – Predicted probabilities.
n_bins (int) – Number of bins.
strategy (str) – "quantile" or "uniform".
show_histogram (bool) – Show a histogram of predicted probabilities beneath the calibration curve.
title (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

References

Van Calster, B., et al. (2019). Calibration: the Achilles heel of predictive analytics. BMC Medicine, 17(1), 230. https://doi.org/10.1186/s12916-019-1466-7

morie.viz.balance_plot(smd_before, smd_after=None, *, threshold=0.1, title='Covariate Balance', theme='publication', ax=None)[source]¶

Love plot showing standardised mean differences (SMD) before and after matching/weighting.

Parameters:

smd_before (dict or Series) – Absolute SMD for each covariate before adjustment.
smd_after (dict, Series, or None) – Absolute SMD after adjustment. If None, only pre-adjustment values are shown.
threshold (float) – SMD threshold line (commonly 0.1).
title (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

References

Austin, P. C. (2009). Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Statistics in Medicine, 28(25), 3083-3107. https://doi.org/10.1002/sim.3697

morie.viz.dag_plot(edges, *, node_positions=None, highlight_path=None, title='', node_color='#2c7bb6', highlight_color='#d7191c', theme='publication', ax=None)[source]¶

Directed acyclic graph visualisation using matplotlib (no graphviz).

Parameters:

edges (list of (str, str)) – Directed edges as (source, target) tuples.
node_positions (dict or None) – {node_name: (x, y)} positions. If None, a simple topological layout is computed.
highlight_path (list of str or None) – A causal path to highlight in a different colour.
title (str)
node_color (str)
highlight_color (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.qq_plot(data, *, dist='norm', dist_params=(), title='Q-Q Plot', theme='publication', ax=None)[source]¶

Quantile-quantile plot against a theoretical distribution.

Parameters:

data (array-like) – Observed data.
dist (str) – Name of a scipy.stats distribution (default "norm").
dist_params (tuple) – Additional shape parameters for the distribution (loc and scale are estimated from data).
title (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.residual_diagnostic_plots(fitted, residuals, *, leverage=None, cooks_d=None, title='Residual Diagnostics', theme='publication')[source]¶

Four-panel residual diagnostic plots: residuals vs fitted, scale-location, Q-Q of residuals, and residuals vs leverage.

Parameters:

fitted (array-like) – Fitted (predicted) values.
residuals (array-like) – Model residuals.
leverage (array-like or None) – Hat-matrix diagonal values (leverage). If None, the leverage panel shows a histogram of residuals instead.
cooks_d (array-like or None) – Cook’s distance. If provided, points exceeding 4/n are highlighted.
title (str)
theme (str or None)

Return type:

matplotlib.figure.Figure

morie.viz.influence_plot(leverage, cooks_d, residuals, *, labels=None, title='Influence Plot', theme='publication', ax=None)[source]¶

Influence plot: Cook’s distance vs leverage, sized by residuals.

Parameters:

leverage (array-like) – Hat-matrix diagonal values.
cooks_d (array-like) – Cook’s distance.
residuals (array-like) – Model residuals (used to size markers).
labels (sequence of str or None) – Observation labels for the most influential points.
title (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.added_variable_plot(data, outcome, focus_var, covariates, *, title=None, theme='publication', ax=None)[source]¶

Added variable plot (partial regression plot) for a single covariate.

Residualises both outcome and focus_var on the remaining covariates and scatterplots the residuals.

Parameters:

data (DataFrame) – Input data.
outcome (str) – Outcome variable name.
focus_var (str) – The covariate to examine.
covariates (list of str) – All covariates including focus_var.
title (str or None)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.component_residual_plot(data, outcome, focus_var, covariates, *, title=None, theme='publication', ax=None)[source]¶

Component-plus-residual (partial residual) plot.

Plots beta_j * x_j + residuals against x_j to detect non-linearity in the effect of focus_var.

Parameters:

data (DataFrame)
outcome (str)
focus_var (str)
covariates (list of str)
title (str or None)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.bland_altman_plot(method1, method2, *, confidence=0.95, title='Bland-Altman Plot', xlabel='Mean of Methods', ylabel='Difference (Method 1 - Method 2)', theme='publication', ax=None)[source]¶

Bland-Altman plot for method agreement assessment.

Parameters:

method1 (array-like) – Measurements from two methods on the same subjects.
method2 (array-like) – Measurements from two methods on the same subjects.
confidence (float) – Confidence level for limits of agreement (default 0.95).
title (str)
xlabel (str)
ylabel (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

References

Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307-310. https://doi.org/10.1016/S0140-6736(86)90837-8

morie.viz.correlation_heatmap(data, *, method='pearson', show_stars=True, mask_upper=True, cmap='RdBu_r', title='Correlation Matrix', fmt='.2f', theme='publication', ax=None)[source]¶

Correlation heatmap with optional significance stars.

Parameters:

data (DataFrame) – Numeric columns to correlate.
method (str) – "pearson", "spearman", or "kendall".
show_stars (bool) – Annotate cells with significance stars (* p<0.05, ** p<0.01, *** p<0.001).
mask_upper (bool) – Mask the upper triangle.
cmap (str) – Matplotlib colourmap.
title (str)
fmt (str) – Number format for annotation.
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.missing_pattern_plot(data, *, max_cols=50, title='Missing Data Pattern', theme='publication', ax=None)[source]¶

Heatmap of missing data patterns (rows = observations, cols = variables).

Parameters:

data (DataFrame)
max_cols (int) – Maximum number of columns to display (sorted by missingness).
title (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.distribution_comparison_plot(groups, *, kind='violin', title='', xlabel='Value', colors=None, theme='publication', ax=None)[source]¶

Compare distributions across groups using violin, histogram, or ridge plots.

Parameters:

groups (dict) – {group_label: values} mapping.
kind (str) – "violin", "histogram", or "ridge".
title (str)
xlabel (str)
colors (list of str or None)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.effect_size_plot(labels, effects, ci_lower, ci_upper, *, colors=None, title='Effect Size Comparison', xlabel='Effect Size', theme='publication', ax=None)[source]¶

Horizontal bar chart of effect sizes with confidence intervals.

Parameters:

labels (sequence of str)
effects (sequence of float)
ci_lower (sequence of float)
ci_upper (sequence of float)
colors (list of str or None)
title (str)
xlabel (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.power_curve_plot(sample_sizes, power_values=None, *, effect_size=0.5, alpha=0.05, test='two_sample_t', target_power=0.8, title='Statistical Power Curve', theme='publication', ax=None)[source]¶

Power curve showing statistical power as a function of sample size.

Parameters:

sample_sizes (sequence of int) – Sample sizes to evaluate.
power_values (sequence of float or None) – Pre-computed power values. If None, computed for a two-sample t-test using a normal approximation.
effect_size (float) – Cohen’s d or equivalent (used only when power_values is None).
alpha (float) – Significance level.
test (str) – Currently "two_sample_t" is supported for auto-computation.
target_power (float) – Target power level (horizontal line).
title (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.propensity_score_plot(ps, treatment, *, title='Propensity Score Distribution', theme='publication', ax=None)[source]¶

Overlaid histograms of propensity scores for treated and control groups.

Parameters:

ps (array-like) – Propensity scores.
treatment (array-like) – Binary treatment indicator.
title (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.cate_heterogeneity_plot(cate_values, *, covariate=None, covariate_name='Covariate', title='CATE Heterogeneity', theme='publication', ax=None)[source]¶

Plot conditional average treatment effect heterogeneity.

If covariate is provided, CATE is plotted against it; otherwise a histogram of CATE values is shown.

Parameters:

cate_values (array-like) – Individual-level CATE estimates.
covariate (array-like or None) – A continuous covariate to plot CATE against.
covariate_name (str)
title (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.event_study_plot(periods, estimates, ci_lower, ci_upper, *, treatment_period=0, title='Event Study', xlabel='Periods Relative to Treatment', ylabel='Estimate', theme='publication', ax=None)[source]¶

Event study plot with pre/post periods and confidence bands.

Parameters:

periods (sequence of int) – Relative time periods (negative = pre, positive = post).
estimates (sequence of float) – Point estimates for each period.
ci_lower (sequence of float) – Confidence interval bounds.
ci_upper (sequence of float) – Confidence interval bounds.
treatment_period (int) – The period at which treatment begins (vertical line).
title (str)
xlabel (str)
ylabel (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.waterfall_plot(labels, values, *, title='Waterfall Plot', ylabel='Value', colors=('#2c7bb6', '#d7191c'), theme='publication', ax=None)[source]¶

Waterfall plot showing sequential contributions to a total.

Parameters:

labels (sequence of str) – Step labels.
values (sequence of float) – Incremental changes at each step.
title (str)
ylabel (str)
colors (tuple of (positive_color, negative_color))
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.volcano_plot(log2_fc, neg_log10_p, *, labels=None, fc_threshold=1.0, p_threshold=1.301, title='Volcano Plot', theme='publication', ax=None)[source]¶

Volcano plot (effect size vs statistical significance).

Parameters:

log2_fc (array-like) – Log2 fold change (or any effect-size metric).
neg_log10_p (array-like) – -log10(p-value).
labels (sequence of str or None) – Feature labels for significant points.
fc_threshold (float) – Fold-change threshold for highlighting.
p_threshold (float) – -log10(p) threshold (default 1.301 = -log10(0.05)).
title (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.manhattan_plot(positions, neg_log10_p, *, chromosomes=None, significance=5e-08, suggestive=1e-05, title='Manhattan Plot', theme='publication', ax=None)[source]¶

Manhattan plot for genome-wide or multiple-testing results.

Parameters:

positions (array-like) – Genomic position or test index.
neg_log10_p (array-like) – -log10(p-value).
chromosomes (array-like or None) – Chromosome/group assignment for alternating colours.
significance (float) – Genome-wide significance threshold (p-value scale).
suggestive (float) – Suggestive significance threshold (p-value scale).
title (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.caterpillar_plot(labels, estimates, ci_lower, ci_upper, *, sort=True, title='Caterpillar Plot', xlabel='Estimate', theme='publication', ax=None)[source]¶

Caterpillar plot – ranked effects with confidence intervals.

Parameters:

labels (sequence of str)
estimates (sequence of float)
ci_lower (sequence of float)
ci_upper (sequence of float)
sort (bool) – Sort by estimate magnitude.
title (str)
xlabel (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.spaghetti_plot(data, time_col, value_col, id_col, *, group_col=None, max_lines=100, show_mean=True, title='Individual Trajectories', theme='publication', ax=None)[source]¶

Spaghetti plot showing individual trajectories over time.

Parameters:

data (DataFrame) – Long-format data.
time_col (str) – Column with the time variable.
value_col (str) – Column with the outcome.
id_col (str) – Column identifying individuals.
group_col (str or None) – Optional grouping variable for coloured trajectories.
max_lines (int) – Maximum number of individual lines to draw.
show_mean (bool) – Overlay the group/overall mean trajectory.
title (str)
theme (str or None)
ax (Axes or None)

Return type:

matplotlib.figure.Figure

morie.viz.save_figure(fig, path, *, formats=None, dpi=300, close=True)[source]¶

Save a matplotlib figure to one or more formats.

Parameters:

fig (Figure) – The figure to save.
path (str) – Base path without extension (e.g. "output/forest").
formats (list of str or None) – File extensions to save (default ["pdf", "png"]).
dpi (int) – Resolution for raster formats.
close (bool) – Whether to close the figure after saving.

Returns:

Paths to saved files.

Return type:

list of str

Rich-based animated UI helpers for long MORIE callables.

Pairs with the DoubleML-style training-output experience: live spinners, progress bars, and streaming per-iteration tables. Wired into the slow callables (Kulldorff MC permutations, longitudinal simulator subject loop, DML cross-fit folds) via an opt-in animate=True kwarg.

Falls back to plain print() if rich is not installed – the helpers are non-fatal decoration, never a hard dependency.

Public API:

animated_bar(total, description) – context-manager Progress with: spinner + bar + percentage + elapsed/remaining time columns.
streaming_table(columns) – rich.live.Live wrapper around: a per-iteration table that updates in place.
morie_banner() – one-shot ASCII banner for CLI: entrypoints (used by morie.demo).

Demo:

python -m morie.demo

morie.animate.animated_bar(total, description='morie', *, transient=False, spinner='dots')[source]¶

Yield a progress handle with .advance() method.

Rich version emits a live-updating spinner + bar + time columns. Fallback version prints periodic dots.

Example

>>> with animated_bar(3, "Kulldorff MC perms") as bar:
...     for _ in range(3):
...         _ = sum(range(1000))   # one permutation's work
...         bar.advance()

Parameters:

total (int)
description (str)
transient (bool)
spinner (str)

morie.animate.streaming_table(columns, *, title='')[source]¶

Yield a callable that appends a row to a live-updating table.

Example

>>> import contextlib, io
>>> best, shown = 0.0, io.StringIO()
>>> with contextlib.redirect_stdout(shown), \
...         streaming_table(["iter", "log_LRT", "best"]) as add:
...     for i in range(3):
...         lrt = 1.5 * i              # one scan permutation's statistic
...         best = max(best, lrt)
...         add(i, f"{lrt:.2f}", f"{best:.2f}")
>>> all(t in shown.getvalue() for t in ("log_LRT", "3.00"))   # rich table or plain rows
True

Parameters:

columns (Iterable[str])
title (str)

morie.animate.morie_banner()[source]¶

Print a one-shot MORIE banner at the start of a CLI session.

Return type:: None

morie.animate.rich_available()[source]¶

True if rich is importable and animated output is possible.

Return type:: bool

Notebook management for MORIE.

Provides utilities for creating, rendering, converting, validating, and managing Quarto (.qmd), Jupyter (.ipynb), and R Markdown (.Rmd) notebooks. Functions wrap the quarto and jupyter CLIs with progress display, error handling, and structured output.

The module supports:

Template-based notebook creation for common epidemiological analyses.
Parameterized notebook run (pass variables at render time).
Batch rendering with progress tracking.
Format conversion between .qmd, .ipynb, and .Rmd.
Notebook validation (check that all cells run without error).
Auto-generation of analysis notebooks from pipeline results.

References

Allaire, J. J. et al. (2022). Quarto: An Open-Source Scientific and Technical Publishing System. https://quarto.org

Kluyver, T. et al. (2016). Jupyter Notebooks – a publishing format for reproducible computational workflows. Positioning and Power in Academic Publishing, 87–90. https://doi.org/10.3233/978-1-61499-649-1-87

class morie.notebook.NotebookInfo(path, filename, format, title, size_bytes, modified, cell_count=0, code_cell_count=0, has_errors=False)[source]¶

Bases: object

Metadata for a notebook file.

Parameters:

path (str)
filename (str)
format (str)
title (str | None)
size_bytes (int)
modified (str)
cell_count (int)
code_cell_count (int)
has_errors (bool)

path: str¶

filename: str¶

format: str¶

title: str | None¶

size_bytes: int¶

modified: str¶

cell_count: int = 0¶

code_cell_count: int = 0¶

has_errors: bool = False¶

class morie.notebook.RenderResult(source, output_path, output_format, success, duration_seconds, stdout='', stderr='', return_code=0)[source]¶

Bases: object

Result from rendering a notebook.

Parameters:

source (str)
output_path (str)
output_format (str)
success (bool)
duration_seconds (float)
stdout (str)
stderr (str)
return_code (int)

source: str¶

output_path: str¶

output_format: str¶

success: bool¶

duration_seconds: float¶

stdout: str = ''¶

stderr: str = ''¶

return_code: int = 0¶

class morie.notebook.ConversionResult(source, target, source_format, target_format, success, message='')[source]¶

Bases: object

Result from a format conversion.

Parameters:

source (str)
target (str)
source_format (str)
target_format (str)
success (bool)
message (str)

source: str¶

target: str¶

source_format: str¶

target_format: str¶

success: bool¶

message: str = ''¶

class morie.notebook.ValidationResult(path, valid, total_cells, executed_cells, failed_cells, errors=<factory>, duration_seconds=0.0)[source]¶

Bases: object

Result from validating a notebook.

Parameters:

path (str)
valid (bool)
total_cells (int)
executed_cells (int)
failed_cells (int)
errors (list[dict[str, Any]])
duration_seconds (float)

path: str¶

valid: bool¶

total_cells: int¶

executed_cells: int¶

failed_cells: int¶

errors: list[dict[str, Any]]¶

duration_seconds: float = 0.0¶

class morie.notebook.NotebookCatalog(notebooks, total_count, by_format, scan_directory)[source]¶

Bases: object

Catalog of notebooks in the project.

Parameters:

notebooks (list[NotebookInfo])
total_count (int)
by_format (dict[str, int])
scan_directory (str)

notebooks: list[NotebookInfo]¶

total_count: int¶

by_format: dict[str, int]¶

scan_directory: str¶

morie.notebook.list_notebooks(directory='.', *, recursive=True, formats=None)[source]¶

List all notebooks in a directory.

Parameters:

directory (str | Path) – Directory to scan (default: current directory).
recursive (bool) – If True, search subdirectories.
formats (set[str] | None) – File extensions to include (default: all supported formats).

Returns:

Sorted by path.

Return type:

list[NotebookInfo]

morie.notebook.build_catalog(directory='.', *, recursive=True)[source]¶

Build a catalog of all notebooks in a directory.

Parameters:

directory (str | Path) – Directory to scan.
recursive (bool) – Search subdirectories.

Return type:

NotebookCatalog

morie.notebook.create_notebook(path, *, template='analysis', title='Untitled Analysis', author='MORIE', date=None, format='qmd', overwrite=False, custom_content=None)[source]¶

Create a new notebook from a template.

Parameters:

path (str | Path) – Output file path (extension added if missing).
template (str) – Template name: analysis, report, causal, table1, survival, presentation.
title (str) – Notebook title.
author (str) – Author name.
date (str | None) – Date string (default: today).
format (str) – Output format: qmd, ipynb, Rmd.
overwrite (bool) – If True, overwrite existing file.
custom_content (str | None) – If provided, use this as the notebook body instead of a template.

Returns:

Path to the created notebook.

Return type:

Path

Raises:

ValueError – If the template name is unknown.
FileExistsError – If the file exists and overwrite is False.

morie.notebook.render_notebook(path, *, output_format='html', output_dir=None, params=None, execute=True, timeout=600)[source]¶

Render a notebook to the specified output format.

Parameters:

path (str | Path) – Path to the notebook file.
output_format (str) – Output format: html, pdf, docx, revealjs, etc.
output_dir (str | Path | None) – Output directory (default: same as source).
params (dict[str, Any] | None) – Parameters to pass (Quarto params or Papermill parameters).
run (bool) – If True, run the notebook before rendering.
timeout (int) – Render timeout in seconds.
execute (bool)

Return type:

RenderResult

morie.notebook.render_all(directory='.', *, output_format='html', output_dir=None, recursive=True, timeout_per_notebook=600)[source]¶

Render all notebooks in a directory.

Parameters:

directory (str | Path) – Directory containing notebooks.
output_format (str) – Target format.
output_dir (str | Path | None) – Output directory for all rendered files.
recursive (bool) – Search subdirectories.
timeout_per_notebook (int) – Per-notebook timeout.

Return type:

list[RenderResult]

morie.notebook.render_preview(path, *, max_cells=5, output_format='html', timeout=120)[source]¶

Quick-render a notebook using only the first N cells.

Creates a temporary copy with truncated cells and renders it.

Parameters:

path (str | Path) – Notebook path.
max_cells (int) – Maximum number of cells to include.
output_format (str) – Output format.
timeout (int) – Render timeout.

Return type:

RenderResult

morie.notebook.execute_notebook(path, *, output_path=None, params=None, kernel='python3', timeout=600)[source]¶

Run a notebook non-interactively without rendering output.

For ipynb files, uses jupyter nbconvert --run --inplace. For qmd files, uses quarto render with run-only.

Parameters:

path (str | Path) – Notebook to run.
output_path (str | Path | None) – Where to write the run notebook (default: in-place for ipynb).
params (dict[str, Any] | None) – Run parameters.
kernel (str) – Kernel name for Jupyter run.
timeout (int) – Run timeout.

Return type:

RenderResult

morie.notebook.validate_notebook(path, *, timeout=600)[source]¶

Validate a notebook by running all cells and checking for errors.

Parameters:

path (str | Path) – Notebook to validate.
timeout (int) – Run timeout.

Return type:

ValidationResult

morie.notebook.convert_notebook(source, target_format, *, output_path=None)[source]¶

Convert a notebook between formats.

Supported conversions: qmd <-> ipynb, Rmd <-> ipynb, qmd <-> Rmd.

Parameters:

source (str | Path) – Source notebook path.
target_format (str) – Target format extension (without dot): qmd, ipynb, Rmd.
output_path (str | Path | None) – Output path. If None, uses source stem with new extension.

Return type:

ConversionResult

morie.notebook.extract_code_cells(path, *, language=None)[source]¶

Extract code cells from a notebook.

Parameters:

path (str | Path) – Notebook path.
language (str | None) – Filter by language (e.g. python, r). If None, return all.

Returns:

Code cell contents as strings.

Return type:

morie.notebook.extract_dependencies(path)[source]¶

Extract Python import statements from a notebook.

Parameters:: path (str | Path) – Notebook path.
Returns:: Keys: imports (module names), from_imports (module names).
Return type:: dict[str, list[str]]

morie.notebook.inject_results(path, results, *, output_path=None, marker='MORIE_RESULT')[source]¶

Inject computed results into a notebook.

Replaces placeholders of the form {{MORIE_RESULT:key}} with values from the results dict.

Parameters:

path (str | Path) – Notebook path.
results (dict[str, Any]) – Key-value pairs to inject.
output_path (str | Path | None) – Output path (default: overwrite in place).
marker (str) – Placeholder prefix (default MORIE_RESULT).

Returns:

Path to the modified notebook.

Return type:

Path

morie.notebook.generate_from_results(results_dir, *, module_name='analysis', output_path=None, title=None)[source]¶

Auto-generate a Quarto notebook from module output CSVs.

Scans a results directory for CSV files and creates a notebook with code cells that load and display each file.

Parameters:

results_dir (str | Path) – Directory containing CSV result files.
module_name (str) – Module name for the title.
output_path (str | Path | None) – Output notebook path. Default: results_dir/analysis_report.qmd.
title (str | None) – Notebook title. Default: auto-generated from module_name.

Returns:

Path to the generated notebook.

Return type:

Path

morie.notebook.get_metadata(path)[source]¶

Extract metadata from a notebook’s YAML frontmatter or JSON.

Parameters:: path (str | Path) – Notebook path.
Returns:: Metadata key-value pairs.
Return type:: dict[str, Any]

morie.notebook.update_metadata(path, updates)[source]¶

Update metadata fields in a notebook’s YAML frontmatter.

Parameters:

path (str | Path) – Notebook path.
updates (dict[str, Any]) – Key-value pairs to update.

Return type:

None

morie.notebook.diff_notebooks(path_a, path_b)[source]¶

Compare two notebooks and report differences.

Parameters:

path_a (str | Path) – First notebook.
path_b (str | Path) – Second notebook.

Returns:

Diff report with keys cells_added, cells_removed, cells_modified, metadata_diff.

Return type:

dict[str, Any]

morie.notebook.test_notebook(path, *, expected_outputs=None, timeout=600)[source]¶

Run a notebook and check outputs against expected values.

Parameters:

path (str | Path) – Notebook to test.
expected_outputs (dict[str, Any] | None) – Mapping of cell index (int) to expected output substring.
timeout (int) – Run timeout.

Return type:

ValidationResult

morie.notebook.generate_index(directory='.', *, output_path=None, output_format='qmd')[source]¶

Generate a notebook index/catalog page.

Creates a Quarto document that lists all notebooks in the directory with their titles, formats, and links.

Parameters:

directory (str | Path) – Directory to index.
output_path (str | Path | None) – Output file path (default: directory/notebook_index.qmd).
output_format (str) – Output format (qmd or md).

Returns:

Path to the generated index.

Return type:

Path

morie.notebook.render_catalog(catalog)[source]¶

Display a notebook catalog with rich formatting.

Parameters:: catalog (NotebookCatalog) – Catalog to display.
Return type:: None

morie.notebook.render_batch_results(results)[source]¶

Display batch render results with rich formatting.

Parameters:: results (list[RenderResult]) – Render results to display.
Return type:: None

Numerical correctness evaluation harness for morie.fn.

Goal: gate every fn behind a golden test before PyPI release. Without a gate, the fn/ tree balloons faster than correctness can be verified, and the package risks shipping silent-NaN stubs.

Source-of-truth references:

scipy.stats / scipy.special / scipy.linalg
statsmodels
R packages (frozen JSON values, since R is not always installed in CI)

Design (v0):

GoldenCase: one input/expected pair with tolerances + provenance.
JSON files at fn/_golden/<fn_name>.json hold the cases (data, not code).
run_golden(fn, name) -> list[CaseResult] runs everything and reports.
Stub detection (returns NaN-only) is reported as “stub” not “fail” so we can xfail unfilled fn cleanly while gating filled ones.

Adding a golden:

Implement (or stub-out) fn/<name>.py.
Generate reference values from scipy/statsmodels/R, save to fn/_golden/<name>.json (schema below).
tests/pytests/test_fn_golden.py auto-discovers + asserts.

JSON schema:

{

“fn”: “<short name>”, “callable”: “<python attribute path to call, eg ‘morie.fn.dnorm.dnorm’>”, “source”: “<reference impl, eg ‘scipy.stats.norm.pdf’>”, “cases”: [

{“inputs”: {…kw}, “expected”: <scalar|list|dict>,
“rtol”: 1e-7, “atol”: 0, “notes”: “”}

]

}

class morie.eval.GoldenCase(inputs: 'dict[str, Any]', expected: 'Any', source: 'str', rtol: 'float' = 1e-05, atol: 'float' = 1e-08, notes: 'str' = '')[source]¶

Bases: object

Parameters:

inputs (dict[str, Any])
expected (Any)
source (str)
rtol (float)
atol (float)
notes (str)

inputs: dict[str, Any]¶

expected: Any¶

source: str¶

rtol: float = 1e-05¶

atol: float = 1e-08¶

notes: str = ''¶

class morie.eval.CaseResult(case: 'GoldenCase', actual: 'Any', status: 'str', error: 'float | None' = None, detail: 'str' = '')[source]¶

Bases: object

Parameters:

case (GoldenCase)
actual (Any)
status (str)
error (float | None)
detail (str)

case: GoldenCase¶

actual: Any¶

status: str¶

error: float | None = None¶

detail: str = ''¶

class morie.eval.FnReport(fn_name: 'str', callable_path: 'str', source: 'str', results: 'list[CaseResult]' = <factory>)[source]¶

Bases: object

Parameters:

fn_name (str)
callable_path (str)
source (str)
results (list[CaseResult])

fn_name: str¶

callable_path: str¶

source: str¶

results: list[CaseResult]¶

property n_pass: int¶

property n_fail: int¶

property n_stub: int¶

property n_error: int¶

morie.eval.load_golden_file(path)[source]¶

Read a golden-value JSON file for the evaluation harness.

The file holds fn, callable, an optional source and a list of cases, each with inputs, expected and optional rtol, atol, source and notes (defaults 1e-5, 1e-8 and the file’s source).

Parameters:: path (Path) – the JSON file.
Returns:: (fn, callable, source, cases) with cases as GoldenCase records.
Return type:: tuple[str, str, str, list[GoldenCase]]

morie.eval.run_golden(callable_path, fn_name, source, cases)[source]¶

Run all golden cases for one fn and produce a report.

Parameters:

callable_path (str)
fn_name (str)
source (str)
cases (Iterable[GoldenCase])

Return type:

FnReport

morie.eval.discover()[source]¶

Return all golden JSON files in fn/_golden/.

Return type:: list[Path]

morie.eval.run_all()[source]¶

Run every golden file in fn/_golden/. Used by CI + the test harness.

Return type:: list[FnReport]

morie.eval.summary(reports)[source]¶

Aggregate pass/fail/stub/error counts across reports.

Parameters:: reports (Iterable[FnReport])
Return type:: dict[str, int]

Higher-level evaluation pipeline – end-to-end integration gates over BigQuery-mirrored datasets, each pinned to a published ground-truth claim about a headline coefficient or shape statistic.

Why this exists alongside morie.eval:

morie.eval (per-fn goldens) catches drift in one function – “did dnorm(0) change from 0.3989… last week?”. That is correctness at the leaf. But nothing there catches the kind of bug where a function change quietly breaks an integration – e.g., a switch from HC1 to HC3 SEs in the OLS path that flips a published coefficient from significant to insignificant on a real-world dataset.

morie.eval_pipeline catches that: each gate is a claim of the form “this analysis on this dataset should produce a headline coefficient in range X” – sourced from a peer-reviewed paper or a government report. When a gate fails, the bug is somewhere in the chain (data loading, canonicalisation, the model fit, the SE formula); the evaluation points at the broken integration even when every per-fn unit test still passes.

The harness ships seed gates against well-known public datasets. New gates land as DatasetGate(...) entries in SEED_GATES below, or via morie.eval_pipeline.register(...) from anywhere downstream.

class morie.eval_pipeline.DatasetGate(name, slug, description, citation, runner, expected)[source]¶

Bases: object

One end-to-end claim about an analysis result.

name : short ID, used in test parametrize and reports. slug : database slug (under the configured remote endpoint

or matching a local SQLite mirror).

description : the claim being verified, in plain English. citation : URL or publication this claim is sourced from. runner : callable(slug) -> dict of computed coefficients. expected : per-key (lower, upper) tolerance interval. Failure

if the actual value falls outside the interval. Use (None, threshold) for “must be <= threshold”, or (threshold, None) for “must be >= threshold”.

Parameters:

name (str)
slug (str)
description (str)
citation (str)
runner (Callable[[str], dict[str, Any]])
expected (dict[str, tuple[float | None, float | None]])

name: str¶

slug: str¶

description: str¶

citation: str¶

runner: Callable[[str], dict[str, Any]]¶

expected: dict[str, tuple[float | None, float | None]]¶

class morie.eval_pipeline.GateResult(gate: 'DatasetGate', actual: 'dict[str, Any] | None', status: 'str', failures: 'list[str]' = <factory>, detail: 'str' = '')[source]¶

Bases: object

Parameters:

gate (DatasetGate)
actual (dict[str, Any] | None)
status (str)
failures (list[str])
detail (str)

gate: DatasetGate¶

actual: dict[str, Any] | None¶

status: str¶

failures: list[str]¶

detail: str = ''¶

morie.eval_pipeline.run_gate(g)[source]¶

Run one gate end-to-end. Catches all exceptions so the suite reports per-gate, not a single hard failure.

Parameters:: g (DatasetGate)
Return type:: GateResult

morie.eval_pipeline.register(gate)[source]¶

Append a gate to the registry. Useful for downstream test suites that want to extend coverage without editing this file.

Parameters:: gate (DatasetGate)
Return type:: None

morie.eval_pipeline.gates()[source]¶

Snapshot of the current registry.

Return type:: list[DatasetGate]

morie.eval_pipeline.run_all()[source]¶

Run every registered evaluation gate and return their results in registry order.

Return type:: list[GateResult]

morie.eval_pipeline.summary(results)[source]¶

Count gate results by status.

Parameters:: results (list[GateResult]) – the GateResult list from run_all.
Returns:: A dict with pass, fail, skip, error and total.
Return type:: dict[str, int]

Examples

>>> summary([])
{'pass': 0, 'fail': 0, 'skip': 0, 'error': 0, 'total': 0}

Assistant, LLM and model tooling¶

Agentic loop for Perseus – MORIE’s resident AI with tool calling.

SECURITY / TRUST MODEL¶

This agent only runs when the LOCAL USER explicitly starts it against an LLM endpoint they configured (default: their own local ollama). It is never imported or executed on plain import morie. Because LLM output is untrusted, every execution sink is guarded by morie._exec_guard:

tool_run_shell – shlex-tokenized, shell=False, binary allowlist (no pipes, redirection, or command substitution);
all exec of LLM-generated setup code – AST-validated (import whitelist, no underscore attributes, restricted builtins);
MORIE_NO_EXEC=1 disables every sink for CI/shared environments.

class morie.agent.AgentResponse(text, tool_calls_made=<factory>, iterations=0, model='', failed=False)[source]¶

Bases: object

Response from a Perseus agent loop.

Parameters:

text (str)
tool_calls_made (list[dict])
iterations (int)
model (str)
failed (bool)

text: str¶

tool_calls_made: list[dict]¶

iterations: int = 0¶

model: str = ''¶

failed: bool = False¶

morie.agent.tool_read_file(path, *, max_lines=200, sandbox=None)[source]¶

Read a file, sandboxed to project root.

Parameters:

path (str)
max_lines (int)
sandbox (Path | None)

Return type:

morie.agent.tool_write_file(path, content, *, sandbox=None)[source]¶

Write a file, sandboxed to project root.

Parameters:

path (str)
content (str)
sandbox (Path | None)

Return type:

morie.agent.tool_execute_code(code, language='python')[source]¶

Execute LLM-generated Python under the execution guard.

This tool deliberately does NOT expose the multi-language polyglot engine (shell/R/SQL/native) to the LLM: that would be an unrestricted remote-code-execution path driven by untrusted model output. The LLM runs shell via the allowlisted run_shell tool and statistics via run_morie_function instead. Python here is AST-validated (import whitelist, no dunder access, restricted builtins) by morie._exec_guard, and MORIE_NO_EXEC=1 disables it entirely.

Parameters:

code (str)
language (str)

Return type:

morie.agent.tool_web_fetch(url, *, max_chars=5000)[source]¶

Fetch a URL and return text content.

Parameters:

url (str)
max_chars (int)

Return type:

morie.agent.tool_search_functions(query, *, limit=10)[source]¶

Search the fn/ registry by keyword.

Parameters:

query (str)
limit (int)

Return type:

morie.agent.tool_list_files(path='.', *, pattern='*', sandbox=None)[source]¶

List files in a directory, sandboxed.

Parameters:

path (str)
pattern (str)
sandbox (Path | None)

Return type:

morie.agent.tool_inspect_error()[source]¶

Get the last Python traceback for debugging.

Return type:: str

morie.agent.tool_get_cheatsheet(name)[source]¶

Get the cheatsheet for an morie function (usage, signature, examples).

Parameters:: name (str)
Return type:: str

morie.agent.tool_search_codebase(pattern, *, file_glob='*.py', max_results=20, sandbox=None)[source]¶

Search the morie codebase for a pattern (grep-like).

Parameters:

pattern (str)
file_glob (str)
max_results (int)
sandbox (Path | None)

Return type:

morie.agent.tool_run_shell(command, *, timeout=30)[source]¶

Run an allowlisted command without a shell.

LLM-chosen commands are untrusted input: the command is shlex- tokenized (no shell, so no pipes/redirection/substitution) and only binaries on morie._exec_guard’s allowlist may run. MORIE_NO_EXEC=1 disables this tool entirely.

Parameters:

command (str)
timeout (int)

Return type:

morie.agent.tool_describe_data(code='')[source]¶

Describe a dataset: load from morie.data or describe a DataFrame expression.

Parameters:: code (str)
Return type:: str

morie.agent.tool_run_morie_function(name, kwargs=None)[source]¶

Run any registered morie fn/ function by name.

Parameters:

name (str)
kwargs (dict | None)

Return type:

morie.agent.tool_domain_guide(domain)[source]¶

Get complete guide for a statistical domain: functions, workflows, textbook refs.

Parameters:: domain (str)
Return type:: str

morie.agent.tool_recommend_analysis(question)[source]¶

Given a research question, recommend specific morie functions and analysis workflow.

Parameters:: question (str)
Return type:: str

morie.agent.tool_category_tree()[source]¶

Show full taxonomy of all function categories with counts and examples.

Return type:: str

morie.agent.tool_similar_functions(name, limit=10)[source]¶

Find functions similar to a given one (same category, related keywords).

Parameters:

name (str)
limit (int)

Return type:

morie.agent.tool_textbook_reference(topic)[source]¶

Look up which textbook and chapter covers a statistical topic.

Parameters:: topic (str)
Return type:: str

morie.agent.tool_run_pipeline(steps)[source]¶

Run a sequence of morie functions as a pipeline. Results from step N available as $N in later steps.

steps: JSON array like [{“fn”:”moran”,”kwargs”:{“values”:[1,2,3]}}, …]

Parameters:: steps (str)
Return type:: str

morie.agent.tool_compare_methods(methods, data_code='')[source]¶

Run multiple statistical methods on same data, compare results.

methods: comma-separated fn/ names (e.g. “d,g,cles,cliff”) data_code: Python setup code (e.g. “group1=[1,2,3]; group2=[4,5,6]”)

Parameters:

methods (str)
data_code (str)

Return type:

morie.agent.tool_run_suite(domain, data_code='')[source]¶

Run a complete analysis suite for a domain using its core functions.

domain: one of spatial, causal, hypothesis, effect_size, reliability, etc. data_code: Python setup code for your data (optional; some domains have demos)

Parameters:

domain (str)
data_code (str)

Return type:

PerseusAgent | PerseusCloudAgent

class morie.agent.PerseusAgent(model='perseus:e2b', base_url='http://localhost:11434', *, sandbox_root=None, max_iterations=10)[source]¶

Bases: object

Full agentic loop for Perseus with tool calling.

Parameters:

model (str)
base_url (str)
sandbox_root (str | Path | None)
max_iterations (int)

chat(message, *, stream=False)[source]¶

Run a full agent loop: send message, handle tool calls, return final response.

Parameters:

message (str)
stream (bool)

Return type:

AgentResponse

chat_stream(message)[source]¶

Streaming variant: yields text chunks, executes tools between chunks.

Parameters:: message (str)
Return type:: Iterator[str]

reset()[source]¶

Clear conversation history.

Return type:: None

close()[source]¶

Clean up resources.

Return type:: None

class morie.agent.PerseusCloudAgent(url, token=None)[source]¶

Bases: object

Agent that connects to a remote Perseus relay server.

The relay runs on Pi (or any machine) and handles tool execution server-side. The client just sends questions and gets full results back – zero local compute needed.

This is how Perseus becomes a cloud service: run the relay on Pi, expose via Cloudflare Tunnel or port forward, and anyone with the URL can use Perseus with full tool-calling capabilities.

Parameters:

url (str)
token (str | None)

chat(message, **_kwargs)[source]¶

Parameters:

message (str)
_kwargs (Any)

Return type:

AgentResponse

chat_stream(message, **_kwargs)[source]¶

Parameters:

message (str)
_kwargs (Any)

Return type:

Iterator[str]

close()[source]¶

Return type:: None

morie.agent.create_agent(model=None, base_url=None, *, cloud_url=None, cloud_token=None, **kwargs)[source]¶

Create the best available Perseus agent.

Priority: 1. Explicit cloud URL – remote Perseus relay (full tools, server-side execution) 2. Local/remote Ollama – native tool calling (fastest, most capable) 3. Perseus relay on MORIE_PI_HOST 4. Fallback – local Ollama endpoint (may not be running)

Environment variables: - PERSEUS_CLOUD_URL: Auto-connect to a Perseus relay - PERSEUS_CLOUD_TOKEN: Auth token for the relay - OLLAMA_HOST: Remote Ollama instance - MORIE_PI_HOST: Pi hostname (used to construct Ollama URL)

Parameters:

model (str | None)
base_url (str | None)
cloud_url (str | None)
cloud_token (str | None)
kwargs (Any)

Return type:

morie.agent.warmup(model=None, base_url=None)[source]¶

Pre-load the model into GPU/RAM so subsequent queries are fast.

Sets keep_alive=-1 (infinite) so the model stays resident. Call this once when morie starts – after that, tool-calling queries take 2-10s instead of 30-120s.

Parameters:

model (str | None)
base_url (str | None)

Return type:

Interactive chat REPL for MORIE – Claude Code-like terminal experience.

Provides a multi-turn conversational interface with streaming LLM responses, slash commands for module run, and agent persona switching. Uses only rich (core dependency) – does NOT require Textual.

Usage:

morie chat                        # default assistant
morie chat --agent morie-architect # agent-specific persona

Slash commands:

/run <module>       Run an analysis module with progress display
/list               List available modules
/doctor             Run environment diagnostics
/profile <csv>      Profile a dataset
/inspect <path>     Inspect output CSV(s)
/verify <path>      Verify statistical outputs
/agent <name>       Switch to an agent persona
/agents             List available agents
/provider           Show current LLM provider
/history            Show conversation history
/clear              Clear conversation history
/help               Show available commands
/quit               Exit the REPL

class morie.chat.ChatMessage(role, content, timestamp=0.0, provider=None)[source]¶

Bases: object

A single message in the conversation.

Parameters:

role (str)
content (str)
timestamp (float)
provider (str | None)

role: str¶

content: str¶

timestamp: float = 0.0¶

provider: str | None = None¶

class morie.chat.SlashCommand(name, description, handler)[source]¶

Bases: object

Registry entry for a slash command.

Parameters:

name (str)
description (str)
handler (Callable[[list[str]], str | None])

name: str¶

description: str¶

handler: Callable[[list[str]], str | None]¶

morie.chat.list_agents()[source]¶

List available agent definitions.

Returns:: Each dict has name and description keys.
Return type:: list[dict[str, str]]

morie.chat.load_agent_prompt(agent_name)[source]¶

Load an agent’s markdown file and return it as a system prompt.

Parameters:: agent_name (str) – Agent filename stem (e.g., "morie-architect").
Returns:: The agent file content, or None if not found.
Return type:: str | None

class morie.chat.ChatSession(*, system_prompt=None, agent=None)[source]¶

Bases: object

Manages a multi-turn chat conversation.

Parameters:

system_prompt (str | None) – Custom system prompt. When None, the default MORIE agent prompt is used (with module/data context injected).
agent (str | None) – Agent name to load. Overrides system_prompt if provided.

history: list[ChatMessage]¶

slash_commands: dict[str, SlashCommand]¶

dataset_context: str | None¶

set_dataset_context(df, name='loaded')[source]¶

Inject current dataset metadata so the LLM knows what data is loaded.

Parameters:

df (pd.DataFrame)
name (str)

Return type:

None

send(user_input, *, stream=True)[source]¶

Send user input and get a response.

Handles slash commands internally. For regular messages, sends to the LLM provider chain.

Parameters:

user_input (str) – The user’s message or slash command.
stream (bool) – If True, return a streaming iterator for LLM responses.

Returns:

The response text or streaming iterator.

Return type:

str | Iterator[str]

cancel_stream()[source]¶

Signal that the current stream should be cancelled.

Return type:: None

morie.chat.run_chat_repl(*, agent=None)[source]¶

Launch the interactive chat REPL.

Parameters:: agent (str | None) – Agent name to load at startup.
Returns:: Exit code (always 0 for normal exit).
Return type:: int

Language models for MORIE: the one you bring.

Nothing is hosted by the project and no prompt goes anywhere you did not set up. ask uses, in the automatic order (route = auto), the same one rmoriebricklayer uses:

own – your own OpenAI-compatible server (LM Studio, vLLM, llama.cpp, a gateway)

provider – the first provider whose key is set: OpenAI, Anthropic, Google Gemini,
OpenRouter, Groq, Together, Fireworks, DeepSeek, Mistral, xAI, Cerebras, Perplexity, Hugging Face, OpenCode Zen, Cohere, NVIDIA (morie.providers)

ollama – an Ollama server (ollama serve) with a model pulled

local – MORIE’s keyword help, marked as such, no network

Anthropic’s Messages API and Gemini’s generateContent are spoken natively; the rest are OpenAI-compatible (httpx).

Environment Variables¶

ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, GROQ_API_KEY, …str: A provider’s key (morie.providers.PRESETS lists every one); its model is MORIE_<PROVIDER>_MODEL, else the provider’s default.
MORIE_LLM_ROUTEstr: auto (default), own, ollama or a provider id: force the route ask takes.
OLLAMA_HOST, OLLAMA_MODEL, OLLAMA_API_KEYstr: The Ollama server (OLLAMA_BASE_URL is still read), its model and an optional key (OLLAMA_HOST=https://ollama.com with an Ollama key reaches Ollama’s own cloud).
MORIE_LLM_BASE_URL, MORIE_LLM_API_KEY, MORIE_LLM_MODELstr: Your own OpenAI-compatible server (LLM_API_BASE_URL / LLM_API_KEY are still read).

Every one of these can be saved instead with morie config set KEY VALUE or config() ($XDG_CONFIG_HOME/morie/llm.json, the file rmoriebricklayer’s rmbl config writes too); a variable that is set wins over the saved value. morie config help lists them, morie doctor says which route ask takes.

References

Ollama API docs: https://github.com/ollama/ollama/blob/main/docs/api.md
Anthropic Messages API: https://docs.anthropic.com/en/api/messages
Gemini generateContent: https://ai.google.dev/api/generate-content
OpenAI Chat Completions API: https://platform.openai.com/docs/api-reference/chat

morie.llm.detect_available_provider(route=None)[source]¶

Which language model ask would use: the one you brought.

route (else MORIE_LLM_ROUTE, else the route saved with morie config set route) forces one: "own" (your OpenAI-compatible endpoint), "ollama", or a provider id ("anthropic", "openai", "gemini", "groq", …; morie.providers.IDS). With "auto" (the default) the order is the one rmoriebricklayer uses:

own – your own endpoint is set (morie config set own.url, MORIE_LLM_BASE_URL).
a provider whose key is set (ANTHROPIC_API_KEY, OPENAI_API_KEY, … or morie config set <provider>.key), in the order of morie.providers.PRESETS.
ollama – an Ollama server answers and has a model (one pulled, or OLLAMA_MODEL).
local – nothing is set up: the answer is MORIE’s keyword help, marked as such.

Nothing is hosted by the project: no prompt goes anywhere you did not set up.

Returns:: "own", "ollama", a provider id, or "local".
Return type:: str
Parameters:: route (str | None)

Examples

>>> detect_available_provider() in ("own", "ollama", "local", *__import__("morie.providers").providers.IDS)
True

morie.llm.detect_provider_and_model(route=None)[source]¶

Detect LLM provider and return (provider, human-readable model label).

Returns:: (provider_key, display_label) – e.g. ("anthropic", "Anthropic:claude-opus-5-5").
Return type:: tuple[str, str]
Parameters:: route (str | None)

morie.llm.detect_model_display()[source]¶

Return display info with inner (family:size) and outer (model name).

Returns:: Keys: inner, outer, model, provider.
Return type:: dict[str, str]

morie.llm.route_status()[source]¶

One row per route (own endpoint, each provider with a key, local Ollama): route, status, detail.

Return type:: list[dict[str, str]]

morie.llm.route_summary(route=None)[source]¶

One line on the route ask takes now, e.g. ask uses: Anthropic, model claude-opus-5-5.

Parameters:: route (str | None)
Return type:: str

morie.llm.pick_thinking_word(query)[source]¶

Pick a context-aware thinking word based on the query, or a random one.

Parameters:: query (str)
Return type:: str

morie.llm.build_morie_context(repo_root=None)[source]¶

Build an LLM-friendly context dictionary from the MORIE package state.

The returned dictionary is designed to be injected into the system prompt so the LLM is aware of the available modules, the CPADS data contract, and the current working directory.

Parameters:

repo_root (str | Path | None) – Path to the MORIE repository root. When None the function attempts to resolve the root from this file’s location.

Returns:

A dictionary with keys:

module_list – list of module name/description pairs.
cpads_schema – the CPADS data contract dictionary.
cwd – the current working directory as a string.
repo_root – the resolved repository root, or "unknown".

Return type:

dict[str, Any]

Examples

>>> ctx = build_morie_context()
>>> "module_list" in ctx and "cpads_schema" in ctx
True

morie.llm.get_last_traceback()[source]¶

Return the last Python traceback, if any, for error-context injection.

Return type:: str

exception morie.llm.EmptyAnswerError[source]¶

Bases: Exception

The provider answered with no text, twice: the chain moves on to the next provider.

exception morie.llm.NoModelError[source]¶

Bases: ValueError

A provider whose key is set but whose model is not: said with how to name one.

morie.llm.ask(prompt, context=None, *, stream=False, model=None, provider=None, route=None, system_prompt=None, timeout=120.0)[source]¶

Send a prompt to the best available LLM provider and return the response.

The route chain is the one you brought: your own OpenAI-compatible endpoint -> each provider whose key is set -> Ollama -> MORIE’s keyword help. Each is tried in order; on failure the next is attempted (a forced route never falls back to another one).

Parameters:

prompt (str) – The user’s question or instruction.
context (dict[str, Any] | None) – Optional context dictionary (e.g., from build_morie_context()). Injected into the system prompt to give the LLM awareness of available modules, CPADS schema, and the user’s working directory.
stream (bool) – If True, return an iterator of string chunks for streaming output. If False (default), return the full response as a single string.
model (str | None) – Override the model identifier. When None, a sensible default is chosen per provider.
provider (str | None) – Force a specific provider ("own", "ollama", a provider id such as "anthropic", or "local"). When None, detect_available_provider() is used to auto-detect.
route (str | None) – "auto", "own", "ollama" or a provider id. When None, the route saved with morie config set route (or MORIE_LLM_ROUTE) applies, else "auto". A forced route never falls back to another one.
system_prompt (str | None) – Override the entire system prompt. When None, the standard MORIE system prompt is built from the context parameter.
timeout (float) – HTTP request timeout in seconds.

Returns:

The LLM response text (or a streaming iterator of text chunks). When all providers fail, returns a local fallback help string.

Return type:

str | Iterator[str]

Examples

Needs a model you brought (a provider’s key, your own server, or a local Ollama):

>>> # Non-streaming (returns full text)
>>> response = ask("What is AIPW?")
>>> isinstance(response, str)
True

>>> # Streaming: chunks print as they arrive
>>> for chunk in ask("Explain TMLE", stream=True):
...     print(chunk, end="")

morie.llm.ask_multi(messages, *, stream=False, model=None, provider=None, route=None, timeout=120.0)[source]¶

Send a pre-built messages array to the best available LLM provider.

Unlike ask(), this accepts the full messages array directly, enabling multi-turn conversation support. The caller is responsible for constructing the system and user messages.

Parameters:

messages (list[dict[str, str]]) – The chat messages array (system, user, assistant turns).
stream (bool) – If True, return an iterator of string chunks.
model (str | None) – Override the model identifier.
provider (str | None) – Force a specific provider. Auto-detected when None.
route (str | None) – "auto", "own", "ollama" or a provider id; None uses the saved route.
timeout (float) – HTTP request timeout in seconds.

Returns:

The LLM response text (or a streaming iterator).

Return type:

str | Iterator[str]

morie.llm.agent_available()[source]¶

Return True when at least one live LLM provider is available.

Returns:: True if a live provider is detected, False if only local fallback is available.
Return type:: bool

Examples

>>> isinstance(agent_available(), bool)
True

morie.llm.assistant_available()¶

Return True when at least one live LLM provider is available.

Returns:: True if a live provider is detected, False if only local fallback is available.
Return type:: bool

Examples

>>> isinstance(agent_available(), bool)
True

The MORIE data key: the personal key that opens the curated tables at data.rmorie.com.

No language model is hosted by the project any more (the tier at llm.rmorie.com is retired; a model is brought by the user, see morie.providers). What remains here is the key for the curated tables: minted by morie login (GitHub device flow) or morie login --email at the data service’s own sign-in, or given and stored with morie login --token; checked with the service before it is kept; stored 0600 in $XDG_CONFIG_HOME/morie/credentials.json, the file rmorie, rmoriedata and rmoriebricklayer read too. The addresses come from the signed services document (morie.services).

Environment¶

MORIE_HOSTED_KEY: Override the stored key (CI, containers).
MORIE_HOSTED_AUTH_URL: Override the sign-in service (https://data.rmorie.com/auth).

morie.hosted.hosted_base_url()[source]¶

The data service the key belongs to (the services document’s data.base_url), or None when the document switches it off. A key is only ever sent there.

Return type:: str | None

morie.hosted.hosted_auth_url()[source]¶

The sign-in service (MORIE_HOSTED_AUTH_URL, else the services document’s data auth_url), without a trailing slash.

Return type:: str

morie.hosted.access_hint()[source]¶

One line on how to get a hosted key.

Return type:: str

morie.hosted.credentials_path()[source]¶

$XDG_CONFIG_HOME/morie/credentials.json (shared with the R package).

Return type:: Path

morie.hosted.read_credentials()[source]¶

The stored credentials as a dict; {} when the file is missing, unreadable or not a JSON object.

Return type:: dict

morie.hosted.write_credentials(data)[source]¶

Write the credentials file with owner-only permissions.

Parameters:: data (dict)
Return type:: Path

morie.hosted.hosted_key()[source]¶

The user’s key: MORIE_HOSTED_KEY, else the stored credential.

Return type:: str | None

morie.hosted.reset_probe_cache()[source]¶

Forget the cached key check (accepted, last failure) so the next call checks again.

Return type:: None

morie.hosted.probe_hosted(timeout=5.0)[source]¶

True when a key is stored and the data service accepts it (/v1/whoami; a service without it answers /manifest.json). Cached for the process; never called without a key.

Parameters:: timeout (float)
Return type:: bool

morie.hosted.hosted_failure()[source]¶

Why the last check failed: “rejected” (401/403: the key is no longer valid), “network”, “http NNN”, or None when it succeeded.

Return type:: str | None

morie.hosted.hosted_problem_line(cli='morie')[source]¶

One sentence for a signed-in user whose key check failed, with the remedy.

Parameters:: cli (str)
Return type:: str

morie.hosted.terms_info()[source]¶

The current version of the data access terms and where to read them, from the sign-in service (GET {data}/terms.json). Raises RuntimeError when they cannot be read.

Return type:: dict

morie.hosted.signin_profile(accept_terms=False, name=None, affiliation=None, purpose=None, restricted=None, supervisor_name=None, supervisor_role=None, supervisor_email=None, ask=<built-in function input>, echo=<function _say>)[source]¶

What a new key needs said first, before any sign-in starts.

The data access terms are accepted ([y/N] at the prompt, or accept_terms=True), then a name and the reason for needing the data are given, and, for the restricted tables (Schedule B: ICPSR, NCVS 38604, A2AJ case law), a supervisor, department chair, director or coordinator with their role and email. Arguments win; anything missing is asked with ask; a missing required answer raises ValueError, so no request that could issue a key is made. The data service checks the same things. Returns the fields sent with the sign-in.

Parameters:

accept_terms (bool)
name (str | None)
affiliation (str | None)
purpose (str | None)
restricted (bool | None)
supervisor_name (str | None)
supervisor_role (str | None)
supervisor_email (str | None)

Return type:

morie.hosted.device_login(open_browser=True, poll_max_seconds=600.0, echo=<function _say>, profile=None)[source]¶

Run the GitHub device flow against the data service’s sign-in.

POST {auth}/device/code returns user_code, verification_uri, device_code and interval.
The user enters the code on GitHub (the browser is opened when possible); the CLI polls POST {auth}/device/token until the service has verified the sign-in and minted a key.
The key is stored 0600 in credentials_path() and returned.

profile is what signin_profile() returned: the accepted terms and who is asking, which the service needs before it issues a key.

Parameters:

open_browser (bool)
poll_max_seconds (float)
profile (dict | None)

Return type:

morie.hosted.store_token(token, echo=<function _say>)[source]¶

Store a data key obtained elsewhere (one emailed, or given on request).

The key is checked with the data service first; a key it refuses, or one that cannot be checked, is not kept (the credentials file is left as it was).

Parameters:: token (str)
Return type:: str

morie.hosted.email_login(email, code=None, ask=<built-in function input>, echo=<function _say>, to_email=False, profile=None)[source]¶

Sign in with an emailed one-time code instead of GitHub.

POST {auth}/email/code sends a one-time code (10 minutes, single use); the user types it (or passes code), POST {auth}/email/verify returns the key, stored like the device-flow one. With to_email the service emails the key instead of returning it; nothing is stored and the user pastes it later with morie login --token. profile is what signin_profile() returned, sent with the request for the code.

Parameters:

email (str)
code (str | None)
to_email (bool)
profile (dict | None)

Return type:

morie.hosted.logout(echo=<function _say>)[source]¶

Forget the stored key. Returns True when a key was removed.

Return type:: bool

morie.hosted.status()[source]¶

What morie doctor reports for the data key: the service, whether a key is stored, whether it works.

Return type:: dict

morie.hosted.provider_set(base_url, key, model=None, echo=<function _say>)[source]¶

Attach an OpenAI-compatible endpoint (chat completions at BASE_URL/chat/completions).

Works for OpenAI, Anthropic’s compatibility endpoint (https://api.anthropic.com/v1), OpenRouter, Mistral, Groq, a local LM Studio / vLLM / llama.cpp server, or any other server that speaks that API. Environment variables LLM_API_BASE_URL, LLM_API_KEY and MORIE_API_MODEL still take precedence when set.

Parameters:

base_url (str)
key (str)
model (str | None)

Return type:

morie.hosted.provider_show(echo=<function _say>)[source]¶

Describe the attached OpenAI-compatible endpoint without printing its key.

Parameters:: echo – where the description goes (default: stdout).
Returns:: The stored api_base_url, api_model and api_key entries (None where unset).
Return type:: dict

morie.hosted.provider_unset(echo=<function _say>)[source]¶

Detach the OpenAI-compatible endpoint from the stored credentials.

Other stored credentials are kept; the file is removed only when nothing else is left in it, and nothing is written when no endpoint was attached.

Parameters:: echo – where the outcome line goes (default: stdout).
Returns:: True if an endpoint was attached.
Return type:: bool

morie.hosted.provider_line()[source]¶

One line for morie models when an endpoint is attached or set in the environment.

Return type:: str | None

Curated datasets at data.rmorie.com, opened by the MORIE key.

The MORIE project materialises public BigQuery datasets (Chicago crime, EPA air quality, census, FEC, FDA, NOAA, …) and serves them as CSV from the edge (Cloudflare R2 behind a Worker), so they are reachable whenever the internet is, with the personal data key morie login stores (shared with rmorie, rmoriedata and rmoriebricklayer). /manifest.json lists every table with its rows, columns, size, SHA-256 and the BigQuery source it was built from; /<db>/<table>.csv.gz is the table. Downloads are cached in the dataset store, so a key is needed once per table, not per call.

exception morie.datahub.DataHubAuthError[source]¶

Bases: RuntimeError

No key, or a key the gateway rejects.

morie.datahub.data_url()[source]¶

Base URL of the hosted dataset hub, without a trailing slash.

MORIE_DATA_URL overrides; otherwise the signed services document decides (morie.services). Raises RuntimeError when the document says the service is off.

Examples

>>> data_url().startswith("https://")
True

Return type:: str

morie.datahub.offline()[source]¶

True under MORIE_OFFLINE (set by morie selftest): nothing is fetched, the disk copy is used.

Return type:: bool

morie.datahub.hosted_manifest(refresh=False)[source]¶

The gateway’s manifest, cached for a day under the user cache dir (only the cached copy when offline).

Parameters:: refresh (bool)
Return type:: dict

morie.datahub.cached_manifest()[source]¶

The manifest if it was fetched before (no network, no key needed).

Return type:: dict | None

morie.datahub.hosted_table_name(key)[source]¶

Cache-table name for a hosted dataset key: hub_ plus the key with / written as __.

Examples

>>> hosted_table_name("ethereum_tokens/ethereum_tokens")
'hub_ethereum_tokens__ethereum_tokens'

Parameters:: key (str)
Return type:: str

morie.datahub.is_hosted_key(key)[source]¶

True for a hosted dataset key of the form group/table (not a file path).

Examples

>>> is_hosted_key("ethereum_tokens/ethereum_tokens"), is_hosted_key("./data.csv"), is_hosted_key("cihi849")
(True, False, False)

Parameters:: key (str)
Return type:: bool

morie.datahub.load_hosted_dataset(key, *, db_path=None, refresh=False)[source]¶

db/table from data.rmorie.com: the dataset store first, then one download.

Parameters:

key (str)
db_path (str | Path | None)
refresh (bool)

morie.datahub.hosted_to_csv(key, out)[source]¶

Write db/table straight to a CSV file, row by row: (rows, columns).

For tables too large to build in memory (chicago_crime/incidents, 8.6M rows, was killed after its download); nothing is cached.

Parameters:

key (str)
out (str | Path)

Return type:

tuple[int, int]

morie.datahub.hosted_entries(manifest)[source]¶

Catalog-shaped rows for list_datasets().

Parameters:: manifest (dict | None)
Return type:: list[dict]

Perseus – the MORIE resident AI agent.

Delegates to the provider-chain in morie.llm. The LLM module handles Ollama, Gemini, OpenAI-compatible APIs, direct OpenAI, and a local static fallback.

When stream=True is passed to ask_percy(), the returned dictionary contains an output_stream key (an iterator of string chunks) instead of output_text.

morie.perseus.build_prompt(question, context=None)[source]¶

Build a prompt from a user question and optional context.

Parameters:

question (str)
context (str | None)

Return type:

morie.perseus.ask_percy(question, *, context=None, model=None, system_prompt='You are Perseus, the MORIE agent for methods for observational inference and robust analysis of interventions in sociolegal studies.\nHelp users understand datasets, methods, debugging steps, testing strategy, and interpretation.\nBe explicit about assumptions, limitations, missing data concerns, and reproducibility risks.\nDo not invent data access or approval status for restricted datasets.', allow_fallback=True, stream=False, use_agent=True, route=None)[source]¶

Query Perseus via the LLM provider chain.

When use_agent=True (default) and Ollama is available, Perseus uses the full agentic loop with 13 tools (search, run, read/write, shell, data). Falls back to simple LLM chat or static text when tools are unavailable.

Returns a dict with mode, model, and either output_text (str) or output_stream (Iterator[str]).

route ("auto", "own", "ollama", "hosted") forces the route; None uses the one saved with morie config set route (else auto).

Parameters:

question (str)
context (str | None)
model (str | None)
system_prompt (str)
allow_fallback (bool)
stream (bool)
use_agent (bool)
route (str | None)

Return type:

morie.perseus.ask_morie_assistant(question, *, context=None, model=None, system_prompt='You are Perseus, the MORIE agent for methods for observational inference and robust analysis of interventions in sociolegal studies.\nHelp users understand datasets, methods, debugging steps, testing strategy, and interpretation.\nBe explicit about assumptions, limitations, missing data concerns, and reproducibility risks.\nDo not invent data access or approval status for restricted datasets.', allow_fallback=True, stream=False, use_agent=True, route=None)¶

Query Perseus via the LLM provider chain.

When use_agent=True (default) and Ollama is available, Perseus uses the full agentic loop with 13 tools (search, run, read/write, shell, data). Falls back to simple LLM chat or static text when tools are unavailable.

Returns a dict with mode, model, and either output_text (str) or output_stream (Iterator[str]).

route ("auto", "own", "ollama", "hosted") forces the route; None uses the one saved with morie config set route (else auto).

Parameters:

question (str)
context (str | None)
model (str | None)
system_prompt (str)
allow_fallback (bool)
stream (bool)
use_agent (bool)
route (str | None)

Return type:

morie.perseus.morie_ask_percy(question, *, context=None, model=None, system_prompt='You are Perseus, the MORIE agent for methods for observational inference and robust analysis of interventions in sociolegal studies.\nHelp users understand datasets, methods, debugging steps, testing strategy, and interpretation.\nBe explicit about assumptions, limitations, missing data concerns, and reproducibility risks.\nDo not invent data access or approval status for restricted datasets.', allow_fallback=True, stream=False, use_agent=True, route=None)¶

Query Perseus via the LLM provider chain.

When use_agent=True (default) and Ollama is available, Perseus uses the full agentic loop with 13 tools (search, run, read/write, shell, data). Falls back to simple LLM chat or static text when tools are unavailable.

Returns a dict with mode, model, and either output_text (str) or output_stream (Iterator[str]).

route ("auto", "own", "ollama", "hosted") forces the route; None uses the one saved with morie config set route (else auto).

Parameters:

question (str)
context (str | None)
model (str | None)
system_prompt (str)
allow_fallback (bool)
stream (bool)
use_agent (bool)
route (str | None)

Return type:

morie.perseus.build_assistant_prompt(question, context=None)¶

Build a prompt from a user question and optional context.

Parameters:

question (str)
context (str | None)

Return type:

Perseus Relay – serve Perseus as a cloud API endpoint.

Run on Pi (or any machine with Ollama) to let remote users access Perseus with full tool-calling capabilities over the internet.

Usage:: python -m morie.perseus_relay # default :8421 python -m morie.perseus_relay –port 9000 # custom port python -m morie.perseus_relay –token mysecret # require auth token
Then from any machine:: morie percy –cloud https://your-server:8421 “What is Moran’s I?”

Or set PERSEUS_CLOUD_URL in .env and it auto-connects.

Security: The relay only exposes Perseus agent capabilities (search, run functions, read files within sandbox); with the interactive layer the agent’s tools can also run shell commands, run code and write files on the serving machine, which is why a bind beyond loopback requires –token. No filesystem writes outside the project. Optional token auth for production use.

class morie.perseus_relay.PerseusRelayHandler(request, client_address, server)[source]¶

Bases: BaseHTTPRequestHandler

agent = None¶

auth_token = None¶

do_POST()[source]¶

do_GET()[source]¶

log_message(format, *args)[source]¶

Log an arbitrary message.

This is used by all other logging functions. Override it if you have specific logging wishes.

The first argument, FORMAT, is a format string for the message to be logged. If the format string contains any % escapes requiring parameters, they should be specified as subsequent arguments (it’s just like printf!).

The client ip and current date/time are prefixed to every message.

Unicode control characters are replaced with escaped hex before writing the output to stderr.

class morie.perseus_relay.PerseusCloudClient(url, token=None)[source]¶

Bases: object

Client for connecting to a remote Perseus relay.

Parameters:

url (str)
token (str | None)

ask(question, *, timeout=120.0)[source]¶

Parameters:

question (str)
timeout (float)

Return type:

health(timeout=5.0)[source]¶

Parameters:: timeout (float)
Return type:: dict[str, Any]

is_available()[source]¶

Return type:: bool

morie.perseus_relay.answer_question(agent, question)[source]¶

Answer through the local tool-calling agent when it works, otherwise through the route chain (your own endpoint, a provider key, Ollama). 503 when nothing answered: a relay must never hand a client “LLM request failed” with status 200.

Parameters:

agent (Any)
question (str)

Return type:

tuple[int, dict[str, Any]]

morie.perseus_relay.serve(port=8421, token=None, bind='127.0.0.1')[source]¶

Serve the Perseus relay over HTTP until interrupted.

Answers POST /v1/percy with the local tool-calling agent when an Ollama server answers, else through the provider chain.

Parameters:

port (int) – TCP port.
token (str | None) – when set, clients must send it as a Bearer token.
bind (str) – address to bind (default loopback only).

morie.perseus_relay.main()[source]¶: Command-line entry of the Perseus relay: parse --port, --token and --bind and serve.

Vertex AI client – Gemini via Google Cloud with service-account auth.

Lightweight pure-httpx path: gets an access token via gcloud auth print-access-token (works when GOOGLE_APPLICATION_CREDENTIALS points at a valid service-account JSON), then POSTs to the Vertex AI REST API.

No dependency on google-cloud-aiplatform or google-genai – those would add ~50 MB of transitive deps. Users who want the full SDK can install it separately; this module gives the small-dep path.

Compatible with Vertex AI Gemini 2.5 Flash / Pro models on us-central1. Configure with:

GOOGLE_APPLICATION_CREDENTIALS – path to service-account JSON

GOOGLE_CLOUD_PROJECT – project id

MORIE_VERTEX_REGION (optional) – default us-central1

class morie.vertex.VertexConfig(project, location='us-central1', model='gemini-2.5-flash', token_ttl_s=3300, gcloud_path='gcloud')[source]¶

Bases: object

Resolved Vertex configuration from env vars.

Parameters:

project (str)
location (str)
model (str)
token_ttl_s (int)
gcloud_path (str)

project: str¶

location: str = 'us-central1'¶

model: str = 'gemini-2.5-flash'¶

token_ttl_s: int = 3300¶

gcloud_path: str = 'gcloud'¶

morie.vertex.resolve_config()[source]¶

Read Vertex config from environment, with sensible defaults.

Return type:: VertexConfig

morie.vertex.ask_gemini(prompt, *, model=None, instructions=None, temperature=0.1, max_output_tokens=2048, timeout_s=120.0, cfg=None)[source]¶

Send a single-turn prompt to Gemini via Vertex AI; return the text.

Parameters:

prompt (str) – The user prompt.
model (str, optional) – Model override. Defaults to VERTEX_MODEL env or gemini-2.5-flash.
instructions (str, optional) – Instructions for the model (persona, constraints), sent as the systemInstruction field of the Gemini REST API.
temperature (float, default 0.1) – Deterministic for scientific work. Match morie.llm default.
max_output_tokens (int, default 2048) – Gemini 2.5 Flash supports up to 8192; 2048 is a sane default.
timeout_s (float, default 120) – HTTP client timeout.
cfg (VertexConfig, optional) – Pre-resolved config. If None, resolve_config() is called.

Returns:

The generated text, trimmed.

Return type:

Raises:

ImportError – If httpx isn’t installed.
RuntimeError – If gcloud auth, project env, or Vertex call fails.

morie.vertex.health_check()[source]¶

Run a tiny smoke test – returns a dict suitable for JSON logging.

Return type:: dict[str, Any]

MORIE Inference Engine – Transformer forward pass with TurboQuant KV-cache.

Minimal LLM inference engine with two backends:

MLX (Apple Silicon GPU via Metal) – requires pip install mlx on Python 3.14. Use .venv-314/ for MLX acceleration.
NumPy (CPU fallback) – works on Python 3.15, any platform.

The engine loads GGUF models via morie.gguf_loader.GGUFModel, compresses the KV-cache with morie.kv_cache.TurboQuantKVCache, and tokenizes with morie.tokenizer.Tokenizer.

Architecture: Llama-family (RMSNorm, RoPE, GQA, SwiGLU FFN) and autoresearch GPT (ReLU², value embeddings, residual lambdas, logit softcap).

References

Karpathy, A. (2023). llama2.c – inference in pure C.
TurboQuant: Zandieh et al. (2026). ICLR 2026. arXiv:2504.19874
MLX: Apple (2023). ml-explore/mlx – array framework for Apple Silicon.

morie.engine.backend()[source]¶

Return the active compute backend: 'mlx' or 'numpy'.

Return type:: str

class morie.engine.GenerationResult(text, tokens, n_tokens=0, time_seconds=0.0, tokens_per_second=0.0, kv_compression_ratio=0.0, backend='numpy')[source]¶

Bases: object

Container for generation output.

Parameters:

text (str)
tokens (list[int])
n_tokens (int)
time_seconds (float)
tokens_per_second (float)
kv_compression_ratio (float)
backend (str)

text: str¶

tokens: list[int]¶

n_tokens: int = 0¶

time_seconds: float = 0.0¶

tokens_per_second: float = 0.0¶

kv_compression_ratio: float = 0.0¶

backend: str = 'numpy'¶

class morie.engine.MORIEEngine(model_path, kv_bits=3, use_kv_compression=True)[source]¶

Bases: object

MORIE’s own LLM inference engine with TurboQuant KV-cache compression.

Parameters:

model_path (str or Path) – Path to a GGUF model file.
kv_bits (int) – TurboQuant quantization bits for KV-cache (2, 3, or 4).
use_kv_compression (bool) – Whether to use TurboQuant KV-cache compression.

Examples

Needs a GGUF model file on disk (the path below is a placeholder):

>>> engine = MORIEEngine("~/.ollama/models/blobs/sha256-abc123", kv_bits=3)
>>> result = engine.generate("The capital of France is", max_tokens=20)
>>> print(result.text)

forward(token_id, position)[source]¶

One transformer forward pass – returns logits.

Parameters:

token_id (int) – Input token ID.
position (int) – Sequence position (for RoPE).

Returns:

Logits over vocabulary (vocab_size,).

Return type:

ndarray

generate(prompt, max_tokens=100, temperature=0.1, top_p=0.9)[source]¶

Generate text from a prompt.

Parameters:

prompt (str) – Input text.
max_tokens (int) – Maximum tokens to generate.
temperature (float) – Sampling temperature (0.1 recommended for science).
top_p (float) – Nucleus sampling threshold.

Return type:

GenerationResult

reset()[source]¶

Clear the KV-cache for a new generation.

Return type:: None

close()[source]¶

Release resources.

Return type:: None

ctypes bridge for MORIE engine C kernels.

Loads engine_kernels.dylib (macOS) or engine_kernels.so (Linux) for Accelerate.framework-backed SIMD operations. Falls back to pure NumPy if the shared library is not available.

Security:

Library path is resolved relative to this file only (no PATH search)
No user-supplied paths, no evaluation/run
All C functions validate inputs and return error codes

morie.engine_bridge.is_available()[source]¶

Return True if the C kernel library is loaded.

Return type:: bool

morie.engine_bridge.rmsnorm(x, weight, eps=1e-05)[source]¶

RMS normalization using C kernel or NumPy fallback.

Parameters:

x (marr)
weight (marr)
eps (float)

Return type:

marr

morie.engine_bridge.rope(q, k, position, head_dim, freq_base=10000.0)[source]¶

Apply RoPE using C kernel or NumPy fallback.

Parameters:

q (marr)
k (marr)
position (int)
head_dim (int)
freq_base (float)

Return type:

tuple[marr, marr]

morie.engine_bridge.matvec(A, x)[source]¶

Matrix-vector multiply using C kernel (Accelerate BLAS) or NumPy.

Parameters:

A (marr)
x (marr)

Return type:

marr

morie.engine_bridge.silu_inplace(x)[source]¶

SiLU activation in-place using C kernel or NumPy.

Parameters:: x (marr)
Return type:: marr

morie.engine_bridge.softmax(x)[source]¶

Softmax using C kernel or NumPy.

Parameters:: x (marr)
Return type:: marr

morie.engine_bridge.argmax(x)[source]¶

Argmax using C kernel or NumPy.

Parameters:: x (marr)
Return type:: int

TurboQuant – data-oblivious vector quantization for MORIE.

Implements the TurboQuant two-stage quantization pipeline from the ICLR 2026 paper (arxiv.org/abs/2504.19874).

What this module IS: A standalone compression library that quantizes arbitrary vectors using rotation + Lloyd-Max scalar quantization + optional QJL 1-bit error correction. Validated against the paper’s theoretical bounds.

What this module is NOT: An inference-time KV-cache optimizer. It does not hook into Ollama, llama.cpp, or HuggingFace transformers attention layers. For runtime KV-cache compression with Ollama, use OLLAMA_KV_CACHE_TYPE=q8_0.

Algorithms¶

Stage 1 – TurboQuant_MSE (Zandieh et al. 2026, arXiv:2504.19874):

Random rotation via QR decomposition (Python) or WHT (C)
Normalize to unit vector, store L2 norm
Scalar-quantize each coordinate via Lloyd-Max codebook optimized for the Beta((d-1)/2, (d-1)/2) distribution

Stage 2 – QJL (Zandieh et al. 2025, arXiv:2406.03482):

Compute residual: r = x - dequant(quant(x))
1-bit sign encoding via random projection: sign(S·r)
Guarantees unbiased inner-product estimation: E[<y, r̂>] = <y, r>

Also includes PolarQuant utilities (Han et al. 2026, arXiv:2502.02617):

polar_transform() and inverse_polar() for recursive Cartesian->Polar, but these are NOT used by turboquant_mse(). They are available for research and comparison.

Theoretical Bounds¶

MSE distortion: D_mse ≤ (√3π / 2) · (1 / 4^b) ≈ 2.72 / 4^b Inner-product distortion: D_prod ≤ (√3π² · ||y||² / d) · (1 / 4^b) Info-theoretic lower: D_mse ≥ 1 / 4^b (gap: ~2.72×)

Validated Results (2026-03-31)¶

3-bit, d=256: MSE=0.028 (bound=0.043), cosine=0.987, compression=5.1× 4-bit, d=256: MSE=0.009 (bound=0.011), cosine=0.996, compression=3.9×

morie.quant.rotation_matrix(d, seed=None)[source]¶

Generate a random orthogonal rotation matrix via QR decomposition.

The rotation randomizes coordinate-wise distributions so that angular components after the polar transform follow a concentrated, predictable distribution – eliminating the need for per-block normalization constants.

Parameters:

d (int) – Dimension of the square rotation matrix.
seed (int, optional) – Random seed for reproducibility.

Returns:

Orthogonal matrix satisfying Π^T · Π = I.

Return type:

ndarray of shape (d, d)

Notes

Uses the QR decomposition of a matrix with i.i.d. N(0,1) entries, with sign correction to ensure a uniform distribution over O(d).

morie.quant.verify_orthogonal(Q, atol=1e-10)[source]¶

Verify that Q is orthogonal: Q^T · Q ≈ I.

Examples

>>> verify_orthogonal(rotation_matrix(8, seed=1))
True

Parameters:

Q (marr)
atol (float)

Return type:

morie.quant.lloyd_max_codebook(d, bits, n_iter=200)[source]¶

Compute optimal Lloyd-Max codebook for Beta-distributed coordinates.

Solves the continuous 1-D k-means problem:: min_{c_1,…,c_K} E[ min_k |X - c_k|² ]

where X ~ Beta((d-1)/2, (d-1)/2) scaled to [-1, 1].

Parameters:

d (int) – Embedding dimension (determines the Beta distribution shape).
bits (int) – Number of quantization bits. K = 2^bits centroids.
n_iter (int) – Max Lloyd iterations.

Returns:

Sorted codebook centroids.

Return type:

ndarray of shape (2^bits,)

morie.quant.get_codebook(d, bits)[source]¶

Get or compute the Lloyd-Max codebook for (d, bits).

Parameters:

d (int)
bits (int)

Return type:

marr

morie.quant.polar_transform(x)[source]¶

Recursive Cartesian->Polar transform in O(d log d).

Parameters:

x (ndarray of shape (d,)) – Input vector (should be rotated first: y = Π · x).

Returns:

radius (float) – ||x||₂
angles (list of ndarray) – angles[ℓ] contains the angles at level ℓ (0-indexed). Level 0: d/2 angles in [0, 2π) (atan2 of adjacent pairs) Level ℓ≥1: d/2^(ℓ+1) angles in [0, π/2] (atan2 of norm ratios)

Return type:

tuple[float, list[marr]]

morie.quant.inverse_polar(radius, angles)[source]¶

Inverse polar transform: reconstruct Cartesian vector from polar.

Parameters:

radius (float) – ||x||₂
angles (list of ndarray) – As returned by polar_transform().

Returns:

Reconstructed vector.

Return type:

ndarray of shape (d,)

morie.quant.quantize_angles(angles, d, bits)[source]¶

Quantize polar angles using Lloyd-Max codebooks.

Parameters:

angles (list of ndarray) – As returned by polar_transform().
d (int) – Original vector dimension.
bits (int or list of int) – Bits per level. If int, same for all levels. Typical: 4 bits for level 0 (full [0,2π) range), 2-3 for higher.

Returns:

Quantized index arrays, one per level.

Return type:

list of ndarray[uint8]

morie.quant.dequantize_angles(indices, d, bits)[source]¶

Dequantize angle indices back to angles.

Parameters:

indices (list of ndarray[uint8]) – As returned by quantize_angles().
d (int) – Original vector dimension.
bits (int or list of int) – Bits per level (must match quantization).

Returns:

Reconstructed angles.

Return type:

list of ndarray

morie.quant.qjl_projection_matrix(d, seed=None)[source]¶

Generate the random projection matrix S for QJL.

Parameters:

d (int) – Dimension (same as the vector dimension).
seed (int, optional) – Random seed.

Returns:

i.i.d. N(0, 1) matrix.

Return type:

ndarray of shape (d, d)

morie.quant.qjl_encode(residual, S)[source]¶

QJL 1-bit sign encoding.

Q_qjl(r) = sign(S · r), stored alongside ||r||₂.

Parameters:

residual (ndarray of shape (d,)) – Quantization residual r = x - dequant(quant(x)).
S (ndarray of shape (d, d)) – Random projection matrix.

Returns:

signs (ndarray of shape (d,) dtype int8) – Sign bits: +1 or -1.
norm (float) – ||r||₂, needed for dequantization.

Return type:

tuple[marr, float]

morie.quant.qjl_decode(signs, norm, S)[source]¶

QJL dequantization – unbiased inner-product estimation.

Q_qjl^{-1}(z) = (√(π/2) / d) · S^T · z

Guarantee: E[<y, Q_qjl^{-1}(Q_qjl(r))>] = <y, r> (unbiased).

Parameters:

signs (ndarray of shape (d,)) – Sign bits from qjl_encode().
norm (float) – ||r||₂ from encoding.
S (ndarray of shape (d, d)) – Same projection matrix used for encoding.

Returns:

Reconstructed residual estimate.

Return type:

ndarray of shape (d,)

class morie.quant.TQBlock(d, bits, radius, angle_indices, qjl_signs=None, qjl_norm=0.0, rotation_seed=42, qjl_seed=None)[source]¶

Bases: object

Storage format for a TurboQuant-compressed vector block.

Variables:

d (int) – Original dimension.
bits (int or list of int) – Bits per level.
radius (float) – ||x||₂ (stored in full precision).
angle_indices (list of ndarray[uint8]) – Quantized angle indices per level.
qjl_signs (ndarray[int8] or None) – QJL error-correction signs (Stage 2 only).
qjl_norm (float) – Residual norm for QJL decode.
rotation_seed (int) – Seed used for the rotation matrix (reproducibility).
qjl_seed (int or None) – Seed used for QJL projection matrix.

Parameters:

d (int)
bits (int | list[int])
radius (float)
angle_indices (list[marr])
qjl_signs (marr | None)
qjl_norm (float)
rotation_seed (int)
qjl_seed (int | None)

d: int¶

bits: int | list[int]¶

radius: float¶

angle_indices: list[marr]¶

qjl_signs: marr | None = None¶

qjl_norm: float = 0.0¶

rotation_seed: int = 42¶

qjl_seed: int | None = None¶

property total_bits: int¶: Total storage bits (excluding metadata).

property compression_ratio: float¶: Compression ratio vs FP16 storage.

morie.quant.turboquant_mse(x, bits=3, rotation_seed=42)[source]¶

TurboQuant MSE-optimal quantization (Stage 1 only).

Per Algorithm 1 (Zandieh et al. 2026, arXiv:2504.19874): 1. Rotate: y = Π · x 2. Normalize to unit vector, store norm 3. Scalar-quantize each coordinate of y_unit via Lloyd-Max codebook 4. Store indices

MSE distortion bound: D_mse ≤ (√3π / 2) · (1 / 4^b).

Parameters:

x (ndarray of shape (d,)) – Input vector. d must be a power of 2.
bits (int) – Quantization bits per coordinate.
rotation_seed (int) – Seed for rotation matrix (must be same for encode/decode).

Returns:

Compressed representation.

Return type:

TQBlock

morie.quant.turboquant_mse_decode(block)[source]¶

Decode a TurboQuant MSE block back to a vector.

Parameters:: block (TQBlock) – Compressed block from turboquant_mse().
Returns:: Reconstructed vector.
Return type:: ndarray of shape (d,)

morie.quant.turboquant_prod(x, bits=3, rotation_seed=42, qjl_seed=137)[source]¶

TurboQuant inner-product-optimal quantization (Stage 1 + QJL).

Applies MSE quantization with (bits-1) bits, then QJL error correction on the residual. Inner-product distortion bound:

D_prod ≤ (√3π² · ||y||² / d) · (1 / 4^b)

Parameters:

x (ndarray of shape (d,)) – Input vector.
bits (int) – Total effective bits (Stage 1 uses bits-1, Stage 2 uses 1 bit).
rotation_seed (int) – Seed for rotation matrix.
qjl_seed (int) – Seed for QJL projection matrix.

Returns:

Compressed representation with QJL error correction.

Return type:

TQBlock

morie.quant.turboquant_prod_decode(block)[source]¶

Decode a TurboQuant inner-product block.

Parameters:: block (TQBlock) – Compressed block from turboquant_prod().
Returns:: Reconstructed vector (MSE reconstruction + QJL residual estimate).
Return type:: ndarray of shape (d,)

morie.quant.compress_kv_cache(keys, values, bits=3, method='mse', rotation_seed=42, qjl_seed=137)[source]¶

Compress attention KV cache using TurboQuant.

Parameters:

keys (ndarray of shape (n_tokens, d)) – Key vectors from attention layer.
values (ndarray of shape (n_tokens, d)) – Value vectors from attention layer.
bits (int) – Quantization bits.
method (str) – "mse" for Stage 1 only, "prod" for Stage 1 + QJL.
rotation_seed (int) – Shared rotation seed for all vectors.
qjl_seed (int) – QJL seed (only for method=”prod”).

Returns:

key_blocks (list of TQBlock)
value_blocks (list of TQBlock)

Return type:

tuple[list[TQBlock], list[TQBlock]]

morie.quant.decompress_kv_cache(key_blocks, value_blocks, method='mse')[source]¶

Decompress KV cache blocks back to arrays.

Parameters:

key_blocks (list of TQBlock)
value_blocks (list of TQBlock)
method (str) – "mse" or "prod" (must match compression).

Returns:

keys (ndarray of shape (n_tokens, d))
values (ndarray of shape (n_tokens, d))

Return type:

tuple[marr, marr]

morie.quant.mse_distortion_bound(bits)[source]¶

Theoretical MSE distortion upper bound: D_mse ≤ (√3π/2) · (1/4^b).

Parameters:: bits (int)
Return type:: float

morie.quant.inner_product_distortion_bound(bits, norm_sq, d)[source]¶

Theoretical inner-product distortion bound.

D_prod ≤ (√3π² · ||y||² / d) · (1/4^b)

Parameters:

bits (int)
norm_sq (float)
d (int)

Return type:

morie.quant.pack_indices(indices, bits)[source]¶

Pack uint8 indices into a compact byte representation.

Parameters:

indices (ndarray of uint8) – Quantization indices, each in [0, 2^bits).
bits (int) – Bits per index.

Returns:

Packed byte string.

Return type:

morie.quant.unpack_indices(data, bits, count)[source]¶

Unpack bit-packed indices back to uint8 array.

Parameters:

data (bytes) – Packed byte string from pack_indices().
bits (int) – Bits per index.
count (int) – Number of indices to unpack.

Return type:

ndarray of uint8

class morie.quant.TQOutlierBlock(bulk, outlier_indices, outlier_values)[source]¶

Bases: object

TurboQuant block with sidecar outliers.

Outliers are stored as (index, fp16-value) pairs; bulk indices are the standard TQBlock layout. Decode is bulk-decode-then-overwrite-outliers.

Parameters:

bulk (TQBlock)
outlier_indices (marr)
outlier_values (marr)

bulk: TQBlock¶

outlier_indices: marr¶

outlier_values: marr¶

morie.quant.turboquant_mse_outlier(x, bits=3, rotation_seed=42, outlier_z=4.0, max_outlier_frac=0.01)[source]¶

Outlier-aware TurboQuant.

Step 1: detect outliers via |x − mean| > outlier_z · std. Step 2: cap outlier count at max_outlier_frac · d (the worst by |x|). Step 3: replace those positions with the median, then run vanilla

turboquant_mse on the cleaned vector.

Step 4: ship the outliers separately as (index, fp16-value).

Outlier overhead at 4-bit, 1% outliers, d=4096: 41 idx (4 B each) + 41 fp16 vals (2 B) = 246 B; bulk is 4096·4/8 = 2048 B + 8 B norm = 2056 B. +12% storage; recovery improves measurably.

Parameters:

x (marr)
bits (int)
rotation_seed (int)
outlier_z (float)
max_outlier_frac (float)

Return type:

TQOutlierBlock

morie.quant.turboquant_mse_outlier_decode(block)[source]¶

Decode an outlier-aware TQ block – bulk decode then overwrite outliers.

Parameters:: block (TQOutlierBlock)
Return type:: marr

Python ctypes bridge for GGML TurboQuant C kernels.

Provides GGMLTurboQuant which loads the compiled shared library and exposes quantize/dequantize/dot_product as Python callables.

The C implementation lives in quant_ggml.h (header) and the eventual quant_ggml.c (to be compiled). Until the C library is built, this module falls back to the pure-NumPy implementation in morie.quant.

Usage¶

>>> from morie.quant_bridge import GGMLTurboQuant
>>> tq = GGMLTurboQuant()
>>> if tq.available:
...     block = tq.quantize(vector, bits=3)
... else:
...     # Falls back to pure Python
...     from morie.quant import turboquant_mse
...     block = turboquant_mse(vector, bits=3)

class morie.quant_bridge.BlockTBQ3[source]¶

Bases: Structure

ctypes mirror of block_tbq3_0 (102 bytes per 256 elements).

indices¶: Structure/Union member

norm¶: Structure/Union member

seed¶: Structure/Union member

class morie.quant_bridge.BlockTBQ4[source]¶

Bases: Structure

ctypes mirror of block_tbq4_0 (134 bytes per 256 elements).

indices¶: Structure/Union member

norm¶: Structure/Union member

seed¶: Structure/Union member

class morie.quant_bridge.BlockTBQ2[source]¶

Bases: Structure

ctypes mirror of block_tbq2_0 (70 bytes per 256 elements).

indices¶: Structure/Union member

norm¶: Structure/Union member

seed¶: Structure/Union member

class morie.quant_bridge.GGMLTurboQuant(lib_path=None)[source]¶

Bases: object

Python interface to the GGML TurboQuant C library.

Falls back to pure-NumPy when the shared library isn’t compiled.

Parameters:: lib_path (str or Path, optional) – Path to the compiled shared library (.dylib / .so). If None, searches in the package directory.

property available: bool¶: True if the C library is loaded and ready.

quantize(vector, bits=3, seed=42)[source]¶

Quantize a vector using the C kernel or NumPy fallback.

Parameters:

vector (ndarray of float32, shape (d,)) – Input vector. d must be 256 (TQ_BLOCK_SIZE).
bits (int) – Quantization bits (2, 3, or 4).
seed (int) – Rotation matrix seed.

Returns:

Compressed block.

Return type:

TQBlock or ctypes Structure

dequantize(block, bits=3)[source]¶

Dequantize a block back to float32 vector.

Parameters:

block (Any)
bits (int)

Return type:

marr

morie.quant_bridge.compile_ggml_lib(output_dir=None, optimize=True)[source]¶

Attempt to compile the GGML TurboQuant C library.

Requires cc (clang or gcc) on the system.

Parameters:

output_dir (str or Path, optional) – Where to write the shared library. Defaults to the morie package dir.
optimize (bool) – Use -O2 -march=native (default True).

Returns:

Path to compiled library, or None on failure.

Return type:

Path or None

TurboQuant-compressed KV cache for MORIE’s inference engine.

Stores attention key/value vectors as compressed TQBlocks instead of raw float tensors, achieving 4-6x memory reduction during inference.

This plugs into morie.engine.MORIEEngine to provide KV-cache compression during actual transformer attention computation.

References

TurboQuant: Zandieh et al. (2026). ICLR 2026. arXiv:2504.19874
QJL: Zandieh et al. (2025). AAAI 2025. arXiv:2406.03482

class morie.kv_cache.CacheStats(compressed_bytes=0, uncompressed_bytes=0, n_tokens=0, n_layers=0)[source]¶

Bases: object

Memory statistics for a TurboQuantKVCache.

Parameters:

compressed_bytes (int)
uncompressed_bytes (int)
n_tokens (int)
n_layers (int)

compressed_bytes: int = 0¶

uncompressed_bytes: int = 0¶

n_tokens: int = 0¶

n_layers: int = 0¶

property compression_ratio: float¶

property savings_mb: float¶

class morie.kv_cache.TurboQuantKVCache(n_layers, head_dim, bits=3, rotation_seed=42)[source]¶

Bases: object

KV cache that stores keys and values as TurboQuant-compressed blocks.

Each key/value vector is quantized via morie.quant.turboquant_mse() on append(), and decompressed on get_keys() / get_values().

Parameters:

n_layers (int) – Number of transformer layers.
head_dim (int) – Dimension per attention head (must be power of 2).
bits (int) – TurboQuant quantization bits (2, 3, or 4).
rotation_seed (int) – Shared rotation seed for reproducibility.

Examples

>>> cache = TurboQuantKVCache(n_layers=32, head_dim=128, bits=3)
>>> k = np.random.randn(128)
>>> v = np.random.randn(128)
>>> cache.append(layer=0, k_vec=k, v_vec=v)
>>> keys = cache.get_keys(0)   # (1, 128) decompressed
>>> values = cache.get_values(0)
>>> round(cache.stats.compression_ratio, 3)   # 3-bit codes plus the scale: 64 / 13
4.923

append(layer, k_vec, v_vec)[source]¶

Compress and cache a new key/value pair for one token.

Parameters:

layer (int) – Transformer layer index.
k_vec (ndarray of shape (head_dim,)) – Key vector (will be quantized).
v_vec (ndarray of shape (head_dim,)) – Value vector (will be quantized).

Return type:

None

get_keys(layer)[source]¶

Decompress all cached keys for a layer.

Return type:: ndarray of shape (seq_len, head_dim)
Parameters:: layer (int)

get_values(layer)[source]¶

Decompress all cached values for a layer.

Return type:: ndarray of shape (seq_len, head_dim)
Parameters:: layer (int)

property seq_len: int¶: Number of tokens currently cached (from layer 0).

clear()[source]¶

Clear all cached blocks.

Return type:: None

property stats: CacheStats¶: Compute memory statistics.

class morie.kv_cache.UncompressedKVCache(n_layers, head_dim)[source]¶

Bases: object

Baseline uncompressed KV cache for comparison benchmarks.

Parameters:

n_layers (int)
head_dim (int)

append(layer, k_vec, v_vec)[source]¶

Parameters:

layer (int)
k_vec (marr)
v_vec (marr)

Return type:

None

get_keys(layer)[source]¶

Parameters:: layer (int)
Return type:: marr

get_values(layer)[source]¶

Parameters:: layer (int)
Return type:: marr

property seq_len: int¶

clear()[source]¶

Return type:: None

property memory_bytes: int¶

GGUF model file loader for MORIE’s inference engine.

Parses GGUF v3 files (the format used by Ollama and llama.cpp) and provides access to model metadata and weight tensors.

References

GGUF spec: https://github.com/ggml-org/ggml/blob/master/docs/gguf.md
ligguf (reference): https://github.com/matrixsmaster/ligguf

morie.gguf_loader.find_local_gguf(root=None)[source]¶

The first GGUF model file on this machine, or None.

Looks in Ollama’s model store ($OLLAMA_MODELS, else ~/.ollama/models) for a blob that starts with the GGUF magic, largest first (the weights, not a projector or adapter).

Examples

>>> import tempfile
>>> d = Path(tempfile.mkdtemp())
>>> (d / "blobs").mkdir()
>>> _ = (d / "blobs" / "sha256-a").write_bytes(b"GGUF" + bytes(12))
>>> _ = (d / "blobs" / "sha256-b").write_bytes(b"{}")
>>> find_local_gguf(d).name
'sha256-a'
>>> find_local_gguf(d / "nowhere") is None
True

Parameters:: root (str | Path | None)
Return type:: Path | None

class morie.gguf_loader.TensorInfo(name, shape, dtype, offset, n_elements=0)[source]¶

Bases: object

Metadata for one tensor in the GGUF file.

Parameters:

name (str)
shape (tuple[int, ...])
dtype (int)
offset (int)
n_elements (int)

name: str¶

shape: tuple[int, ...]¶

dtype: int¶

offset: int¶

n_elements: int = 0¶

property dtype_name: str¶

class morie.gguf_loader.GGUFModel(path)[source]¶

Bases: object

Load and parse a GGUF model file.

Parameters:: path (str or Path) – Path to the GGUF file.

Examples

Needs a GGUF model file on disk (the path below is a placeholder):

>>> model = GGUFModel("~/.ollama/models/blobs/sha256-abc123")
>>> print(model.config)
{'architecture': 'llama', 'n_layers': 32, ...}
>>> names = model.tensor_names()
>>> weight = model.get_tensor("token_embd.weight")

property config: dict[str, Any]¶: Extract model configuration from metadata.

tensor_names()[source]¶

List all tensor names.

Return type:: list[str]

tensor_info(name)[source]¶

Get metadata for a tensor.

Parameters:: name (str)
Return type:: TensorInfo

get_tensor_raw(name)[source]¶

Get raw bytes for a tensor (quantized).

Parameters:: name (str)
Return type:: bytes

get_tensor(name)[source]¶

Load and dequantize a tensor to float32.

Currently supports F32, F16, and Q8_0. Other quantized types return raw bytes as a flat uint8 array (for future dequantization).

Parameters:: name (str)
Return type:: marr

close()[source]¶

Return type:: None

Convert autoresearch PyTorch checkpoints to GGUF format.

Bridges autoresearch’s train.py GPT architecture to MORIE’s inference engine. Supports optional TurboQuant weight compression during conversion.

TRUST MODEL – what this module can do to your machine¶

This converter deserializes two files: a PyTorch .pt checkpoint and a tokenizer.pkl. Both formats can, in the general case, run arbitrary code when loaded. This module defends against that by default:

the checkpoint is read by morie._pt_reader under the same allowlist contract as weights_only=True (no code objects), and ONLY when MORIE_TRUST_CHECKPOINT=1 and logs a warning when it does;
tokenizer.pkl is read through a restricted unpickler that only permits tiktoken/stdlib-container globals (see _TokenizerUnpickler).

Use this module ONLY with checkpoint and tokenizer files you produced yourself or obtained from a source you fully trust.

Usage:: python -m morie.pt2gguf –checkpoint model_baseline.pt –output morie_model.gguf python -m morie.pt2gguf –checkpoint model_baseline.pt –output morie_3bit.gguf –turbo-bits 3

morie.pt2gguf.convert(checkpoint_path, output_path, tokenizer_dir=None, turbo_bits=0)[source]¶

Convert a PyTorch checkpoint to a GGUF file.

The checkpoint is read only when MORIE_TRUST_CHECKPOINT=1: unpickling an untrusted file can run code, so the gate is checked before the file is opened.

Parameters:

checkpoint_path – the .pt zip checkpoint.
output_path – the GGUF file to write.
tokenizer_dir – directory with the tokenizer files, if any.
turbo_bits – quantisation bits for the TurboQuant path (0 keeps full precision).

Returns:

The path written.

Tokenizer for MORIE’s inference engine.

Loads tokenization data from GGUF model metadata (the tokenizer.ggml.* keys) and provides encode/decode without requiring SentencePiece at runtime.

Falls back to SentencePiece if a .model file is provided explicitly.

class morie.tokenizer.Tokenizer(model_path=None, gguf_model=None)[source]¶

Bases: object

BPE / SentencePiece tokenizer loaded from GGUF metadata.

Parameters:

model_path (str or Path, optional) – Path to a GGUF file (reads tokenizer.ggml.* metadata) or a SentencePiece .model file.
gguf_model (GGUFModel, optional) – An already-loaded morie.gguf_loader.GGUFModel instance.

Examples

Needs a GGUF model file on disk (the path below is a placeholder):

>>> from morie.gguf_loader import GGUFModel
>>> model = GGUFModel("path/to/model.gguf")
>>> tok = Tokenizer(gguf_model=model)
>>> ids = tok.encode("Hello world")
>>> tok.decode(ids)
'Hello world'

encode(text, *, add_bos=True)[source]¶

Encode text to token IDs.

Uses SentencePiece if available, otherwise greedy BPE matching.

Parameters:

text (str)
add_bos (bool)

Return type:

decode(ids)[source]¶

Decode token IDs back to text.

Parameters:: ids (list[int])
Return type:: str

property vocab_size: int¶: Total vocabulary size.

property eos_id: int¶: End-of-sequence token ID.

property bos_id: int¶: Beginning-of-sequence token ID.

Headless polyglot REPL engine – multi-language execution with variable bridging.

Supports: Python, R, Shell (bash/zsh), Julia, SQL (SQLite/PostgreSQL), Node.js,: Go, Rust, C, C++, OCaml, Lua, TypeScript, LaTeX

Variable bridging: automatic bidirectional across all active languages.

TRUST MODEL – what this module can do to your machine¶

This is a REPL: it runs whatever code the LOCAL USER types into their own session, in whichever language they select – the same trust model as python/R/bash themselves. It compiles/executes source and shells out to language toolchains. It does NOT execute remote or network-supplied code. Every language path funnels through execute(), which honours MORIE_NO_EXEC=1 to disable all execution (set it on shared/CI machines). Only feed this engine code you would be willing to run at your own shell.

morie.polyglot.detect_language(code, default='python')[source]¶

Parameters:

code (str)
default (str)

Return type:

class morie.polyglot.ExecResult(language: 'str', stdout: 'str' = '', stderr: 'str' = '', success: 'bool' = True, variables: 'dict' = <factory>)[source]¶

Bases: object

Parameters:

language (str)
stdout (str)
stderr (str)
success (bool)
variables (dict)

language: str¶

stdout: str = ''¶

stderr: str = ''¶

success: bool = True¶

variables: dict¶

class morie.polyglot.PolyglotEngine(polyglot=True, auto_detect=True, output_fn=None)[source]¶

Bases: object

Multi-language REPL engine with variable bridging.

Parameters:

polyglot (bool)
auto_detect (bool)
output_fn (Callable[[str], None] | None)

strip_prefix(code)[source]¶

Parameters:: code (str)
Return type:: tuple[str, str | None]

execute(code)[source]¶

Parameters:: code (str)
Return type:: ExecResult

close()[source]¶

Return type:: None

available_languages()[source]¶

Return type:: dict[str, bool]

morie.polyglot.run_headless_repl(polyglot=True, auto_detect=True, lang='python')[source]¶

Parameters:

polyglot (bool)
auto_detect (bool)
lang (str)

Return type:

Runner and environment¶

CLI entrypoints for MORIE package workflows and assistant mode.

Provides the morie command with subcommands for running analysis modules, querying the LLM agent, and checking environment health.

morie.runner.execute_pipeline(modules=None, *, cpads_csv='/home/runner/work/morie/morie/src/morie/data/cpads_synthetic.csv', dataset_key=None, output_dir=None, silent=False, track_carbon=True)[source]¶

Run the specified epidemiologic analysis modules.

Parameters:

modules (list[str], optional) – A list of module names to run, defaults to None.
silent (bool, optional) – If True, skips the safety confirmation prompt, defaults to False.
track_carbon (bool, optional) – If True, track CO₂ emissions with the vendored morie.emissions tracker.
cpads_csv (str)
dataset_key (str | None)
output_dir (str | None)

Return type:

morie.runner.build_parser()[source]¶

Create the CLI argument parser.

Return type:: ArgumentParser

morie.runner.main()[source]¶

Entry point for the MORIE command line interface.

Return type:: int

Module-level run surface for MORIE dataset analyses.

morie.modules.proportion_effectsize(prop1, prop2)[source]¶: Cohen’s h = 2 asin(sqrt(p1)) - 2 asin(sqrt(p2)) (statsmodels-free).

class morie.modules.ModuleSpec(name: 'str', description: 'str', output_files: 'tuple[str, ...]')[source]¶

Bases: object

Parameters:

name (str)
description (str)
output_files (tuple[str, ...])

name: str¶

description: str¶

output_files: tuple[str, ...]¶

morie.modules.list_modules()[source]¶

Return the currently implemented CPADS module surface.

Return type:: list[dict[str, object]]

morie.modules.load_cpads_analysis_data(cpads_csv='/home/runner/work/morie/morie/src/morie/data/cpads_synthetic.csv', *, column_mapping=None, auto_map=False)[source]¶

Load and canonicalize a CPADS CSV into MORIE analysis columns.

When the caller does not pass an explicit cpads_csv and the real Statistics Canada PUMF file is not on disk, this falls back to the bundled 1,200-row synthetic frame and emits a warning. Outputs produced from the synthetic frame are useful for testing and demo purposes only; do not interpret them as findings about the real Canadian population.

Schema-agnostic mode¶

Pass column_mapping={"my_wt": "weight", "drinks_yn": "alcohol_past12m", ...} to rename your dataset’s columns into morie’s canonical names BEFORE the CPADS contract validator runs. Pass auto_map=True to have morie infer the mapping with a fuzzy match against a synonym table (morie.schema.infer_mapping); a warning will surface the inferred mapping so you can review before trusting it.

Parameters:

cpads_csv (str | Path)
column_mapping (dict[str, str] | None)
auto_map (bool)

Return type:

DataFrame

morie.modules.run_power_design_module(cpads_csv='/home/runner/work/morie/morie/src/morie/data/cpads_synthetic.csv', *, output_dir=None)[source]¶

Build real-data power summaries from the CPADS CSV.

Parameters:

cpads_csv (str | Path)
output_dir (str | Path | None)

Return type:

dict[str, DataFrame]

morie.modules.r_route_problem(module_name, exc)[source]¶

One line for the terminal when an R-backed module cannot run.

Parameters:

module_name (str)
exc (BaseException)

Return type:

morie.modules.run_module(module_name, *, cpads_csv='/home/runner/work/morie/morie/src/morie/data/cpads_synthetic.csv', dataset_key=None, output_dir=None)[source]¶

Run one implemented MORIE module.

Parameters:

module_name (str) – Name from MODULE_SPECS (e.g. “power-design”).
cpads_csv (str or Path) – Legacy: direct path to CPADS CSV.
dataset_key (str, optional) – Short key from DATASET_CATALOG. When provided, loads from DB instead of the CSV path. Default: None (uses cpads_csv).
output_dir (str or Path, optional) – Directory for CSV outputs.

Return type:

dict[str, object]

morie.modules.run_modules(module_names=None, *, cpads_csv='/home/runner/work/morie/morie/src/morie/data/cpads_synthetic.csv', dataset_key=None, output_dir=None)[source]¶

Run multiple implemented modules and return their output tables.

Parameters:

module_names (list[str] | None)
cpads_csv (str | Path)
dataset_key (str | None)
output_dir (str | Path | None)

Return type:

dict[str, dict[str, DataFrame]]

MORIE environment self-diagnostics.

morie doctor checks that all required components are installed and reachable before the user runs analysis modules. Each check prints a pass/fail row; the overall exit code is 0 only if all required checks pass.

Uses rich for formatted terminal output when available, with a plain text fallback for minimal environments.

morie.doctor.run_checks()[source]¶

Run all diagnostics and return a structured results dict.

Return type:: dict[str, Any]

morie.doctor.run_doctor(fix=False)[source]¶

Run diagnostics and print a summary table.

Parameters:: fix (bool) – When True (morie doctor --fix), attempt to remediate failed checks – install missing Python packages, create the cache directory – then re-run and re-render the diagnostics.
Returns:: 0 if all required checks pass, 1 otherwise.
Return type:: int

MORIE self-test – comprehensive smoke test of all subsystems.

Verifies that every major component of the MORIE IDE is functional without requiring pytest or any test framework. Designed to give users and developers confidence that the software works.

Run with:

morie selftest

morie.selftest.run_selftest()[source]¶

Run all self-tests and print results.

Return type:: int

Pure-Python emissions tracker – drop-in replacement for codecarbon.

Replicates the CodeCarbon EmissionsTracker methodology without pydantic or any Rust dependencies, enabling Python 3.15+ support.

Methodology (matches codecarbon 3.2.x):

CPU/GPU power: powermetrics on Apple Silicon, /proc/stat TDP on Linux
RAM power: heuristic based on DIMM count (1.5 W/DIMM ARM, 5 W/DIMM x86)
Carbon intensity: per-country energy mix -> weighted gCO2/kWh
Emissions: energy_kWh × carbon_intensity_kgCO2_per_kWh × PUE
Water: energy_kWh × WUE

References

CodeCarbon: https://github.com/mlco2/codecarbon (MIT license)
Global energy mix data: IEA / Our World in Data

class morie.emissions.EmissionsData(timestamp='', project_name='morie', run_id='', experiment_id='', duration=0.0, emissions=0.0, emissions_rate=0.0, cpu_power=0.0, gpu_power=0.0, ram_power=0.0, cpu_energy=0.0, gpu_energy=0.0, ram_energy=0.0, energy_consumed=0.0, water_consumed=0.0, country_name='', country_iso_code='', region='', os_info='', python_version='', tracker_version='morie-1.0.0', cpu_count=0, cpu_model='', gpu_count=0, gpu_model='', longitude=0.0, latitude=0.0, ram_total_size=0.0, tracking_mode='machine', cpu_utilization_percent=0.0, gpu_utilization_percent=0.0, ram_utilization_percent=0.0, ram_used_gb=0.0, on_cloud='N', pue=1.0, wue=0.0, cloud_provider='', cloud_region='')[source]¶

Bases: object

All tracked emissions data for a single run.

Parameters:

timestamp (str)
project_name (str)
run_id (str)
experiment_id (str)
duration (float)
emissions (float)
emissions_rate (float)
cpu_power (float)
gpu_power (float)
ram_power (float)
cpu_energy (float)
gpu_energy (float)
ram_energy (float)
energy_consumed (float)
water_consumed (float)
country_name (str)
country_iso_code (str)
region (str)
os_info (str)
python_version (str)
tracker_version (str)
cpu_count (int)
cpu_model (str)
gpu_count (int)
gpu_model (str)
longitude (float)
latitude (float)
ram_total_size (float)
tracking_mode (str)
cpu_utilization_percent (float)
gpu_utilization_percent (float)
ram_utilization_percent (float)
ram_used_gb (float)
on_cloud (str)
pue (float)
wue (float)
cloud_provider (str)
cloud_region (str)

timestamp: str = ''¶

project_name: str = 'morie'¶

run_id: str = ''¶

experiment_id: str = ''¶

duration: float = 0.0¶

emissions: float = 0.0¶

emissions_rate: float = 0.0¶

cpu_power: float = 0.0¶

gpu_power: float = 0.0¶

ram_power: float = 0.0¶

cpu_energy: float = 0.0¶

gpu_energy: float = 0.0¶

ram_energy: float = 0.0¶

energy_consumed: float = 0.0¶

water_consumed: float = 0.0¶

country_name: str = ''¶

country_iso_code: str = ''¶

region: str = ''¶

os_info: str = ''¶

python_version: str = ''¶

tracker_version: str = 'morie-1.0.0'¶

cpu_count: int = 0¶

cpu_model: str = ''¶

gpu_count: int = 0¶

gpu_model: str = ''¶

longitude: float = 0.0¶

latitude: float = 0.0¶

ram_total_size: float = 0.0¶

tracking_mode: str = 'machine'¶

cpu_utilization_percent: float = 0.0¶

gpu_utilization_percent: float = 0.0¶

ram_utilization_percent: float = 0.0¶

ram_used_gb: float = 0.0¶

on_cloud: str = 'N'¶

pue: float = 1.0¶

wue: float = 0.0¶

cloud_provider: str = ''¶

cloud_region: str = ''¶

property csv_header: str¶

csv_row()[source]¶

Return type:: str

morie.emissions.iso3(code)[source]¶

"fr" -> "FRA"; an ISO-3 code (or anything else) comes back upper-cased. Every two-letter code of ISO 3166 is known, not a hand-picked subset.

Examples

>>> iso3("fr"), iso3("CAN"), iso3("kz")
('FRA', 'CAN', 'KAZ')

Parameters:: code (str)
Return type:: str

morie.emissions.known_country_codes()[source]¶

ISO-3 codes the energy-mix table knows (empty when the table is not bundled).

Return type:: set[str]

class morie.emissions.EmissionsTracker(project_name='morie', output_dir='.', output_file='emissions.csv', measure_power_secs=15.0, tracking_mode='machine', pue=1.0, wue=0.0, log_level='warning', save_to_file=True, save_to_logger=False, country_iso_code='', region='', capsule=True, **kwargs)[source]¶

Bases: object

Pure-Python emissions tracker compatible with codecarbon’s API.

Usage:

tracker = EmissionsTracker(project_name="my-analysis")
tracker.start()
# ... run computation ...
emissions_kg = tracker.stop()

Context manager:

with EmissionsTracker() as tracker:
    ...  # tracked

Parameters:

project_name (str)
output_dir (str)
output_file (str)
measure_power_secs (float)
tracking_mode (str)
pue (float)
wue (float)
log_level (str)
save_to_file (bool)
save_to_logger (bool)
country_iso_code (str)
region (str)
capsule (bool)
kwargs (Any)

capsule: dict[str, Any] | None¶

start()[source]¶

Return type:: EmissionsTracker

stop()[source]¶

Stop tracking and return total emissions in kg CO2.

Return type:: float

morie.emissions.write_capsule(output_dir, data, csv_name='emissions.csv', *, carbon_intensity=0.0)[source]¶

Seal an emissions run in a capsule next to its CSV.

Writes emissions_manifest.json: a bricklayer manifest (meta with the measurements, the method and its sources, results empty, environment with the interpreter and platform) that rmoriebricklayer::fromjs() reads back as a manifest. When Rscript and the rmoriebricklayer package are available the manifest and the CSV are then bundled and signed with a per-run ML-DSA-44 key into capsule_bundle.json (rmoriebricklayer::cpsbndl()), which cpsbndlvrf() or rmorie::morie_emissions_verify() checks: a verifier learns the two files have not changed since sealing. The bundle lists only those two files, so cpsbndlvrf() fails its no_unlisted_files row in a directory that holds anything else (another run, say): give each run its own output_dir. Without R the manifest alone is written and signed is False – said so, never pretended.

Parameters:

output_dir (Path | str)
data (EmissionsData)
csv_name (str)
carbon_intensity (float)

Return type:

morie.emissions.run_check(seconds=3.0, output_dir='emissions', *, capsule=True, country_iso_code='')[source]¶

Busy-loop for seconds under the tracker and return the row (the morie emissions verb).

Parameters:

seconds (float)
output_dir (str)
capsule (bool)
country_iso_code (str)

Return type:

EmissionsData

morie.emissions.summary_text(data, capsule)[source]¶

Four-line plain-text summary of a tracked run’s emissions.

Lists kg CO2eq and duration, energy (CPU and RAM), the CPU and its utilisation, and the carbon intensity used (the country set by MORIE_COUNTRY_ISO, else the world average). A provenance capsule, when given, adds a line naming its manifest and whether it was signed.

Parameters:

data (EmissionsData) – the measured EmissionsData.
capsule (dict[str, Any] | None) – the capsule record from writing the run’s provenance, or None.

Returns:

The summary, lines joined by newlines.

Return type:

Docker container management for MORIE.

Provides Python wrappers around Docker and Docker Compose CLI commands for building, running, inspecting, and managing MORIE analysis containers. All functions shell out to the docker / docker compose binaries and add structured output, progress display via rich, and robust error handling.

The container workflow supports:

Building reproducible analysis images from the current environment.
Running MORIE modules inside containers with volume-mounted data.
Inspecting container contents and verifying environment health.
CI simulation (run the full test suite inside a container).
Resource monitoring during pipeline run.
Ollama sidecar networking for local LLM integration.

References

Docker Engine API:: https://docs.docker.com/engine/api/

Boettiger, C. (2015). An introduction to Docker for reproducible research. ACM SIGOPS Operating Systems Review, 49(1), 71–79. https://doi.org/10.1145/2723872.2723882

class morie.container.DockerResult(command, return_code, stdout, stderr, duration_seconds)[source]¶

Bases: object

Structured result from a Docker command.

Parameters:

command (str)
return_code (int)
stdout (str)
stderr (str)
duration_seconds (float)

command: str¶

return_code: int¶

stdout: str¶

stderr: str¶

duration_seconds: float¶

property success: bool¶: True if the command exited with code 0.

class morie.container.ContainerInfo(container_id, name, image, status, created, ports, size='')[source]¶

Bases: object

Metadata for a running or stopped MORIE container.

Parameters:

container_id (str)
name (str)
image (str)
status (str)
created (str)
ports (str)
size (str)

container_id: str¶

name: str¶

image: str¶

status: str¶

created: str¶

ports: str¶

size: str = ''¶

class morie.container.ImageInfo(image_id, repository, tag, size, created)[source]¶

Bases: object

Metadata for an MORIE Docker image.

Parameters:

image_id (str)
repository (str)
tag (str)
size (str)
created (str)

image_id: str¶

repository: str¶

tag: str¶

size: str¶

created: str¶

class morie.container.HealthCheckResult(python_ok, python_version, r_ok, r_version, morie_importable, morie_version, data_accessible, data_path, checks=<factory>)[source]¶

Bases: object

Result of a container health check.

Parameters:

python_ok (bool)
python_version (str)
r_ok (bool)
r_version (str)
morie_importable (bool)
morie_version (str)
data_accessible (bool)
data_path (str)
checks (list[dict[str, Any]])

python_ok: bool¶

python_version: str¶

r_ok: bool¶

r_version: str¶

morie_importable: bool¶

morie_version: str¶

data_accessible: bool¶

data_path: str¶

checks: list[dict[str, Any]]¶

property all_ok: bool¶: True if all health checks passed.

class morie.container.ResourceSnapshot(timestamp, container_id, cpu_percent, memory_usage_mb, memory_limit_mb, memory_percent, net_io_rx_mb, net_io_tx_mb)[source]¶

Bases: object

CPU and memory snapshot from a container.

Parameters:

timestamp (str)
container_id (str)
cpu_percent (float)
memory_usage_mb (float)
memory_limit_mb (float)
memory_percent (float)
net_io_rx_mb (float)
net_io_tx_mb (float)

timestamp: str¶

container_id: str¶

cpu_percent: float¶

memory_usage_mb: float¶

memory_limit_mb: float¶

memory_percent: float¶

net_io_rx_mb: float¶

net_io_tx_mb: float¶

class morie.container.EnvironmentDiff(host_python, container_python, host_r, container_r, host_os, container_os, package_diffs=<factory>, missing_on_host=<factory>, missing_in_container=<factory>)[source]¶

Bases: object

Differences between container and host environments.

Parameters:

host_python (str)
container_python (str)
host_r (str)
container_r (str)
host_os (str)
container_os (str)
package_diffs (list[dict[str, str]])
missing_on_host (list[str])
missing_in_container (list[str])

host_python: str¶

container_python: str¶

host_r: str¶

container_r: str¶

host_os: str¶

container_os: str¶

package_diffs: list[dict[str, str]]¶

missing_on_host: list[str]¶

missing_in_container: list[str]¶

morie.container.docker_available()[source]¶

Check whether the Docker CLI is accessible and the daemon is running.

Returns:: True if docker info succeeds.
Return type:: bool

morie.container.build_image(context_dir='.', *, image_name='morie', tag='latest', dockerfile=None, build_args=None, no_cache=False, target=None, platform_arch=None, timeout=1200)[source]¶

Build the MORIE Docker image.

Wraps docker build with sensible defaults for the MORIE project.

Parameters:

context_dir (str | Path) – Build context directory (default: current directory).
image_name (str) – Image repository name (default morie).
tag (str) – Image tag (default latest).
dockerfile (str | Path | None) – Path to Dockerfile (default: context_dir/Dockerfile).
build_args (dict[str, str] | None) – --build-arg key=value pairs.
no_cache (bool) – If True, pass --no-cache.
target (str | None) – Multi-stage build target stage name.
platform_arch (str | None) – Target platform (e.g. linux/amd64).
timeout (int) – Build timeout in seconds (default 1200).

Return type:

morie.container.build_multistage(context_dir='.', *, image_name='morie', stages=None, final_tag='latest', timeout=1800)[source]¶

Orchestrate a multi-stage Docker build.

Builds each named stage separately (useful for caching and debugging) and then builds the final image.

Parameters:

context_dir (str | Path) – Build context.
image_name (str) – Base image name.
stages (list[str] | None) – Stage names to build individually before the final build. If None, only the final build is run.
final_tag (str) – Tag for the final image.
timeout (int) – Per-stage timeout in seconds.

Returns:

One result per stage plus the final build.

Return type:

list[DockerResult]

morie.container.list_images(*, filter_morie=True)[source]¶

List Docker images, optionally filtered to MORIE images.

Parameters:: filter_morie (bool) – If True, only return images with repository matching MORIE_IMAGE_NAME.
Return type:: list[ImageInfo]

morie.container.remove_image(image, *, force=False)[source]¶

Remove a Docker image.

Parameters:

image (str) – Image name or ID.
force (bool) – If True, force removal.

Return type:

morie.container.cleanup_images(*, keep_latest=True, max_age_days=30)[source]¶

Remove old MORIE images.

Parameters:

keep_latest (bool) – If True, keep the latest tagged image.
max_age_days (int) – Remove images older than this many days.

Returns:

One result per removed image.

Return type:

list[DockerResult]

morie.container.push_image(image='morie:latest', *, registry=None)[source]¶

Push an image to a registry.

Parameters:

image (str) – Local image tag.
registry (str | None) – Registry prefix (e.g. ghcr.io/rootcoder007). If provided, the image is tagged with the registry prefix before pushing.

Return type:

morie.container.pull_image(image='morie:latest', *, registry=None)[source]¶

Pull an image from a registry.

Parameters:

image (str) – Image tag.
registry (str | None) – Registry prefix.

Return type:

morie.container.run_container(*, image='morie:latest', name=None, data_dir=None, output_dir=None, config_dir=None, command=None, env_vars=None, ports=None, detach=False, remove=True, network=None, cpus=None, memory=None, timeout=3600)[source]¶

Run an MORIE container.

Parameters:

image (str) – Docker image to run.
name (str | None) – Container name. Auto-generated if None.
data_dir (str | Path | None) – Host directory to mount at /morie/data.
output_dir (str | Path | None) – Host directory to mount at /morie/output.
config_dir (str | Path | None) – Host directory to mount at /morie/config.
command (str | None) – Override the default CMD.
env_vars (dict[str, str] | None) – Environment variables to set in the container.
ports (dict[int, int] | None) – Port mappings {host_port: container_port}.
detach (bool) – If True, run in background.
remove (bool) – If True, auto-remove container on exit.
network (str | None) – Docker network to attach to.
cpus (float | None) – CPU limit (e.g. 2.0 for two cores).
memory (str | None) – Memory limit (e.g. 4g).
timeout (int) – Timeout in seconds (default 3600).

Return type:

morie.container.exec_in_container(container, command, *, workdir=None, user=None, env_vars=None, interactive=False, timeout=300)[source]¶

Run a command inside a running container.

Parameters:

container (str) – Container name or ID.
command (str) – Command to run.
workdir (str | None) – Working directory inside the container.
user (str | None) – User to run as (e.g. root).
env_vars (dict[str, str] | None) – Additional environment variables.
interactive (bool) – If True, pass -it flags.
timeout (int) – Timeout in seconds.

Return type:

morie.container.container_shell(container, *, program='/bin/bash')[source]¶

Open an interactive shell in a running container.

Parameters:

container (str) – Container name or ID.
program (str) – The shell program to start (default /bin/bash).

Return type:

morie.container.stop_container(container, *, timeout_seconds=10)[source]¶

Stop a running container gracefully.

Parameters:

container (str) – Container name or ID.
timeout_seconds (int) – Grace period before SIGKILL.

Return type:

morie.container.remove_container(container, *, force=False, remove_volumes=False)[source]¶

Remove a container.

Parameters:

container (str) – Container name or ID.
force (bool) – If True, force removal of running container.
remove_volumes (bool) – If True, also remove associated volumes.

Return type:

morie.container.list_containers(*, all_states=True, filter_morie=True)[source]¶

List MORIE containers.

Parameters:

all_states (bool) – Include stopped containers.
filter_morie (bool) – Only show containers with names starting with the MORIE prefix.

Return type:

list[ContainerInfo]

morie.container.cleanup_containers(*, remove_stopped=True, remove_running=False)[source]¶

Remove old MORIE containers.

Parameters:

remove_stopped (bool) – Remove containers that have exited.
remove_running (bool) – Force-remove running containers.

Return type:

list[DockerResult]

morie.container.inspect_container(container, *, list_files=True, check_versions=True)[source]¶

Inspect a container: list files, check installed versions.

Parameters:

container (str) – Container name or ID.
list_files (bool) – List files in the MORIE working directory.
check_versions (bool) – Check Python, R, and key package versions.

Returns:

Inspection results with keys files, python_version, r_version, packages.

Return type:

dict[str, Any]

morie.container.health_check(container, *, data_path='/morie/data')[source]¶

Run health checks on a container.

Verifies that Python, R, and the morie package are importable and that the data directory is accessible.

Parameters:

container (str) – Container name or ID.
data_path (str) – Expected data mount path inside the container.

Return type:

HealthCheckResult

morie.container.environment_diff(container)[source]¶

Compare container environment against the host.

Checks Python version, R version, OS, and installed Python packages on both host and container, then reports differences.

Parameters:: container (str) – Container name or ID.
Return type:: EnvironmentDiff

morie.container.verify_pipeline(container, *, modules=None, cpads_csv=None, output_dir='/morie/output', timeout=3600)[source]¶

Run MORIE modules inside a container and verify outputs.

Parameters:

container (str) – Container name or ID.
modules (list[str] | None) – Module names to run. If None, runs all.
cpads_csv (str | None) – Path to CPADS CSV inside the container.
output_dir (str) – Output directory inside the container.
timeout (int) – Per-module timeout in seconds.

Returns:

Keys: results (per-module status), all_passed (bool), duration_seconds (float).

Return type:

dict[str, Any]

morie.container.run_ci_simulation(*, image='morie:latest', test_command='python -m pytest -q --tb=short', data_dir=None, timeout=1800)[source]¶

Simulate CI by running the test suite inside a container.

Parameters:

image (str) – Docker image to use.
test_command (str) – Test command to run.
data_dir (str | Path | None) – Host data directory to mount.
timeout (int) – Test timeout in seconds.

Return type:

morie.container.export_results(container, container_path, host_path)[source]¶

Copy files from a container to the host.

Parameters:

container (str) – Container name or ID.
container_path (str) – Source path inside the container.
host_path (str | Path) – Destination path on the host.

Return type:

morie.container.import_data(container, host_path, container_path='/morie/data')[source]¶

Copy data files from the host into a container.

Parameters:

container (str) – Container name or ID.
host_path (str | Path) – Source path on the host.
container_path (str) – Destination path inside the container.

Return type:

morie.container.snapshot_container(container, *, tag=None, message=None)[source]¶

Commit the current state of a container as a new image.

Parameters:

container (str) – Container name or ID.
tag (str | None) – Tag for the new image. Auto-generated if None.
message (str | None) – Commit message.

Return type:

morie.container.get_resource_stats(container)[source]¶

Get a single resource usage snapshot for a container.

Parameters:: container (str) – Container name or ID.
Returns:: Snapshot of CPU, memory, and network usage. None if the container is not running or stats cannot be read.
Return type:: ResourceSnapshot | None

morie.container.monitor_resources(container, *, duration_seconds=60, interval_seconds=2.0)[source]¶

Monitor container resource usage over time.

Parameters:

container (str) – Container name or ID.
duration_seconds (int) – Total monitoring duration.
interval_seconds (float) – Seconds between samples.

Returns:

Time series of resource snapshots.

Return type:

list[ResourceSnapshot]

morie.container.get_container_logs(container, *, tail=None, since=None, follow=False, timestamps=True)[source]¶

Retrieve logs from a container.

Parameters:

container (str) – Container name or ID.
tail (int | None) – Only return the last N lines.
since (str | None) – Show logs since a timestamp or duration (e.g. 10m, 2024-01-01).
follow (bool) – If True, follow log output (blocking).
timestamps (bool) – If True, include timestamps.

Return type:

morie.container.render_container_logs(container, *, tail=100)[source]¶

Display container logs with rich formatting.

Parameters:

container (str) – Container name or ID.
tail (int) – Number of lines to display.

Return type:

None

morie.container.compose_up(*, compose_file=None, services=None, detach=True, build=False, timeout=300)[source]¶

Start services defined in docker-compose.yml.

Parameters:

compose_file (str | Path | None) – Compose file path.
services (list[str] | None) – Specific services to start. If None, starts all.
detach (bool) – Run in background.
build (bool) – Rebuild images before starting.
timeout (int) – Startup timeout.

Return type:

morie.container.compose_down(*, compose_file=None, remove_volumes=False, remove_images=None, timeout=120)[source]¶

Stop and remove compose services.

Parameters:

compose_file (str | Path | None) – Compose file path.
remove_volumes (bool) – Remove named volumes.
remove_images (str | None) – all or local to remove images.
timeout (int) – Shutdown timeout.

Return type:

morie.container.compose_ps(*, compose_file=None)[source]¶

List running compose services.

Parameters:: compose_file (str | Path | None) – Compose file path.
Return type:: DockerResult

morie.container.compose_logs(*, compose_file=None, services=None, tail=100)[source]¶

Retrieve logs from compose services.

Parameters:

compose_file (str | Path | None) – Compose file path.
services (list[str] | None) – Specific services.
tail (int) – Number of lines per service.

Return type:

morie.container.create_network(name='morie-net', *, driver='bridge')[source]¶

Create a Docker network for MORIE services.

Parameters:

name (str) – Network name.
driver (str) – Network driver (default bridge).

Return type:

morie.container.remove_network(name='morie-net')[source]¶

Remove a Docker network.

Parameters:: name (str) – Network name.
Return type:: DockerResult

morie.container.setup_ollama_sidecar(*, network='morie-net', ollama_image='ollama/ollama:latest', model='qwen2.5:7b', gpu=False)[source]¶

Set up an Ollama sidecar container for local LLM inference.

Creates a Docker network, starts an Ollama container, and pulls the specified model.

Parameters:

network (str) – Docker network name.
ollama_image (str) – Ollama Docker image.
model (str) – Model to pull (e.g. qwen2.5:7b).
gpu (bool) – If True, pass --gpus=all to enable GPU acceleration.

Returns:

Keys: network, container, model_pull.

Return type:

dict[str, DockerResult]

morie.container.generate_dockerfile(*, python_version='3.11', include_r=True, include_quarto=False, extras=None, output_path=None)[source]¶

Generate a Dockerfile for the current MORIE environment.

Creates a multi-stage Dockerfile that installs Python, optionally R and Quarto, and the morie package with requested extras.

Parameters:

python_version (str) – Python version for the base image.
include_r (bool) – Include R installation.
include_quarto (bool) – Include Quarto CLI installation.
extras (list[str] | None) – pip extras to install (e.g. ["ai", "carbon"]).
output_path (str | Path | None) – If provided, write the Dockerfile to this path.

Returns:

Dockerfile contents.

Return type:

morie.container.generate_compose_file(*, include_ollama=True, output_path=None)[source]¶

Generate a docker-compose.yml for MORIE services.

Parameters:

include_ollama (bool) – Include Ollama LLM sidecar service.
output_path (str | Path | None) – If provided, write the file to this path.

Returns:

docker-compose.yml contents.

Return type:

morie.container.render_health_check(result)[source]¶

Display health check results with rich formatting.

Parameters:: result (HealthCheckResult) – Health check to render.
Return type:: None

morie.container.render_containers(containers)[source]¶

Display container list with rich formatting.

Parameters:: containers (list[ContainerInfo]) – Containers to display.
Return type:: None

morie.container.render_images(images)[source]¶

Display image list with rich formatting.

Parameters:: images (list[ImageInfo]) – Images to display.
Return type:: None

morie.container.render_environment_diff(diff)[source]¶

Display environment diff with rich formatting.

Parameters:: diff (EnvironmentDiff) – Environment comparison to render.
Return type:: None

Enriched cheatsheets for fn/ entries.

the author’s brief: “make this fully descriptive, and assume our users know nothing about anything (just like how I did a few years ago due to the wrong teachers).”

Plain cheatsheet() on each fn returns one terse line (e.g. “huberw: Huber psi-weight function”). That is fine for tab-complete hints but is useless to someone who has never met an M-estimator before. This module produces a multi-section help block with:

Short description (from the fn’s existing one-line cheatsheet)

When to use this – plain-English category guidance

A category-themed quote, because learning sticks better when it’s threaded through stories the reader already knows

Reference (paper / book the formula comes from, when known)

Module path (so the reader can from morie.fn import …)

Public API: cheatsheet(fn_name) -> str. Used by:

the TUI Help screen (‘h’ key) when the user expands a fn
morie cheatsheet <fn> CLI subcommand
the documentation generator (Sphinx pre-build hook)

morie.cheatsheet.cheatsheet(fn_name)[source]¶

Multi-section enriched help block for a single fn/ entry.

Falls back gracefully when bits are missing – every section is independently optional, so a fn with only a one-line cheatsheet still gets a usable card (just without the “When to use” guidance).

Parameters:: fn_name (str) – The short import alias (e.g. "huberw", "icc1").
Returns:: Multi-line block ready to print to a terminal.
Return type:: str

morie.tutorial — interactive first-time-user walkthrough.

Reached via morie tutorial on the CLI. Walks the user through the same sequence documented in TUTORIAL.md but runs each step live and prompts before continuing, so a non-coder doesn’t have to copy-paste from a markdown file in another window.

Design notes:

Each step prints a short explanation, runs its command(s) in-process, then pauses for [Enter] continue / [s]kip / [q]uit.

Commands run through the same entry point as the morie command (morie.runner.main, in this process), so the user sees the same output they would type by hand.

No external state — output lands in ~/morie-tutorial-<date>/ so re-running the tutorial doesn’t overwrite a previous attempt.

morie.tutorial.run()[source]¶

Entry point. Returns CLI exit code.

Return type:: int

Central command registry for MORIE StatScreen and REPL.

Wraps all 620+ backend functions as user-facing commands plus aliases, wrangling helpers, R bridge commands, and compound workflows. Both StatScreen and REPL dispatch through this registry.

All backend imports are lazy (inside handler bodies) to avoid loading scipy/numpy/pandas at import time.

class morie.stat_commands.StatCommand(name, category, usage, description, handler_stat, handler_repl, aliases=<factory>, module='', is_compound=False, is_r_bridge=False)[source]¶

Bases: object

A single user-facing command.

Parameters:

name (str)
category (str)
usage (str)
description (str)
handler_stat (Callable)
handler_repl (Callable)
aliases (list[str])
module (str)
is_compound (bool)
is_r_bridge (bool)

name: str¶

category: str¶

usage: str¶

description: str¶

handler_stat: Callable¶

handler_repl: Callable¶

aliases: list[str]¶

module: str = ''¶

is_compound: bool = False¶

is_r_bridge: bool = False¶

morie.stat_commands.register(cmd)[source]¶

Register a command and its aliases.

Parameters:: cmd (StatCommand)
Return type:: None

morie.stat_commands.resolve(name)[source]¶

Look up a command by name or alias.

Parameters:: name (str)
Return type:: StatCommand | None

morie.stat_commands.all_command_names()[source]¶

All names including aliases, sorted.

Return type:: list[str]

morie.stat_commands.commands_by_category()[source]¶

Commands grouped by category.

Return type:: dict[str, list[StatCommand]]

Editors, terminal UI and tooling¶

morie.agent — call the rmorie terminal agent (the rmorie R package’s launcher).

morie already ships a native LLM layer (morie.ask / morie.llm). This is the ecosystem-uniform entry point that shells out to the optional rmorie CLI agent — the same agent rmorie::agent / rmoriedata::mdask / rmoriebricklayer::agentbndl call — so every MORIE package exposes one consistent agent. Optional dependency: needs the rmorie binary on PATH and, for cloud backends, RMORIE_AGENT_API_KEY in the environment (the key is never passed as an argument).

morie.agent_cli.agent(task, *, model=None, backend='auto', dry_run=False)[source]¶

Run the rmorie CLI agent on task and return its output.

Parameters:

task (str) – The prompt/task for the agent.
model (str | None) – Model id, e.g. "claude-sonnet-5", "gpt-4o", "gemini-2.5-flash", or an Ollama tag such as "llama3.2". Defaults to the CLI’s default model.
backend (str) – "auto" (route on the model id), "anthropic", "openai", "gemini", or "ollama".
dry_run (bool) – If True the agent only plans (no network call).

Returns:

The agent’s combined stdout/stderr, or a clear message if the rmorie binary is not installed.

Return type:

Benchmarking and performance analysis for MORIE.

Provides comprehensive tools for profiling module run, tracking memory usage, detecting performance regressions, and generating benchmark reports. All benchmarks capture system metadata for reproducibility.

The benchmarking framework is designed for scientific computing workloads where both computational cost and statistical output quality matter. Integration with CodeCarbon emissions tracking is supported through the morie[carbon] optional dependency.

References

Boisvert, R. F. et al. (2001). The Architecture of Scientific Software (Vol. 60). Springer. https://doi.org/10.1007/978-1-4615-1339-4

Kalibera, T. & Jones, R. (2013). Rigorous benchmarking in reasonable time. Proceedings of the 2013 International Symposium on Memory Management, 63–74. https://doi.org/10.1145/2464157.2464160

class morie.bench.SystemInfo(python_version, platform, machine, cpu_count, total_memory_gb, os_version, timestamp, package_versions=<factory>)[source]¶

Bases: object

Host system information captured at benchmark time.

Parameters:

python_version (str)
platform (str)
machine (str)
cpu_count (int)
total_memory_gb (float)
os_version (str)
timestamp (str)
package_versions (dict[str, str])

python_version: str¶

platform: str¶

machine: str¶

cpu_count: int¶

total_memory_gb: float¶

os_version: str¶

timestamp: str¶

package_versions: dict[str, str]¶

to_dict()[source]¶

Convert to dict for serialization.

Return type:: dict[str, Any]

class morie.bench.BenchmarkResult(name, iterations, times, peak_memory_mb, output_size_bytes=0, system_info=None, metadata=<factory>)[source]¶

Bases: object

Result from a single benchmark run.

Parameters:

name (str)
iterations (int)
times (list[float])
peak_memory_mb (float)
output_size_bytes (int)
system_info (SystemInfo | None)
metadata (dict[str, Any])

name: str¶

iterations: int¶

times: list[float]¶

peak_memory_mb: float¶

output_size_bytes: int = 0¶

system_info: SystemInfo | None = None¶

metadata: dict[str, Any]¶

property mean_time: float¶: Mean run time in seconds.

property median_time: float¶: Median run time.

property std_time: float¶: Standard deviation of run time.

property min_time: float¶: Minimum run time.

property max_time: float¶: Maximum run time.

property ci_95: tuple[float, float]¶: 95% confidence interval for mean time (normal approximation).

property cv: float¶: Coefficient of variation.

to_dict()[source]¶

Convert to dict for serialization.

Return type:: dict[str, Any]

class morie.bench.MemoryProfile(name, peak_mb, snapshots, traceback_peak='')[source]¶

Bases: object

Memory usage profile over time.

Parameters:

name (str)
peak_mb (float)
snapshots (list[tuple[float, float]])
traceback_peak (str)

name: str¶

peak_mb: float¶

snapshots: list[tuple[float, float]]¶

traceback_peak: str = ''¶

to_dataframe()[source]¶

Convert snapshots to a DataFrame.

Return type:: DataFrame

class morie.bench.BenchmarkComparison(name_a, name_b, result_a, result_b, speedup, memory_ratio, statistically_significant, p_value)[source]¶

Bases: object

Comparison of two benchmark results.

Parameters:

name_a (str)
name_b (str)
result_a (BenchmarkResult)
result_b (BenchmarkResult)
speedup (float)
memory_ratio (float)
statistically_significant (bool)
p_value (float)

name_a: str¶

name_b: str¶

result_a: BenchmarkResult¶

result_b: BenchmarkResult¶

speedup: float¶

memory_ratio: float¶

statistically_significant: bool¶

p_value: float¶

summary()[source]¶

Human-readable summary.

Return type:: str

class morie.bench.BenchmarkSuite(name, results=<factory>, timestamp='', system_info=None)[source]¶

Bases: object

Collection of benchmark results.

Parameters:

name (str)
results (list[BenchmarkResult])
timestamp (str)
system_info (SystemInfo | None)

name: str¶

results: list[BenchmarkResult]¶

timestamp: str = ''¶

system_info: SystemInfo | None = None¶

to_dataframe()[source]¶

Convert all results to a summary DataFrame.

Return type:: DataFrame

class morie.bench.RegressionReport(comparisons, regressions, improvements, threshold_pct)[source]¶

Bases: object

Performance regression detection report.

Parameters:

comparisons (list[BenchmarkComparison])
regressions (list[BenchmarkComparison])
improvements (list[BenchmarkComparison])
threshold_pct (float)

comparisons: list[BenchmarkComparison]¶

regressions: list[BenchmarkComparison]¶

improvements: list[BenchmarkComparison]¶

threshold_pct: float¶

property has_regressions: bool¶: True if any significant regressions detected.

morie.bench.capture_system_info()[source]¶

Capture current system information for benchmark reproducibility.

Returns:: System metadata including Python version, OS, CPU, RAM, and key package versions.
Return type:: SystemInfo

morie.bench.benchmark(func, *args, name=None, iterations=5, warmup=1, track_memory=True, gc_collect=True, **kwargs)[source]¶

Benchmark a function’s run time and memory usage.

Parameters:

func (callable) – Function to benchmark.
*args – Positional arguments passed to func.
name (str | None) – Benchmark name (default: function name).
iterations (int) – Number of timed iterations (default 5).
warmup (int) – Number of warmup iterations to discard (default 1).
track_memory (bool) – If True, track peak memory allocation.
gc_collect (bool) – If True, run garbage collection before each iteration.
**kwargs – Keyword arguments passed to func.

Return type:

BenchmarkResult

morie.bench.benchmark_module(module_name, *, cpads_csv=None, output_dir=None, iterations=3, warmup=1)[source]¶

Benchmark a single MORIE module.

Parameters:

module_name (str) – Name of the module to benchmark (from MODULE_SPECS).
cpads_csv (str | Path | None) – Path to CPADS CSV file.
output_dir (str | Path | None) – Output directory for module results.
iterations (int) – Number of iterations.
warmup (int) – Number of warmup iterations.

Return type:

BenchmarkResult

morie.bench.benchmark_all_modules(*, cpads_csv=None, output_dir=None, iterations=2)[source]¶

Benchmark all MORIE modules.

Parameters:

cpads_csv (str | Path | None) – CPADS CSV path.
output_dir (str | Path | None) – Base output directory.
iterations (int) – Iterations per module.

Return type:

BenchmarkSuite

morie.bench.memory_profile(func, *args, name=None, interval_seconds=0.1, **kwargs)[source]¶

Profile memory usage of a function over time.

Samples memory allocation at regular intervals during run.

Parameters:

func (callable) – Function to profile.
*args – Positional arguments.
name (str | None) – Profile name.
interval_seconds (float) – Sampling interval (default 0.1s).
**kwargs – Keyword arguments.

Return type:

MemoryProfile

morie.bench.cpu_profile(func, *args, name=None, sort_by='cumulative', top_n=20, **kwargs)[source]¶

Profile CPU usage per function using cProfile.

Parameters:

func (callable) – Function to profile.
*args – Positional arguments.
name (str | None) – Profile name.
sort_by (str) – Sort key for results (cumulative, tottime, calls).
top_n (int) – Number of top entries to include.
**kwargs – Keyword arguments.

Returns:

Profile results with keys total_time, top_functions, total_calls.

Return type:

dict[str, Any]

morie.bench.scalability_analysis(func, df, *, sizes=None, fractions=None, iterations=3, name=None, seed=42)[source]¶

Analyze how run time scales with data size.

Runs a function with increasing subsets of the input data and records run time and memory for each size.

Parameters:

func (callable) – Function that takes a DataFrame as input.
df (pd.DataFrame) – Full dataset.
sizes (list[int] | None) – Absolute sample sizes to test.
fractions (list[float] | None) – Fraction of data to test (e.g. [0.1, 0.25, 0.5, 1.0]).
iterations (int) – Iterations per size.
name (str | None) – Benchmark name.
seed (int) – Random seed for sampling.

Returns:

Columns: n, mean_time_s, std_time_s, peak_memory_mb.

Return type:

pd.DataFrame

morie.bench.compare_benchmarks(result_a, result_b, *, alpha=0.05)[source]¶

Compare two benchmark results with statistical testing.

Uses a two-sample t-test (Welch’s) to determine if the performance difference is statistically significant.

Parameters:

result_a (BenchmarkResult) – Baseline benchmark.
result_b (BenchmarkResult) – Comparison benchmark.
alpha (float) – Significance level.

Return type:

BenchmarkComparison

morie.bench.ab_compare(func_a, func_b, *args, name_a='A', name_b='B', iterations=10, warmup=2, **kwargs)[source]¶

Run A/B comparison between two implementations.

Parameters:

func_a (callable) – First implementation.
func_b (callable) – Second implementation.
*args – Arguments passed to both functions.
name_a (str) – Name for first implementation.
name_b (str) – Name for second implementation.
iterations (int) – Number of iterations per implementation.
warmup (int) – Warmup iterations.
**kwargs – Keyword arguments passed to both functions.

Return type:

BenchmarkComparison

morie.bench.detect_regressions(baseline, current, *, threshold_pct=10.0, alpha=0.05)[source]¶

Detect performance regressions between two benchmark suites.

Parameters:

baseline (BenchmarkSuite) – Baseline benchmarks (e.g. from the previous release).
current (BenchmarkSuite) – Current benchmarks.
threshold_pct (float) – Percentage slowdown to flag as regression.
alpha (float) – Significance level for t-test.

Return type:

RegressionReport

morie.bench.summarize_benchmark(result)[source]¶

Produce a detailed statistical summary of a benchmark.

Parameters:: result (BenchmarkResult) – Benchmark to summarize.
Returns:: Summary statistics including outlier detection.
Return type:: dict[str, Any]

morie.bench.save_results(results, path, *, format='json')[source]¶

Save benchmark results to disk.

Parameters:

results (BenchmarkResult | BenchmarkSuite) – Results to save.
path (str | Path) – Output file path.
format (str) – json or csv.

Returns:

Path to saved file.

Return type:

Path

morie.bench.load_results(path)[source]¶

Load benchmark results from a JSON file.

Parameters:: path (str | Path) – Path to saved results.
Return type:: BenchmarkSuite

morie.bench.append_to_history(result, history_path='benchmarks/history.jsonl')[source]¶

Append a benchmark result to the history file (JSONL format).

Parameters:

result (BenchmarkResult) – Result to append.
history_path (str | Path) – Path to the JSONL history file.

Return type:

Path

morie.bench.load_history(history_path='benchmarks/history.jsonl', *, name_filter=None)[source]¶

Load benchmark history into a DataFrame.

Parameters:

history_path (str | Path) – Path to JSONL history file.
name_filter (str | None) – Only load entries matching this name.

Return type:

pd.DataFrame

morie.bench.plot_benchmarks(suite, *, output_path=None, figsize=(12, 6))[source]¶

Create a bar chart of benchmark results.

Parameters:

suite (BenchmarkSuite) – Benchmark suite to visualize.
output_path (str | Path | None) – If provided, save the figure to this path.
figsize (tuple[float, float]) – Figure size in inches.

Returns:

The matplotlib figure object.

Return type:

matplotlib.figure.Figure

morie.bench.plot_scalability(df, *, output_path=None, figsize=(10, 5))[source]¶

Plot scalability analysis results.

Parameters:

df (pd.DataFrame) – Output from scalability_analysis().
output_path (str | Path | None) – If provided, save the figure.
figsize (tuple[float, float]) – Figure size.

Return type:

matplotlib.figure.Figure

morie.bench.generate_report(suite, *, output_path=None)[source]¶

Generate a Markdown benchmark report.

Parameters:

suite (BenchmarkSuite) – Benchmark suite to report.
output_path (str | Path | None) – If provided, save the report.

Returns:

Markdown report text.

Return type:

morie.bench.render_result(result)[source]¶

Display a benchmark result with rich formatting.

Parameters:: result (BenchmarkResult)
Return type:: None

morie.bench.render_suite(suite)[source]¶

Display a benchmark suite with rich formatting.

Parameters:: suite (BenchmarkSuite)
Return type:: None

morie.bench.render_regression_report(report)[source]¶

Display regression detection results.

Parameters:: report (RegressionReport)
Return type:: None

MORIE animated end-to-end demo.

Runs a ~30-second showcase of the MRM empirical-paper callables on the bundled reference samples, with live spinners, progress bars, and streaming result tables. Intended to give a user the DoubleML/Optuna-style “is it actually doing something?” feeling.

Run:: python -m morie.demo python -m morie.demo –no-rich # plain text fallback python -m morie.demo –skip-slow # skip Kulldorff (saves 30s)

morie.demo.step1_otis_callables()[source]¶

Demo step 1: placement concentration, segregation duration and alert co-occurrence on the bundled OTIS samples.

Return type:: None

morie.demo.step2_longsim()[source]¶

Demo step 2: simulate a 50-subject, 20-timepoint VAR(1) panel with AR(1) covariance and print its shape and means.

Return type:: None

morie.demo.step3_levy()[source]¶

Demo step 3: the Levy-flight Hill exponent of step lengths on the bundled TPS assault sample.

Return type:: None

morie.demo.step4_kulldorff()[source]¶

Demo step 4: a Kulldorff space-time scan with 49 Monte Carlo permutations on the bundled TPS sample.

Return type:: None

morie.demo.step5_license_check()[source]¶

Demo step 5: print morie’s license metadata (see morie_license_metadata).

Return type:: None

morie.demo.main(argv=None)[source]¶

Run the animated morie demo (python -m morie.demo).

Steps: the OTIS callables on the bundled samples, the longitudinal simulator, the TPS Levy-flight exponent, the Kulldorff scan (skipped with --skip-slow) and the license posture. --no-rich forces the plain-text fallback.

Parameters:: argv (list[str] | None) – command-line arguments (None reads sys.argv).
Returns:: 0 on completion.
Return type:: int

Minimal file editor entry point (native, no TUI framework).

The previous implementation embedded a Textual TUI. Textual was the last interactive-extra dependency; the native replacement opens the user’s own editor ($VISUAL / $EDITOR, falling back to vi) on the file, which is what a terminal user expects anyway, and never imports a UI framework.

morie.editor.edit_file(path, lang_hint=None)[source]¶

Open path in the user’s editor; returns the exit code.

Parameters:

path (str)
lang_hint (str | None)

Return type:

morie.editor.main(argv=None)[source]¶

Entry point of morie-edit <file>: open the file in the built-in editor.

Returns:: 2 with a usage line when no file is given, else the editor’s exit code.
Parameters:: argv (list[str] | None)
Return type:: int

morie.fast — the opt-in JIT acceleration surface.

Public re-export of the kernels in morie._jit. import morie.fast gives a stable, user-facing namespace whose functions are numerically identical to their scipy / numpy equivalents but compile to native code when numba is installed.

Usage¶

>>> import morie.fast as mf
>>> mf.is_jit_available() in (True, False)  # True with the compiled core loaded
True
>>> [round(float(v), 4) for v in mf.normal_pdf([0.0, 1.0], 0.0, 1.0)]
[0.3989, 0.242]

Functions¶

normal_pdf(x, mean, sd)

Drop-in for scipy.stats.norm.pdf. ~5-10× faster on small arrays once JIT-compiled.

normal_logpdf(x, mean, sd)

Drop-in for scipy.stats.norm.logpdf.

mean_jit(arr) / std_jit(arr, ddof=1) / var_jit(arr, ddof=1)

Drop-ins for arr.mean() / np.std(arr, ddof=...) / np.var(arr, ddof=...).

cor_pearson_jit(x, y)

Drop-in for scipy.stats.pearsonr(x, y)[0] (point estimate only).

euclid_dist_jit(a, b)

Drop-in for np.linalg.norm(a - b).

is_jit_available()

Returns True iff the compiled C++ core is loaded and MORIE_DISABLE_JIT is not set.

jit_if_available(*args, **kwargs)

Identity decorator factory (kept for API stability; the compiled core does the accelerating). For users decorating their own kernels:

from morie.fast import jit_if_available

@jit_if_available(cache=True, nogil=True)
def my_hot_loop(arr):
    ...

morie.fast.normal_pdf(x, mean, sd)[source]¶

Normal PDF over an array – the kernel inside dnorm.

Parameters:

mean (float)
sd (float)

Return type:

marr

morie.fast.normal_logpdf(x, mean, sd)[source]¶

Normal log-density – preferred for likelihoods (avoids underflow).

Parameters:

mean (float)
sd (float)

Return type:

marr

morie.fast.mean_jit(arr)[source]¶

Arithmetic mean of a 1-D array.

Return type:: float

morie.fast.std_jit(arr, ddof=1)[source]¶

Sample standard deviation with optional ddof.

Parameters:: ddof (int)
Return type:: float

morie.fast.var_jit(arr, ddof=1)[source]¶

Sample variance with optional ddof.

Parameters:: ddof (int)
Return type:: float

morie.fast.cor_pearson_jit(x, y)[source]¶

Pearson correlation coefficient.

Return type:: float

morie.fast.euclid_dist_jit(a, b)[source]¶

Euclidean (L2) distance between two equal-length vectors.

Return type:: float

morie.fast.bootstrap_mean_jit(arr, B, seed)[source]¶

Bootstrap-replicate means: B resamples of len(arr) drawn with replacement, returning each replicate’s mean.

Runs in the compiled C++ core (std::mt19937_64); a given seed is fully reproducible. Unlike the other kernels there is no pure-numpy fallback – a different RNG would silently change the replicate values – so the compiled core (morie._core) is required.

Parameters:

B (int)
seed (int)

Return type:

marr

morie.fast.trimmed_ipw_weights_jit(treat, propensity, trim_lo=0.01, trim_hi=0.99)[source]¶

IPW weights with propensity-score clipping at [trim_lo, trim_hi].

Parameters:

trim_lo (float)
trim_hi (float)

Return type:

marr

morie.fast.is_jit_available()[source]¶

True when the compiled C++ core (morie._core) is available.

Return type:: bool

morie.fast.jit_if_available(*args, **kwargs)[source]¶

Conditional numba.njit decorator.

Returns numba.njit(*args, **kwargs) when numba is importable and MORIE_DISABLE_JIT is unset; returns an identity decorator otherwise.

The signature mirrors numba.njit so existing JIT-decorated code can swap to this drop-in without other changes.

Use either form:

@jit_if_available
def f(x): ...

@jit_if_available(cache=True, nogil=True)
def f(x): ...

morie.i18n — minimal English / French string surface.

Reaches only the high-traffic CLI strings (cheatsheet labels, tutorial headings, doctor headings, friendly-error hints). Methodology documentation stays English-only — translating dense statistical prose is its own scoped project.

Activation: set MORIE_LOCALE=fr before running morie. Defaults to English when unset or set to anything other than fr.

This is a deliberately lightweight i18n layer (no gettext, no .po files) because:

Canadian sociolegal users are the target audience; bilingual EN/FR is the regulatory norm; nothing else needs to ship today.

gettext requires C-extension compilation on first run; we want pure-Python so the wheel installs anywhere.

Adding a new locale is a one-file edit: add another dict and route through _select_locale.

morie.i18n.t(key, **fmt)[source]¶

Translate a key; if the key isn’t registered, return the key itself.

Parameters:

key (str)
fmt (object)

Return type:

morie.i18n.current_locale()[source]¶

Return the two-letter locale code currently in effect (‘en’ when MORIE_LOCALE names none we carry).

Return type:: str

Output inspection and statistical verification for MORIE.

Provides two command families:

morie inspect <path>: Browse output CSVs – show schema, row counts, summary statistics.
morie verify <path>: Validate statistical outputs – check p-values, confidence intervals, standard errors, odds ratios, and other common statistical quantities for correctness.

Both commands work entirely offline (no LLM required) and render output using rich when a TTY is detected.

class morie.inspector.InspectionResult(file_path, rows, columns, column_names, dtypes, missing_counts, head, summary_stats=None)[source]¶

Bases: object

Summary of a single data file.

Parameters:

file_path (str)
rows (int)
columns (int)
column_names (list[str])
dtypes (dict[str, str])
missing_counts (dict[str, int])
head (DataFrame)
summary_stats (DataFrame | None)

file_path: str¶

rows: int¶

columns: int¶

column_names: list[str]¶

dtypes: dict[str, str]¶

missing_counts: dict[str, int]¶

head: DataFrame¶

summary_stats: DataFrame | None = None¶

morie.inspector.inspect_output(path)[source]¶

Inspect a single output file.

Parameters:: path (str | Path) – Path to a CSV, TSV, Excel, or Parquet file.
Return type:: InspectionResult

morie.inspector.inspect_directory(directory, *, module_name=None)[source]¶

Inspect all CSV files in a directory.

Parameters:

directory (str | Path) – Directory to scan.
module_name (str | None) – If provided, only inspect files expected by this module.

Return type:

list[InspectionResult]

class morie.inspector.VerificationCheck(name, passed, message, severity='error')[source]¶

Bases: object

A single verification check result.

Parameters:

name (str)
passed (bool)
message (str)
severity (str)

name: str¶

passed: bool¶

message: str¶

severity: str = 'error'¶

class morie.inspector.VerificationReport(file_path, checks=<factory>)[source]¶

Bases: object

Verification report for a single file.

Parameters:

file_path (str)
checks (list[VerificationCheck])

file_path: str¶

checks: list[VerificationCheck]¶

property passed: bool¶: True if no error-severity checks failed.

property error_count: int¶

property warning_count: int¶

morie.inspector.verify_statistical_output(path)[source]¶

Run statistical validation checks on an output file.

Checks¶

p-values in [0, 1]
Standard errors >= 0
Odds ratios > 0
Confidence intervals: lower <= upper
Confidence intervals contain point estimate (when identifiable)
No NaN in critical columns (estimates, p-values)
Sample sizes > 0 and integer-valued
R-squared in [0, 1]
AIC/BIC are finite

param path:: Path to a CSV file with statistical output.
type path:: str | Path
rtype:: VerificationReport

Parameters:: path (str | Path)
Return type:: VerificationReport

morie.inspector.verify_directory(directory, *, module_name=None)[source]¶

Verify all CSV files in a directory.

Parameters:

directory (str | Path) – Directory to scan.
module_name (str | None) – If provided, only verify files expected by this module.

Return type:

list[VerificationReport]

morie.inspector.render_inspection(result)[source]¶

Print an inspection result to the terminal.

Parameters:: result (InspectionResult)
Return type:: None

morie.inspector.render_verification(report)[source]¶

Print a verification report to the terminal.

Parameters:: report (VerificationReport)
Return type:: None

Local Ollama client for MORIE.

Pure httpx-based client for a local Ollama instance running at localhost:11434. Provides model management (pull, list, remove), chat, and streaming – no external deps beyond httpx.

This module backs the ollama provider slot in morie.llm.

Environment Variables¶

OLLAMA_HOST, OLLAMA_BASE_URLstr: Override the Ollama endpoint. Default: http://localhost:11434.
OLLAMA_MODEL, MORIE_OLLAMA_MODELstr: Override the default local model. Default: gemma4:e2b.

Both can be saved instead with morie config set ollama.url|ollama.model.

class morie.loc.ModelInfo(name, size=0, parameter_size='', family='', quantization='', modified_at='')[source]¶

Bases: object

Metadata for a locally available Ollama model.

Parameters:

name (str)
size (int)
parameter_size (str)
family (str)
quantization (str)
modified_at (str)

name: str¶

size: int = 0¶

parameter_size: str = ''¶

family: str = ''¶

quantization: str = ''¶

modified_at: str = ''¶

property size_gb: float¶

property label: str¶

class morie.loc.LocalOllama(base_url=None, model=None, timeout=300.0)[source]¶

Bases: object

Client for a local Ollama instance.

Parameters:

base_url (str, optional) – Override OLLAMA_BASE_URL.
model (str, optional) – Override MORIE_OLLAMA_MODEL.
timeout (float, optional) – Request timeout in seconds (default 120).

Examples

is_running() reports whether an Ollama server answers; the other calls need one:

>>> client = LocalOllama()
>>> client.is_running() in (True, False)
True
>>> models = client.list_models()
>>> response = client.chat("What is IPW?")

property model: str¶: Active model – auto-detected from Ollama if not explicitly set.

is_running(timeout=2.0)[source]¶

Check if Ollama is reachable.

Parameters:: timeout (float)
Return type:: bool

list_models()[source]¶

List locally available models.

Return type:: list[ModelInfo]

model_names()[source]¶

Return just the model name strings.

Return type:: list[str]

has_model(name)[source]¶

Check if a specific model is available locally.

Parameters:: name (str)
Return type:: bool

pull(name, *, stream=True, timeout=600.0)[source]¶

Pull (download) a model.

Parameters:

name (str) – Model name, e.g. gemma3:4b or llama3.2:3b.
stream (bool) – If True, yield progress dicts as they arrive.
timeout (float) – Download timeout (default 600s).

Yields:

dict – Progress updates with status, digest, total, completed.

Return type:

Iterator[dict[str, Any]] | dict[str, Any]

remove(name)[source]¶

Delete a local model. Returns True on success.

Parameters:: name (str)
Return type:: bool

show(name=None)[source]¶

Get model details (parameters, template, license).

Parameters:: name (str | None)
Return type:: dict[str, Any]

chat(prompt, *, model=None, instructions=None, context=None, temperature=0.1, num_predict=4096)[source]¶

Send a chat message and return the full response.

Parameters:

prompt (str) – User message.
model (str, optional) – Override the default model.
instructions (str, optional) – Instructions for the model (sent as the system prompt).
context (list, optional) – Prior messages as [{"role": "user", "content": "..."}, ...].
temperature (float) – Sampling temperature.
num_predict (int) – Max tokens to generate.

Returns:

The assistant’s response text.

Return type:

stream_chat(prompt, *, model=None, instructions=None, context=None, temperature=0.1, num_predict=4096)[source]¶

Stream chat response chunks.

Yields:

str – Content chunks as they arrive from the model.

Parameters:

prompt (str)
model (str | None)
instructions (str | None)
context (list[dict[str, str]] | None)
temperature (float)
num_predict (int)

Return type:

Iterator[str]

generate(prompt, *, model=None, instructions=None, stream=False, temperature=0.1, num_predict=4096)[source]¶

Raw generation endpoint (non-chat). Returns full text or stream.

Parameters:

prompt (str)
model (str | None)
instructions (str | None)
stream (bool)
temperature (float)
num_predict (int)

Return type:

str | Iterator[str]

Rich live-progress wrapper for MORIE pipeline run.

Provides PipelineTracker which wraps morie.modules.run_module() with rich.progress bars, live status tables, and output-file validation. Falls back to plain print() statements when stdout is not a TTY (pipes, CI runners, redirected output).

Usage¶

Programmatic:

from morie.progress import PipelineTracker
tracker = PipelineTracker(["power-design", "logistic-models"], cpads_csv="data.csv")
results = tracker.run()

CLI (automatic when TTY detected):

morie pipeline --all -y   # rich progress bars in interactive terminals

class morie.progress.ModuleResult(name, status='pending', elapsed_seconds=0.0, output_files_expected=0, output_files_actual=0, error_message=None, outputs=<factory>)[source]¶

Bases: object

Outcome of a single module run.

Parameters:

name (str)
status (str)
elapsed_seconds (float)
output_files_expected (int)
output_files_actual (int)
error_message (str | None)
outputs (dict)

name: str¶

status: str = 'pending'¶

elapsed_seconds: float = 0.0¶

output_files_expected: int = 0¶

output_files_actual: int = 0¶

error_message: str | None = None¶

outputs: dict¶

class morie.progress.PipelineTracker(module_names, *, cpads_csv='/home/runner/work/morie/morie/src/morie/data/cpads_synthetic.csv', output_dir=None, track_carbon=True, use_live=None, dataset_key=None)[source]¶

Bases: object

Run analysis modules with live progress display.

Parameters:

module_names (list[str]) – Module names to run (must be keys in MODULE_SPECS).
cpads_csv (str) – Path to the CPADS CSV input file.
output_dir (str | None) – Directory for CSV outputs. None uses the module default.
track_carbon (bool) – If True and codecarbon is installed, track CO2 emissions.
use_live (bool | None) – Force live display on/off. None auto-detects from TTY.
dataset_key (str | None)

results: list[ModuleResult]¶

run()[source]¶

Run all modules, returning a list of ModuleResult.

Return type:: list[ModuleResult]

run_single(module_name)[source]¶

Run one module with optional spinner.

Parameters:: module_name (str)
Return type:: ModuleResult

summary_table()[source]¶: Build a rich.table.Table summarising all results.

morie.progress.execute_pipeline_with_progress(modules=None, *, cpads_csv='/home/runner/work/morie/morie/src/morie/data/cpads_synthetic.csv', dataset_key=None, output_dir=None, silent=False, track_carbon=True)[source]¶

Run the MORIE pipeline with rich progress display.

Drop-in replacement for morie.runner.execute_pipeline(). Auto-detects TTY for rich vs plain output.

Parameters:

modules (list[str] | None) – Module names to run. None runs all implemented modules.
cpads_csv (str) – Path to the CPADS CSV.
output_dir (str | None) – Output directory for CSVs.
silent (bool) – Skip confirmation prompt.
track_carbon (bool) – Track CO2 emissions via CodeCarbon.
dataset_key (str | None)

Returns:

0 on success, 1 on abort or failure.

Return type:

REPL bootstrap for TIDE (Go TUI) and any external REPL.

Usage from TIDE subprocess: import setup_repl from this module and call it with the caller’s globals() to inject the helpers.

This injects all convenience functions, stat_commands, and fn/ registry into the caller’s namespace – achieving parity with the Python TUI REPL.

morie.repl_init.setup_repl(ns)[source]¶

Inject all MORIE REPL helpers into namespace ns.

Parameters:: ns (dict)
Return type:: None

Launch the TIDE Go binary from within the morie Python package.

Looks for the binary in these locations (in order):

TIDE_BIN environment variable
Adjacent to this file (bundled in wheel)
$PATH (standalone install via go install or brew)
Build from source if Go toolchain is available

morie.tide_launcher.main()[source]¶

Run the TIDE binary with this process’s arguments and exit with its code.

The binary is found via TIDE_BIN or PATH; without one, this prints how to install it and exits 1. Ctrl-C exits 130.

Return type:: None

Full-screen Textual terminal IDE for MORIE.

Requires pip install morie[interactive] (textual >= 0.85).

Provides a Claude Code-inspired TUI with multiple screens:

ChatScreen – streaming LLM chat with slash commands
PipelineScreen – run analysis modules with live progress
DoctorScreen – environment diagnostics
DatasetScreen – browse and profile datasets
HelpScreen – built-in documentation and command reference
DebugScreen – debug log viewer and diagnostics
ReplScreen – Python/R interactive REPL
StatScreen – run statistical analyses (70+ commands)

Launch with:

morie tui        # explicit
morie            # auto-launches TUI when interactive terminal detected

Keybindings:

c  Chat         h  Help/Docs     s  Stats
p  Pipeline     g  Debug         r  REPL
d  Doctor       i  Inspect       q  Quit

morie.tui.launch_tui(*, agent=None)[source]¶

Launch the full-screen MORIE terminal IDE.

Parameters:: agent (str | None)
Return type:: int

morie verify-earth-engine – one-command EE auth smoke check.

Companion to morie verify-pollution (Workstream 6). This CLI confirms whether the Earth Engine auth pipeline is wired up end-to- end before you commit to a real pipeline run.

Steps it attempts, in order:

Is the earthengine-api package installed?
Do we have credentials? (env vars or ADC)
Does ee.Initialize() succeed?
Can we issue a trivial query? (fetch one pixel of a small image)

Exit codes mirror verify-pollution: - 0 – all stages passed - 1 – auth failure (missing credentials, expired token) - 2 – environment error (missing library, bad network)

See also .claude/plans/2026-04-17-pollution-health-mission.md §7 (post-registration smoke check) and the project’s internal Earth Engine registration notes.

morie.verify_earth_engine.handle_verify_earth_engine(args)[source]¶

Main CLI dispatch.

Parameters:: args (Namespace)
Return type:: int

morie.verify_earth_engine.register_subparser(subparsers)[source]¶

Wire the subcommand into morie.runner’s argparse tree.

Return type:: None

morie verify-pollution CLI handler (Workstream 6 of pollution-health plan).

One command wrapping morie.envhealth into a reproducible tool that prints ATE / PAF / mortality-displaced / equity / assumption-log and returns exit 0 on green, 1 on any assumption failure.

See .claude/plans/2026-04-17-pollution-health-mission.md §W6.

morie.verify_pollution.handle_verify_pollution(args)[source]¶

Dispatch for morie verify-pollution.

Returns 0 if all assumptions pass and the pipeline completes, 1 if any assumption fails, 2 on file / data errors.

Parameters:: args (Namespace)
Return type:: int

morie.verify_pollution.register_subparser(subparsers)[source]¶

Called from morie.runner.build_parser() to register this command.

Return type:: None

Function namespace `morie.fn`¶

The morie.fn namespace exposes 15,222 individual callables, indexed by a registry and resolved lazily on first access. To keep the wheel small, the implementations and per-callable guides ship as two compressed archives (_fnsrc.json.xz and describe_docs.json.xz, ~7 MB total) rather than tens of thousands of loose files; the importer resolves morie.fn.<name> from them transparently, and per-callable documentation is available at runtime via morie.fn.describe. The full registry is the canonical catalogue:

Naming registry for morie.fn functions.

Single source of truth: short name -> metadata.

The author-year strings in the section comments below (Kosorok 2008, Horowitz 2009, Armstrong 2014/2021, and so on) are shelf labels, not citations: they say which book-as-spec shelf a block of names came from. The bibliographic entry for each lives in the core module that implements the shelf. Used by fn/__init__.py for auto-export and stat_commands.py for CLI dispatch.

‘Knowledge itself is power. – Francis Bacon’

class morie.fn._registry.FnEntry(short, full, category, description, quote='')[source]¶

Bases: object

Registry entry for one fn/ function.

Parameters:

short (str)
full (str)
category (str)
description (str)
quote (str)

short: str¶

full: str¶

category: str¶

description: str¶

quote: str¶

morie.fn._registry.cheatsheet(category=None)[source]¶

Print a cheatsheet of all morie.fn functions.

>>> from morie.fn._registry import cheatsheet
>>> cheatsheet()
>>> cheatsheet("Test")  # filter by category

Parameters:: category (str | None)
Return type:: None

Result containers shared across the package:

Shared result dataclasses for morie.fn functions.

class morie.fn._containers.RlbRes(raw, std, avgr, k, n, ci_lo=0.0, ci_hi=0.0)[source]¶

Bases: object

Reliability result (Cronbach’s alpha).

Parameters:

raw (float)
std (float)
avgr (float)
k (int)
n (int)
ci_lo (float)
ci_hi (float)

raw: float¶

std: float¶

avgr: float¶

k: int¶

n: int¶

ci_lo: float = 0.0¶

ci_hi: float = 0.0¶

summary()[source]¶

Return type:: DataFrame

class morie.fn._containers.OmgRes(total, hier, alpha, nf, expvar=0.0)[source]¶

Bases: object

Omega result (McDonald’s omega).

Parameters:

total (float)
hier (float)
alpha (float)
nf (int)
expvar (float)

total: float¶

hier: float¶

alpha: float¶

nf: int¶

expvar: float = 0.0¶

class morie.fn._containers.KmoRes(msa, items=<factory>)[source]¶

Bases: object

KMO test result.

Parameters:

msa (float)
items (dict[str, float])

msa: float¶

items: dict[str, float]¶

class morie.fn._containers.BrtRes(chisq, df, pval)[source]¶

Bases: object

Bartlett’s sphericity result.

Parameters:

chisq (float)
df (int)
pval (float)

chisq: float¶

df: int¶

pval: float¶

class morie.fn._containers.RplRes(counts, props, year, sex=None)[source]¶

Bases: object

Regional placement result.

Parameters:

counts (DataFrame)
props (DataFrame)
year (int)
sex (str | None)

counts: DataFrame¶

props: DataFrame¶

year: int¶

sex: str | None = None¶

class morie.fn._containers.AstRes(data, summary)[source]¶

Bases: object

Alert-state combination result.

Parameters:

data (DataFrame)
summary (DataFrame)

data: DataFrame¶

summary: DataFrame¶

class morie.fn._containers.VolRes(data, mean, median)[source]¶

Bases: object

Volatility result.

Parameters:

data (DataFrame)
mean (float)
median (float)

data: DataFrame¶

mean: float¶

median: float¶

class morie.fn._containers.OtDmlR(ate, ate_se, ate_pval, att, att_se, att_pval, n, method='IRM')[source]¶

Bases: object

OTIS DML result.

Parameters:

ate (float)
ate_se (float)
ate_pval (float)
att (float)
att_se (float)
att_pval (float)
n (int)
method (str)

ate: float¶

ate_se: float¶

ate_pval: float¶

att: float¶

att_se: float¶

att_pval: float¶

n: int¶

method: str = 'IRM'¶

class morie.fn._containers.ESRes(measure, estimate, ci_lower=None, ci_upper=None, se=None, n=None, extra=<factory>)[source]¶

Bases: object

Standardised result for every effect-size calculation.

Parameters:

measure (str)
estimate (float)
ci_lower (float | None)
ci_upper (float | None)
se (float | None)
n (int | None)
extra (dict)

measure: str¶

estimate: float¶

ci_lower: float | None = None¶

ci_upper: float | None = None¶

se: float | None = None¶

n: int | None = None¶

extra: dict¶

class morie.fn._containers.TestResult(test_name, statistic, p_value, df=None, method='', n=None, extra=<factory>)[source]¶

Bases: object

Result from a hypothesis test.

Parameters:

test_name (str)
statistic (float)
p_value (float)
df (float | None)
method (str)
n (int | None)
extra (dict)

test_name: str¶

statistic: float¶

p_value: float¶

df: float | None = None¶

method: str = ''¶

n: int | None = None¶

extra: dict¶

summary()[source]¶

Return type:: str

to_dict()[source]¶

Return type:: dict

class morie.fn._containers.RegressionResult(method, coefficients, se, p_values, r_squared=None, adj_r_squared=None, residuals=None, fitted=None, n=0, k=0, extra=<factory>)[source]¶

Bases: object

Result from a regression model.

Parameters:

method (str)
coefficients (dict[str, float])
se (dict[str, float])
p_values (dict[str, float])
r_squared (float | None)
adj_r_squared (float | None)
residuals (marr | None)
fitted (marr | None)
n (int)
k (int)
extra (dict)

method: str¶

coefficients: dict[str, float]¶

se: dict[str, float]¶

p_values: dict[str, float]¶

r_squared: float | None = None¶

adj_r_squared: float | None = None¶

residuals: marr | None = None¶

fitted: marr | None = None¶

n: int = 0¶

k: int = 0¶

extra: dict¶

summary()[source]¶

Return type:: str

to_dict()[source]¶

Return type:: dict

class morie.fn._containers.DescriptiveResult(name, value=None, extra=<factory>)[source]¶

Bases: object

Result from a descriptive / exploratory function.

Parameters:

name (str)
value (float | dict | DataFrame | None)
extra (dict)

name: str¶

value: float | dict | DataFrame | None = None¶

extra: dict¶

summary()[source]¶

Return type:: str

to_dict()[source]¶

Return type:: dict

class morie.fn._containers.TimeSeriesResult(name, values=None, lags=None, ci_upper=None, ci_lower=None, extra=<factory>)[source]¶

Bases: object

Result from a time-series analysis.

Parameters:

name (str)
values (marr | None)
lags (marr | None)
ci_upper (marr | None)
ci_lower (marr | None)
extra (dict)

name: str¶

values: marr | None = None¶

lags: marr | None = None¶

ci_upper: marr | None = None¶

ci_lower: marr | None = None¶

extra: dict¶

summary()[source]¶

Return type:: str

to_dict()[source]¶

Return type:: dict

class morie.fn._containers.SurvivalResult(name, times=None, survival=None, ci_lower=None, ci_upper=None, median_survival=None, n_events=0, n_censored=0, extra=<factory>)[source]¶

Bases: object

Result from a survival analysis.

Parameters:

name (str)
times (marr | None)
survival (marr | None)
ci_lower (marr | None)
ci_upper (marr | None)
median_survival (float | None)
n_events (int)
n_censored (int)
extra (dict)

name: str¶

times: marr | None = None¶

survival: marr | None = None¶

ci_lower: marr | None = None¶

ci_upper: marr | None = None¶

median_survival: float | None = None¶

n_events: int = 0¶

n_censored: int = 0¶

extra: dict¶

summary()[source]¶

Return type:: str

to_dict()[source]¶

Return type:: dict

class morie.fn._containers.DiagnosticResult(name, estimate, ci_lower=None, ci_upper=None, n=0, extra=<factory>)[source]¶

Bases: object

Result from a diagnostic accuracy metric.

Parameters:

name (str)
estimate (float)
ci_lower (float | None)
ci_upper (float | None)
n (int)
extra (dict)

name: str¶

estimate: float¶

ci_lower: float | None = None¶

ci_upper: float | None = None¶

n: int = 0¶

extra: dict¶

summary()[source]¶

Return type:: str

to_dict()[source]¶

Return type:: dict

class morie.fn._containers.IRTResult(model, item_params, theta=None, se_theta=None, fit=<factory>, info=None)[source]¶

Bases: object

Result from an IRT model.

Parameters:

model (str)
item_params (dict)
theta (marr | None)
se_theta (marr | None)
fit (dict)
info (marr | None)

model: str¶

item_params: dict¶

theta: marr | None = None¶

se_theta: marr | None = None¶

fit: dict¶

info: marr | None = None¶

summary()[source]¶

Return type:: str

class morie.fn._containers.DIFResult(method, items=<factory>, flagged=<factory>, group_var='')[source]¶

Bases: object

Result from a DIF analysis.

Parameters:

method (str)
items (DataFrame)
flagged (list)
group_var (str)

method: str¶

items: DataFrame¶

flagged: list¶

group_var: str = ''¶

summary()[source]¶

Return type:: str

class morie.fn._containers.InvarianceResult(level, fit=<factory>, delta_fit=<factory>, passed=False)[source]¶

Bases: object

Result from a measurement invariance test.

Parameters:

level (str)
fit (dict)
delta_fit (dict)
passed (bool)

level: str¶

fit: dict¶

delta_fit: dict¶

passed: bool = False¶

summary()[source]¶

Return type:: str

class morie.fn._containers.RecidivismResult(rate, n_recid, n_total, ci_lower=0.0, ci_upper=1.0, subgroup=None, method='proportion')[source]¶

Bases: object

Result from a recidivism analysis.

Parameters:

rate (float)
n_recid (int)
n_total (int)
ci_lower (float)
ci_upper (float)
subgroup (str | None)
method (str)

rate: float¶

n_recid: int¶

n_total: int¶

ci_lower: float = 0.0¶

ci_upper: float = 1.0¶

subgroup: str | None = None¶

method: str = 'proportion'¶

summary()[source]¶

Return type:: str

class morie.fn._containers.CustodyResult(metric, value, n=0, period=None, region=None, extra=<factory>)[source]¶

Bases: object

Result from a custody/institutional metric.

Parameters:

metric (str)
value (float)
n (int)
period (str | None)
region (str | None)
extra (dict)

metric: str¶

value: float¶

n: int = 0¶

period: str | None = None¶

region: str | None = None¶

extra: dict¶

summary()[source]¶

Return type:: str

class morie.fn._containers.PcaRes(components, explained_variance, explained_variance_ratio, scores, n=0, p=0)[source]¶

Bases: object

Principal Component Analysis result.

Parameters:

components (marr)
explained_variance (marr)
explained_variance_ratio (marr)
scores (marr)
n (int)
p (int)

components: marr¶

explained_variance: marr¶

explained_variance_ratio: marr¶

scores: marr¶

n: int = 0¶

p: int = 0¶

summary()[source]¶

Return type:: str

class morie.fn._containers.FaRes(loadings, communalities, eigenvalues, variance_explained, n_factors=0, rotation='varimax')[source]¶

Bases: object

Exploratory Factor Analysis result.

Parameters:

loadings (marr)
communalities (marr)
eigenvalues (marr)
variance_explained (marr)
n_factors (int)
rotation (str)

loadings: marr¶

communalities: marr¶

eigenvalues: marr¶

variance_explained: marr¶

n_factors: int = 0¶

rotation: str = 'varimax'¶

summary()[source]¶

Return type:: str

class morie.fn._containers.CfaRes(cfi, tli, rmsea, srmr, loadings=<factory>, residuals=None)[source]¶

Bases: object

Confirmatory Factor Analysis result.

Parameters:

cfi (float)
tli (float)
rmsea (float)
srmr (float)
loadings (dict)
residuals (marr | None)

cfi: float¶

tli: float¶

rmsea: float¶

srmr: float¶

loadings: dict¶

residuals: marr | None = None¶

summary()[source]¶

Return type:: str

class morie.fn._containers.MdsRes(coordinates, stress, eigenvalues)[source]¶

Bases: object

Multidimensional Scaling result.

Parameters:

coordinates (marr)
stress (float)
eigenvalues (marr)

coordinates: marr¶

stress: float¶

eigenvalues: marr¶

summary()[source]¶

Return type:: str

class morie.fn._containers.TsneRes(embedding)[source]¶

Bases: object

t-SNE result.

Parameters:: embedding (marr)

embedding: marr¶

summary()[source]¶

Return type:: str

class morie.fn._containers.UmapRes(embedding)[source]¶

Bases: object

UMAP result.

Parameters:: embedding (marr)

embedding: marr¶

summary()[source]¶

Return type:: str

class morie.fn._containers.KmeansRes(labels, centers, inertia, n_iter)[source]¶

Bases: object

K-means clustering result.

Parameters:

labels (marr)
centers (marr)
inertia (float)
n_iter (int)

labels: marr¶

centers: marr¶

inertia: float¶

n_iter: int¶

summary()[source]¶

Return type:: str

class morie.fn._containers.HclstRes(labels, linkage_matrix, distances=None)[source]¶

Bases: object

Hierarchical clustering result.

Parameters:

labels (marr)
linkage_matrix (marr)
distances (marr | None)

labels: marr¶

linkage_matrix: marr¶

distances: marr | None = None¶

summary()[source]¶

Return type:: str

class morie.fn._containers.LdaRes(components, explained_variance_ratio, projected)[source]¶

Bases: object

Linear Discriminant Analysis result.

Parameters:

components (marr)
explained_variance_ratio (marr)
projected (marr)

components: marr¶

explained_variance_ratio: marr¶

projected: marr¶

summary()[source]¶

Return type:: str

class morie.fn._containers.DbscnRes(labels, n_clusters, n_noise)[source]¶

Bases: object

DBSCAN clustering result.

Parameters:

labels (marr)
n_clusters (int)
n_noise (int)

labels: marr¶

n_clusters: int¶

n_noise: int¶

summary()[source]¶

Return type:: str

class morie.fn._containers.SIRResult(model, t=None, S=None, I=None, R=None, E=None, R0=None, extra=<factory>)[source]¶

Bases: object

Result from a compartmental model (SIR/SEIR/etc).

Parameters:

model (str)
t (marr | None)
S (marr | None)
I (marr | None)
R (marr | None)
E (marr | None)
R0 (float | None)
extra (dict)

model: str¶

t: marr | None = None¶

S: marr | None = None¶

I: marr | None = None¶

R: marr | None = None¶

E: marr | None = None¶

R0: float | None = None¶

extra: dict¶

summary()[source]¶

Return type:: str

class morie.fn._containers.SpatialResult(name, statistic, p_value=None, expected=None, variance=None, local_values=None, extra=<factory>)[source]¶

Bases: object

Result from a spatial statistic.

Parameters:

name (str)
statistic (float)
p_value (float | None)
expected (float | None)
variance (float | None)
local_values (marr | None)
extra (dict)

name: str¶

statistic: float¶

p_value: float | None = None¶

expected: float | None = None¶

variance: float | None = None¶

local_values: marr | None = None¶

extra: dict¶

summary()[source]¶

Return type:: str

class morie.fn._containers.GenomicsResult(name, statistic, p_value=None, n=0, extra=<factory>)[source]¶

Bases: object

Result from a genomics/population genetics test.

Parameters:

name (str)
statistic (float)
p_value (float | None)
n (int)
extra (dict)

name: str¶

statistic: float¶

p_value: float | None = None¶

n: int = 0¶

extra: dict¶

summary()[source]¶

Return type:: str

class morie.fn._containers.CrimeResult(name, rate, ci_lower=None, ci_upper=None, n=0, population=0, extra=<factory>)[source]¶

Bases: object

Result from a criminological metric.

Parameters:

name (str)
rate (float)
ci_lower (float | None)
ci_upper (float | None)
n (int)
population (int)
extra (dict)

name: str¶

rate: float¶

ci_lower: float | None = None¶

ci_upper: float | None = None¶

n: int = 0¶

population: int = 0¶

extra: dict¶

summary()[source]¶

Return type:: str

class morie.fn._containers.SignalResult(name, filtered=None, fs=0.0, n_samples=0, extra=<factory>)[source]¶

Bases: object

Result from a signal processing function.

Parameters:

name (str)
filtered (marr | None)
fs (float)
n_samples (int)
extra (dict)

name: str¶

filtered: marr | None = None¶

fs: float = 0.0¶

n_samples: int = 0¶

extra: dict¶

summary()[source]¶

Return type:: str

to_dict()[source]¶

Return type:: dict

class morie.fn._containers.CryptoResult(algorithm, operation, success=True, extra=<factory>)[source]¶

Bases: object

Result from a cryptographic operation.

Parameters:

algorithm (str)
operation (str)
success (bool)
extra (dict)

algorithm: str¶

operation: str¶

success: bool = True¶

extra: dict¶

summary()[source]¶

Return type:: str

to_dict()[source]¶

Return type:: dict

Rich-output result containers for morie.fn.

R-style verbose __repr__ so users see paragraph-level summaries when they print a result, not just a bare dict – modeled after psych::omega(), lavaan summary(), survival::summary.coxph(), etc.

Usage:

res = welcht([1,2,3], [4,5,6])
print(res)        # multi-section ASCII summary
res.statistic     # still attribute-accessible like a dict
res.warnings      # list of advisory strings

class morie.fn._richresult.RichResult(title='', call='', summary_lines=<factory>, sections=<factory>, tables=<factory>, comparison=None, extras=<factory>, warnings=<factory>, interpretation='', payload=<factory>)[source]¶

Bases: dict

Generic rich-printing result dict-like – psych::omega style.

Inherits from dict so isinstance(result, dict) is True and legacy callers that expect a plain dict (.get(…), for k in r, result[“statistic”]) keep working unchanged. The RichResult layer adds title / sections / tables / interpretation / warnings on top – opt-in via print(result) or result.summary().

Supports multi-section output: - title short test/model name (top of output) - call optional call signature line (“Call: foo(x, y, …)”) - summary_lines headline metrics: list of (label, value) tuples - sections list of {“title”: str, “lines”: [(lbl, val)], “table”: […]} dicts

for additional sections (each printed with its own header)

tables list of {“title”: str, “headers”: […], “rows”: [[…]]}
for tabular output (column-aligned)
comparison optional dict with same shape as a section,
printed under “Compare this with …” heading
extras list of free-form paragraph blocks
warnings list of advisory strings – surfaced VERBATIM,
multi-line OK
interpretation optional plain-language concluding sentence(s)
payload arbitrary dict for programmatic access

Parameters:

title (str)
call (str)
summary_lines (list[tuple[str, Any]])
sections (list[dict])
tables (list[dict])
comparison (dict | None)
extras (list[str])
warnings (list[str])
interpretation (str)
payload (dict)

title: str = ''¶

call: str = ''¶

summary_lines: list[tuple[str, Any]]¶

sections: list[dict]¶

tables: list[dict]¶

comparison: dict | None = None¶

extras: list[str]¶

warnings: list[str]¶

interpretation: str = ''¶

payload: dict¶

get(key, default=None)[source]¶

Return the value for key if key is in the dictionary, else default.

Parameters:: key (str)

keys() → a set-like object providing a view on D's keys[source]¶

values() → an object providing a view on D's values[source]¶

items() → a set-like object providing a view on D's items[source]¶

to_dict()[source]¶

Return type:: dict

summary()[source]¶

Return type:: str

morie.fn._richresult.hypothesis_test_result(*, test_name, statistic, pvalue, df=None, alpha=0.05, extra_summary=None, warnings=None, extra_payload=None, callable_name=None)[source]¶

Factory for hypothesis-test results with a uniform layout.

callable_name (optional) – short name of the morie.fn callable that produced this result. If provided, a “Learn more” pointer is appended to the extras suggesting describe(<name>) for the full pedagogical guide. Caller can pass it explicitly OR auto-detect via the inspect-stack fallback in attach_describe_pointer().

Parameters:

test_name (str)
statistic (float)
pvalue (float)
df (float | None)
alpha (float)
extra_summary (list[tuple[str, Any]] | None)
warnings (list[str] | None)
extra_payload (dict | None)
callable_name (str | None)

Return type:

morie.fn._richresult.with_describe_pointer(result, callable_name)[source]¶

Add a describe() pointer to an existing RichResult.

Used by callables that return RichResult directly (without going through hypothesis_test_result()) to opt-in to the same convention.

Parameters:

result (RichResult)
callable_name (str)

Return type: