Test a Dirichlet-process partition cluster count against its prior

Source: Ghosal, S. and van der Vaart, A. (2017), Fundamentals of Nonparametric Bayesian Inference, section 4.1.5, Proposition 4.8, crediting Antoniak (1974), Annals of Statistics 2, 1152-1174. (The Antoniak paper was located only as a scanned image PDF with no text layer and could not be read; the Ghosal and van der Vaart statement is a primary textbook source.)

Usage

Dpsing(partition, alpha = 1, n = NULL)

Arguments

partition

Vector of cluster labels, one per observation; only the number of distinct labels matters. May instead be a single integer cluster count, in which case n is required.

alpha

Dirichlet-process concentration M. Default 1.

n

Sample size; inferred from partition when it is a vector.

Value

list: K, E_K, var_K, z, p_value, p_normal, alpha, n, method.

Details

Proposition 4.8: for an atomless base measure of total mass M the indicators D_i of "observation i is a new value" are INDEPENDENT Bernoulli with P(D_i = 1) = M/(M + i - 1), so K_n = sum_i D_i has exact moments E(K_n) = sum_i M/(M + i - 1) and var(K_n) = sum_i M(i-1)/(M + i - 1)^2, is asymptotically normal, and is close in total variation to Poisson(E K_n).

Since the D_i are independent Bernoulli with known unequal probabilities, the exact null law of K_n is Poisson-binomial, and it is convolved directly here rather than approximated – a fixed n-step recursion with no sampling and no early exit, so the two-sided p-value is exact and identical in both language arms. The normal-approximation deviate is reported alongside.

Examples

Dpsing(c(1, 1, 2, 2, 3, 3, 4, 4, 5, 5), alpha = 1)$p_value
#> [1] 0.09451168