Skip to contents

The six Shrout-Fleiss intraclass correlations from one rating table

Source: R/icc12c.R
Icc12c.Rd

Shrout, P. E. and Fleiss, J. L. (1979), "Intraclass correlations: uses in assessing rater reliability", Psychological Bulletin 86(2), 420-428, doi:10.1037/0033-2909.86.2.420. The paper was opened directly and every formula below was read off a rendered page image, not a text layer.

Usage

Icc12c(X, model = "2,k")

Arguments

X

n-by-k matrix; row i holds the k ratings of target i, one per judge, in a common judge order. The design must be complete and crossed.

model

Which of the six forms estimate reports. Default "2,k", the two-way random average-measure coefficient.

Value

List with estimate, model, icc11, icc1k, icc21, icc2k, icc31, icc3k, BMS, WMS, JMS, EMS, n, k, method.

Details

There is no such thing as "the" ICC. Six coefficients are defined on the SAME n-targets-by-k-judges table and they are different numbers: on the example of the paper itself (Table 2, p. 423) they run from .17 to .91. The choice is a statement about the design, not about the arithmetic. Case 1 (one-way): each target is rated by a DIFFERENT set of k judges, so no judge main effect exists and rater bias is pooled into within-target error. Case 2 (two-way random): the same k judges rate every target and are a RANDOM SAMPLE, so systematic judge differences are charged against reliability – absolute agreement. Case 3 (two-way mixed): these k judges are the ONLY ones of interest, so judge offsets cost nothing – consistency, not agreement, and always at least as large as Case 2. The second index is the unit reported: a single rating, or the mean of all k. ICC(3,k) is the Cronbach (1951) alpha (p. 426).

Estimators, read from the rendered pages (p. 423 and p. 426): ICC(1,1) = (BMS - WMS)/(BMS + (k-1) WMS), ICC(2,1) = (BMS - EMS)/(BMS + (k-1) EMS + k (JMS - EMS)/n), ICC(3,1) = (BMS - EMS)/(BMS + (k-1) EMS), ICC(1,k) = (BMS - WMS)/BMS, ICC(2,k) = (BMS - EMS)/(BMS + (JMS - EMS)/n), ICC(3,k) = (BMS - EMS)/BMS, with BMS between targets, WMS within target, JMS between judges and EMS the residual mean square of the targets-by-judges ANOVA (Table 1, p. 422; the values for the example are in Table 3, p. 423).

All six are returned every time, so the gap between the case a study assumed and the case it could defend is visible rather than hidden behind one number.

References

Shrout, P. E. and Fleiss, J. L. (1979), Psychological Bulletin 86(2):420-428, doi:10.1037/0033-2909.86.2.420.

Examples

# Table 2, p. 423: four ratings on six targets
X <- rbind(c(9, 2, 5, 8), c(6, 1, 3, 2), c(8, 4, 6, 8),
           c(7, 1, 2, 6), c(10, 5, 6, 9), c(6, 2, 4, 7))
round(Icc12c(X, "3,1")$estimate, 2)  # .71, Table 4, p. 424
#> [1] 0.71