A taxonomy of reliability indices for cluster‐level constructs measured by multiple individual‐level indicators

Abstract

Abstract Generalizability theory (GT) provides a variety of intraclass correlation coefficients (ICCs) to quantify reliability across different conditions of measurement—for example, scale and interrater reliability quantify generalization across items and raters, respectively. In a multilevel modelling (MLM) context, an ICC is often used to quantify the proportion of a subject‐level variable's variance that is attributable to cluster‐level differences (e.g., between schools or neighbourhoods). When data from cluster members are their ratings of a group‐level characteristic, the ICC is an inter‐rater reliability coefficient, interpreted as the degree to which observations of a cluster generalize across members of that cluster. Structural equation modelling (SEM) is becoming a popular framework for modelling GT, and the ‘omega’ indices of composite reliability—which are functions of SEM parameters—are special cases of GT (i.e., how measurements of a construct generalize across indicators). The merger of MLM and SEM (i.e., ML‐SEM) provides a unique opportunity to explore how well measurements of cluster‐level constructs generalize across multiple measurement conditions (indicators, cluster members, or both). A taxonomy of ICCs is explicated to appropriately quantify reliability across aspects of a procedure for measuring cluster‐level constructs, adding value to the limited ICCs currently considered in ML‐SEM applications.

The authors' abstract, as published at the source. British Journal of Mathematical and Statistical Psychology, 2026 · DOI ↗