Categorical Omega for Ordered-Categorical Items, With a Confidence Interval

Estimates categorical omega ($\omega_C$; Green & Yang, 2009; Kelley & Pornprasertmanit, 2016) for a homogeneous composite of ordered-categorical items and returns a bootstrap confidence interval.

Usage

reliability_omega_categorical(
  data,
  ci_method = c("bca", "percentile", "bootstrap_se", "bootstrap_se_logistic", "none"),
  conf_level = 0.95,
  B = 10000,
  seed = NULL
)

Arguments

data: A numeric matrix or data frame of ordered-categorical item scores (integer codes for the categories). Rows with any missing values are listwise-deleted.
ci_method: Method for constructing the confidence interval. See Details. When not supplied, no interval is computed: every interval for categorical omega is bootstrap based, and a bootstrap is never run unless requested. Ask for "bca" (the recommended method) or "percentile" to obtain one.
conf_level: Confidence level for the interval. Defaults to 0.95.
B: Number of bootstrap replications. Defaults to 10000.
seed: Random number seed used for bootstrap reproducibility. Defaults to NULL, which leaves the user's current RNG state intact; supply an integer for reproducibility.

Value

A data.frame with columns term and value and rows "estimate" (sample $\omega_C$), "se" (the bootstrap standard deviation across replications, already on the coefficient scale; NA for ci_method = "none"), "lower_limit" and "upper_limit" (clamped to [0, 1]), "conf_level", "N", "N_complete" (the complete cases; equal to "N" here, and carried so the whole reliability family returns one shape), and "J". Attributes coefficient ("omega_categorical"), ci_method, and B record the computation. A bootstrap call also carries the attribute bootstrap, a list recording the replications the interval conditions on: attempted (B), returned (the replications that returned a finite value; the standard error is their standard deviation and the percentile and BCa limits are their quantiles), failed (the difference), reasons (a named count of why replications were set aside: error when the refit raised an error, non_finite when it returned no finite value, as a fit that did not converge does), error_messages (the distinct error messages with their counts, empty when none), acceleration and acceleration_method (the BCa acceleration used and whether it came from the regression or the jackknife; NA unless a BCa interval was formed), and jackknife (the same attempted, returned, failed, reasons, and error_messages for the leave-one-out refits behind the acceleration, all zero when no jackknife ran). The printed table shows these counts in one line, and a warning states them whenever a replication was set aside.

Details

Categorical omega is designed for items measured on an ordered categorical scale (e.g., Likert items). It uses a probit-link single-factor model in which each observed item $X_j$ is modeled as a categorization of an underlying continuous response variable $X_j^{*}$ via thresholds $t_{j,c}$ (Muthén, 1984; Millsap & Yun-Tein, 2004), fit by diagonally weighted least squares with mean- and variance-adjusted test statistic (WLSMV), the standard estimator for ordered categorical items. With the delta parameterization ($Var(X_j^{*}) = 1$), the population categorical omega is $$\omega_C = \frac{\sum_{j=1}^{J} \sum_{j'=1}^{J} \sigma_{jj'}\!\left(\lambda_{j}\lambda_{j'}\right)}{\sum_{j=1}^{J} \sum_{j'=1}^{J} \sigma_{jj'}\!\left(\rho_{X_{j}^{*} X_{j'}^{*}}\right)},$$ where $\sigma_{jj'}(r)$ is the model implied covariance of $(X_{j}, X_{j'})$ computed from a bivariate normal CDF over pairs of category thresholds and a correlation $r$ (Green & Yang, 2009, Eq. 13–14, with the full coefficient their Eq. 21; Kelley & Pornprasertmanit, 2016, Eq. 17–18). The numerator uses model implied polychoric correlations ($\lambda_j \lambda_{j'}$), while the denominator uses observed polychoric correlations estimated from the data via a saturated bivariate model.

Kelley and Pornprasertmanit (2016) found in extensive Monte Carlo simulation that the bias-corrected and accelerated (BCa) bootstrap confidence interval for categorical omega achieved acceptable coverage across a wide variety of threshold patterns, sample sizes, item counts, and population reliability values, with one documented exception: coverage dipped somewhat below the acceptable range when the number of items and the population reliability were both high. They specifically recommend BCa for categorical omega. Because no bootstrap runs in DMAR unless the user requests one, the default output is the point estimate with a message naming the call that produces the recommended interval; request ci_method = "bca" to obtain it. The interval is not a quick one. Every replication refits the ordered-categorical factor model and the saturated polychoric model behind the denominator, a fraction of a second for a scale of moderate length, so the recommended interval at the default B = 10000 runs for tens of minutes; when B is below the number of cases, the jackknife behind the BCa acceleration adds one such refit per case (at the default B the acceleration is read from the replicates themselves at no extra cost; see the "bca" entry below). The example on this page therefore stops at the point estimate; the reported analysis adds ci_method = "bca" to the same call, with seed for reproducibility.

When to use. Use reliability_omega_categorical when items are ordered-categorical, especially when (a) the number of categories is small (e.g., two to five), (b) item distributions are skewed, or (c) threshold patterns differ markedly across items. In Kelley and Pornprasertmanit's (2016) Study 3, treating ordered items as continuous and using reliability_omega with the observed total variance in the denominator achieved acceptable coverage only when threshold patterns were similar across items; in their experience that condition is rare in practice.

Available confidence interval methods (set via ci_method). Every interval here is bootstrap based: the rows of data are resampled with replacement B times (10000 by default) and categorical omega is recomputed, with the full WLSMV model refit, on each replication (Efron & Tibshirani, 1993). Replications whose refit fails or does not converge, most common with small samples and sparse response categories, are set aside, and the interval is computed from the replications that return a value: the standard error is their standard deviation, and the percentile and BCa limits are quantiles of the returned replicates, not of all B. How many were attempted, how many returned a value, and why the rest were set aside is recorded in the bootstrap attribute of the result (see Value), printed in one line under the table, and stated in a warning whenever a replication was set aside, so an interval that conditions on a subset of the replications says so and by how much. Bootstrap results vary from run to run; supply seed for reproducibility.

"bca": The bias-corrected and accelerated bootstrap, the default and the specific recommendation of Kelley and Pornprasertmanit (2016). Where the percentile interval reads its limits directly off the empirical quantiles of the bootstrap estimates, BCa adjusts the two quantile positions for median bias (estimated from the bootstrap distribution) and for the rate at which the estimator's variance changes with the parameter (the acceleration, estimated by the jackknife), making it second-order accurate where the percentile interval is first-order accurate (DiCiccio & Efron, 1996). The adjusted quantile positions sit farther into the tails than the percentile interval uses, which is why the default B = 10000 is larger than the customary 2000; reduce B for exploration, not for a reported analysis. The acceleration is estimated as boot estimates it: by regressing the bootstrap replicates on the resampling frequencies when at least as many replications as cases returned a value, at no extra cost, and otherwise by the jackknife, one leave-one-out refit per case. A leave-one-out refit that fails (when a case is the only respondent in one of an item's categories, for example) is counted and set aside, and the acceleration is estimated from the refits that returned a value; through version 1.0.0 one such failure made the acceleration NA and stopped the call with “estimated adjustment 'a' is NA”. When the acceleration cannot be estimated at all (fewer than two leave-one-out refits returned a value), or the bias correction cannot be (every returned replicate on one side of the estimate), the percentile interval is returned in its place with a warning, and the ci_method attribute records the substitution as "percentile (BCa acceleration could not be estimated: ...)". The counts of leave-one-out refits and the acceleration used are in the bootstrap attribute.
"percentile": Percentile bootstrap: the interval limits are the empirical quantiles of the bootstrap estimates.
"bootstrap_se", "bootstrap_se_logistic": Wald intervals using the bootstrap standard deviation as a standard error, built on the logit scale for the _logistic variant so the endpoints respect [0, 1].
"none": Return only the point estimate.

Comparison with other packages. The psych package's omega fits a Schmid-Leiman hierarchical factor model on continuous (or treated-as-continuous) items and does not implement categorical omega in the sense of Green and Yang (2009). For ordered-categorical items reliability_omega_categorical is the appropriate choice; polychoric provides polychoric correlation estimation as a separate tool.

References

DiCiccio, T. J., & Efron, B. (1996). Bootstrap confidence intervals. Statistical Science, 11(3), 189–228.

Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York, NY: Chapman & Hall/CRC.

Green, S. B., & Yang, Y. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74, 155–167. doi:10.1007/s11336-008-9099-3

Kelley, K., & Cheng, Y. (2012). Estimation of and confidence interval formation for reliability coefficients of homogeneous measurement instruments. Methodology, 8, 39–50. doi:10.1027/1614-2241/a000036

Kelley, K., & Pornprasertmanit, S. (2016). Confidence intervals for population reliability coefficients: Evaluation of methods, recommendations, and software for composite measures. Psychological Methods, 21, 69–92. doi:10.1037/a0040086

Maxwell, S. E., Delaney, H. D., & Kelley, K. (2027). Designing experiments and analyzing data: A model comparison perspective (4th ed.). Routledge.

Millsap, R. E., & Yun-Tein, J. (2004). Assessing factorial invariance in ordered-categorical measures. Multivariate Behavioral Research, 39(3), 479–515. doi:10.1207/s15327906mbr3903_4

Muthén, B. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115–132.

Terry, L. J., & Kelley, K. (2012). Sample size planning for composite reliability coefficients: Accuracy in parameter estimation via narrow confidence intervals. British Journal of Mathematical and Statistical Psychology, 65, 371–401. doi:10.1111/j.2044-8317.2011.02030.x

Author

Ken Kelley kkelley@nd.edu

Examples

set.seed(113)
# Six 5-category items with a single latent factor.
N <- 200
J <- 6
loadings <- rep(0.7, J)
eta <- rnorm(N)
latent <- outer(eta, loadings) +
          matrix(rnorm(N * J), N, J) %*% diag(sqrt(1 - loadings^2))
items <- apply(latent, 2, function(x)
  as.integer(cut(x, breaks = c(-Inf, -1.5, -0.5, 0.5, 1.5, Inf),
                 labels = FALSE)))
colnames(items) <- paste0("y", seq_len(J))

# Default: point estimate only, with a message naming the call that
# produces the recommended interval.
reliability_omega_categorical(data = items)
#> Categorical omega is reported without a confidence interval by default because its interval is bootstrap based. Request it with ci_method = "bca" (the recommended method) or "percentile"; B = 10000 replications is the default when you do.
#>  term        value
#>  estimate    0.832
#>  se          <NA> 
#>  lower_limit <NA> 
#>  upper_limit <NA> 
#>  conf_level  0.95 
#>  N           200  
#>  N_complete  200  
#>  J           6    

# The same items treated as continuous, for contrast. With five
# categories and thresholds spread across the latent scale the two
# coefficients nearly agree on these data; the gap widens as the
# categories get coarser and as the thresholds move into the tails,
# which is where the categorical coefficient is worth its cost.
reliability_omega(data = items)
#> Robust omega is reported without a confidence interval by default because its interval is bootstrap based. Request it with ci_method = "percentile" (or "bca"); B = 10000 replications is the default when you do.
#>  term        value
#>  estimate    0.83 
#>  se          <NA> 
#>  lower_limit <NA> 
#>  upper_limit <NA> 
#>  conf_level  0.95 
#>  N           200  
#>  N_complete  200  
#>  J           6