Probability of Superiority for a Paired-Samples Design
Source:R/probability_of_superiority_paired.R
probability_of_superiority_paired.RdComputes the probability-of-superiority effect size for paired
observations (Grissom & Kim, 2005, 2012), \(P_S = \Pr(Y_1 > Y_2)\),
along with an analytic confidence interval based on the
Brunner-Munzel (2000) U-statistic standard error and a Fisher-
\(\mathrm{arctanh}\) transformation to keep the bounds inside \([0, 1]\).
The paired counterpart of the Vargha-Delaney (2000) \(A\) statistic
/ cliff_delta for two independent groups.
Value
A data.frame with rows for the point estimate of
\(P_S\), the lower / upper CI bounds, the variance, and the
counts of within-pair wins / ties / losses for \(y_1\).
Details
Definition. For paired observations \((x_i, y_i)\), $$P_S \;=\; \Pr(Y > X) + 0.5 \cdot \Pr(Y = X),$$ where ties are split. The sample estimator is the proportion of pairs with \(y_i > x_i\), plus half the proportion of ties. This is the natural paired-data analog of Vargha-Delaney's \(A\) statistic and is unbiased under exchangeability of paired observations.
Why paired-specific. The independent-groups cles and
cliff_delta estimators are biased when the two samples are
paired, because their variance formulas assume independence of the
two groups. For paired data the within-pair correlation reduces the
effective sampling variance, which is captured by the Brunner-Munzel
(2000) variance used here.
Confidence interval. The standard error is built from the
within-pair sign indicators (Brunner-Munzel, 2000):
$$\mathrm{Var}(\hat P_S) \;=\;
\frac{1}{n^2}\sum_{i=1}^{n} (s_i - \bar s)^2,$$
where \(s_i = \mathrm{I}(y_i > x_i) + 0.5 \cdot \mathrm{I}(y_i = x_i)\).
The CI is built on the \(\mathrm{arctanh}(2 P_S - 1)\) scale (mapping
\(P_S \in [0, 1]\) to the real line) and back-transformed to keep
the limits inside the unit interval, exactly mirroring
cliff_delta.
References
Brunner, E., & Munzel, U. (2000). The nonparametric Behrens-Fisher problem: Asymptotic theory and a small-sample approximation. Biometrical Journal, 42(1), 17–25. doi:10.1002/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U
Grissom, R. J., & Kim, J. J. (2005). Effect sizes for research: A broad practical approach. Lawrence Erlbaum.
Grissom, R. J., & Kim, J. J. (2012). Effect sizes for research: Univariate and multivariate applications (2nd ed.). Routledge.
Vargha, A., & Delaney, H. D. (2000). A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25(2), 101–132. doi:10.3102/10769986025002101
See also
cliff_delta, cles,
proportion_of_superiority
Other effect size estimates:
cles(),
cliff_delta(),
correction_for_attenuation(),
eta_squared(),
eta_squared_generalized(),
eta_squared_partial(),
expected_partial_r(),
expected_r(),
expected_smd(),
nnt_from_smd(),
omega_squared(),
omega_squared_partial(),
proportion_of_superiority(),
responder_analysis(),
smd_trimmed()
Author
Ken Kelley kkelley@nd.edu
Examples
# 1. Paired pre/post data:
set.seed(113)
pre <- rnorm(30, mean = 100, sd = 15)
post <- pre + rnorm(30, mean = 5, sd = 10)
probability_of_superiority_paired(x = pre, y = post)
#> term value
#> probability_of_superiority 0.667
#> lower_limit 0.484
#> upper_limit 0.81
#> var_ps 0.00741
#> wins_y_over_x 20
#> losses_y_under_x 10
#> ties 0
#>
#> Confidence level: 95%