Simple Effect F Tests for a Two-Factor Between-Subjects Design
Source:R/simple_effects_AB.R
simple_effects_AB.RdGiven a fitted aov or lm object
for a two-factor between-subjects design, conventionally written
\(Y \sim A \times B\), where \(A\) and \(B\) are crossed fixed
factors, computes the family of simple main effects: the
effect of \(A\) at each level of \(B\) and/or the effect of \(B\)
at each level of \(A\). Each row carries the simple effect F
test, its (optionally adjusted) p-value, and the partial
\(\eta^2\) with a noncentrality-based confidence interval. The error
term can be either the full-model pooled \(\mathit{MS}_W\) (the
textbook default) or a Welch–Satterthwaite test refitted within each
conditioning level (robust to within-level heteroscedasticity).
Usage
simple_effects_AB(
object,
which = "both",
error_term = "pooled",
adjust = "none",
conf_level = 0.95
)Arguments
- object
A fitted
aovorlmobject whose right-hand side has exactly two crossed factors (e.g.\iq_gain ~ treatment * grade). The interaction term is strongly recommended so that the pooled error is the pure within-cell \(\mathit{MS}_W\); the function still runs without it but issues a warning (the additive-model residual includes interaction variance and inflates the error term used for the pooled simple effect F).- which
Which family of simple effects to report:
"both"(default)The \(b\) tests of \(A\) at each level of \(B\) followed by the \(a\) tests of \(B\) at each level of \(A\) (\(a + b\) rows).
"A_at_B"Only the \(b\) tests of the first factor at each level of the second.
"B_at_A"Only the \(a\) tests of the second factor at each level of the first.
The first factor on the right-hand side of the model formula is treated as \(A\); the second as \(B\).
- error_term
Error-term strategy for the simple effect F:
"pooled"(default)Use the full factorial model's \(\mathit{MS}_W\) and its residual df. This is Maxwell, Delaney, and Kelley's preferred default and gives the simple effect F maximum denominator df. Validity rests on homogeneity of variance across all \(a \times b\) cells.
"welch"Refit a Welch–Satterthwaite one-way test on only the data at the conditioning level (using
oneway.testwithvar.equal = FALSE). Robust to heteroscedasticity within the conditioning level at the cost of fewer (and fractional) denominator df.
- adjust
Multiple-comparison adjustment applied to the p-values of the entire family of simple effects returned (\(a + b\) for
which = "both"). One of"none"(default),"bonferroni", or any sequential method supported byp.adjust:"holm","hochberg","BH","BY".- conf_level
Confidence level for each row's partial \(\eta^2\) interval. Default
0.95.
Value
A data.frame with one row per simple effect test and
columns
effect, focal_factor, conditioning_factor,
conditioning_level, F_value, df_effect,
df_error, p_value, p_adjusted,
partial_eta_squared, lower_limit, upper_limit,
n_at_level. Attributes error_term, adjust,
conf_level, factor_A, and factor_B record the
call options.
Details
What a simple effect is. The simple main effect of \(A\) at level \(B = b_j\) tests whether the \(a\) cell means at that single level of \(B\) differ. It is the one-way analysis of variance of \(Y\) on \(A\) restricted to observations with \(B = b_j\). The counterpart, the simple effect of \(B\) at \(A = a_i\), is defined symmetrically.
Test statistic. For the simple effect of \(A\) at \(B = b_j\), let \(\mathit{SS}_{A\,|\,b_j} = \sum_i n_{ij}\,(\bar{Y}_{ij\cdot} - \bar{Y}_{\cdot j\cdot})^2\) be the between-A sum of squares computed at that level, and let \(\mathit{MS}_{A\,|\,b_j} = \mathit{SS}_{A\,|\,b_j} / (a - 1)\).
Pooled \(\mathit{MS}_W\): \(F = \mathit{MS}_{A\,|\,b_j} / \mathit{MS}_W\) with degrees of freedom \((a - 1,\, N - ab)\), where \(\mathit{MS}_W\) and its df come from the fitted full factorial model.
Welch: \(F\) and its (fractional) denominator df come from
oneway.test(y ~ A, subset = (B == b_j), var.equal = FALSE).
Test statistics for \(B\) at \(a_i\) are computed by interchanging the two factors.
Choosing an error term. The pooled denominator borrows strength from all \(N\) observations and is the textbook default in Maxwell, Delaney, and Kelley's treatment. Its validity rests on homogeneity of variance across all \(a \times b\) cells, not merely within the conditioning level. When that assumption is doubtful , for example, if Levene's or the Brown–Forsythe test flags heteroscedasticity, or if cell variances visibly differ, the Welch option provides a level-conditional test that does not require homogeneity across cells. The trade-off is denominator df: pooled carries the full \(N - ab\) residual df, whereas Welch carries the Welch–Satterthwaite df based on the \(a\) cell variances at that level only.
Partial \(\eta^2\) and its CI. The point estimate is
$$\hat{\eta}^2_p = \frac{df_{\text{effect}}\, F}{df_{\text{effect}}\, F + df_{\text{error}}},$$
computed from the F and the df actually used in the test
(so it reflects whichever error term was chosen). The confidence
interval is built by Steiger's (2004) transformation principle: a CI
for the noncentrality parameter \(\lambda\) of the F
distribution is obtained via conf_limits_ncf and then
mapped through \(\eta^2_p = \lambda / (\lambda + N_{\text{ref}})\),
with \(N_{\text{ref}}\) taken to be the total study N for the
pooled error term (treating the simple effect as a contrast within the
full factorial design) and the level-conditional sample size
\(n_{|b_j}\) for the Welch error term (since the Welch test uses only
those observations). When the lower limit on \(\lambda\) is not
identified (i.e., the observed F is below its one-sided critical
value), the lower limit on \(\eta^2_p\) is set to 0; when the upper
limit is at infinity, the upper limit on \(\eta^2_p\) is set to 1.
Multiplicity across the family. With which = "both" the
family is the \(a + b\) simple effects returned in a single call; the
adjustment is applied to that entire family. If only one direction is
wanted, call the function twice with which = "A_at_B" and
which = "B_at_A" so each family is adjusted on its own. The
"bonferroni" adjustment is \(p_{\text{adj}} = \min(1, m\, p)\)
for \(m\) rows; the sequential methods ("holm",
"hochberg", "BH", "BY") are computed via
p.adjust.
When to perform simple effects. It is no longer required that a significant omnibus interaction precede simple effect testing (Maxwell, Delaney, & Kelley, 2027). Simple effects are informative whenever the substantive question is conditional on a level of the other factor, and the multiplicity adjustment controls the family-wise error rate independently of any interaction screen.
Scope. Only fixed-effects between-subjects designs with
exactly two crossed factors are supported. Within-subjects or
mixed designs (aovlist fits) and three- or higher-way designs
are out of scope for this function. Per-cell sample sizes may be
unequal.
References
Maxwell, S. E., Delaney, H. D., & Kelley, K. (2027). Designing experiments and analyzing data: A model comparison perspective (4th ed.). Routledge.
Kelley, K. (2007). Confidence intervals for standardized effect sizes: Theory, application, and implementation. Journal of Statistical Software, 20(8), 1–24. doi:10.18637/jss.v020.i08
Steiger, J. H. (2004). Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological Methods, 9(2), 164–182. doi:10.1037/1082-989X.9.2.164
Welch, B. L. (1951). On the comparison of several mean values: An alternative approach. Biometrika, 38, 330–336.
See also
contrast_test for within-level pairwise or custom
contrasts, eta_squared_partial and
ci_eta_squared_partial for the omnibus effect size
counterparts, conf_limits_ncf for the noncentrality
machinery, ss_power_factorial_anova for power
calculations on the omnibus factorial effects.
Other hypothesis tests:
adjusted_means(),
ancova(),
anova_within(),
ci_dunnett(),
ci_scheffe(),
ci_tukey_kramer(),
compare_cov_structures(),
contrast_test(),
correlations_test(),
equivalence_r(),
equivalence_smd(),
factorial_anova(),
manova_split_plot(),
mauchly_test(),
mixed_anova(),
obrien_test(),
pairwise_within(),
randomization_test(),
randomization_test_paired(),
regions_of_significance(),
summary_t_test(),
welch_t()
Author
Ken Kelley kkelley@nd.edu
Examples
# 2 x 3 factorial: expectancy treatment (A) x grade (B) on the
# pygmalion data. Grades 4 through 6 are omitted so the family of
# simple effects stays short enough to read at a glance.
pyg <- pygmalion[pygmalion$grade <= 3, ]
pyg$grade <- factor(pyg$grade)
fit <- aov(iq_gain ~ treatment * grade, data = pyg)
# Default: pooled MS_W, both families, no adjustment. The expectancy
# effect on IQ gain is concentrated in grades 1 and 2; at grade 3 the
# F is 0.004, so the lower limit on partial eta squared is clamped to
# 0 and the function notes the clamp in a warning.
simple_effects_AB(fit)
#> Warning: The conf_limits_ncf() lower-limit clamp fired in 1 of the simple effect rows (observed F below the alpha_lower critical value of the central F-distribution); the corresponding lower_limit on partial_eta_squared is clamped to 0. See ?conf_limits_ncf for the meaning of the clamp.
#> effect focal_factor conditioning_factor
#> treatment | grade = 1 treatment grade
#> treatment | grade = 2 treatment grade
#> treatment | grade = 3 treatment grade
#> grade | treatment = Control grade treatment
#> grade | treatment = Bloomer grade treatment
#> conditioning_level F_value df_effect df_error p_value p_adjusted
#> 1 7.39 1 157 0.0073 0.0073
#> 2 5.26 1 157 0.0231 0.0231
#> 3 0.00438 1 157 0.9473 0.9473
#> Control 3.96 2 157 0.0211 0.0211
#> Bloomer 6.46 2 157 0.0020 0.0020
#> partial_eta_squared lower_limit upper_limit n_at_level
#> 0.0449 0.00322 0.119 52
#> 0.0324 0.000173 0.1 58
#> 2.79e-05 0 0.00907 53
#> 0.048 0.000596 0.117 131
#> 0.076 0.0112 0.155 32
#>
#> Confidence level: 95%
# Only the simple effects of grade within each treatment level, with
# a Holm adjustment across that family of two tests.
simple_effects_AB(fit, which = "B_at_A", adjust = "holm")
#> effect focal_factor conditioning_factor
#> grade | treatment = Control grade treatment
#> grade | treatment = Bloomer grade treatment
#> conditioning_level F_value df_effect df_error p_value p_adjusted
#> Control 3.96 2 157 0.0211 0.0211
#> Bloomer 6.46 2 157 0.0020 0.0040
#> partial_eta_squared lower_limit upper_limit n_at_level
#> 0.048 0.000596 0.117 131
#> 0.076 0.0112 0.155 32
#>
#> Confidence level: 95%
# Welch error term: refits a Welch one-way at each conditioning
# level. The Welch denominator df fall well below the pooled 157, so
# more of the lower limits are clamped to 0.
simple_effects_AB(fit, error_term = "welch")
#> Warning: The conf_limits_ncf() lower-limit clamp fired in 4 of the simple effect rows (observed F below the alpha_lower critical value of the central F-distribution); the corresponding lower_limit on partial_eta_squared is clamped to 0. See ?conf_limits_ncf for the meaning of the clamp.
#> effect focal_factor conditioning_factor
#> treatment | grade = 1 treatment grade
#> treatment | grade = 2 treatment grade
#> treatment | grade = 3 treatment grade
#> grade | treatment = Control grade treatment
#> grade | treatment = Bloomer grade treatment
#> conditioning_level F_value df_effect df_error p_value p_adjusted
#> 1 6.87 1 9.03 0.0277 0.0277
#> 2 2.97 1 12.5 0.1096 0.1096
#> 3 0.00712 1 24.5 0.9335 0.9335
#> Control 3.21 2 80.7 0.0455 0.0455
#> Bloomer 7.48 2 14.5 0.0058 0.0058
#> partial_eta_squared lower_limit upper_limit n_at_level
#> 0.432 0 0.313 52
#> 0.192 0 0.196 58
#> 0.00029 0 0.0357 53
#> 0.0737 0 0.128 131
#> 0.508 0.0393 0.541 32
#>
#> Confidence level: 95%
# Bonferroni across the full a + b = 5-test family.
simple_effects_AB(fit, adjust = "bonferroni")
#> Warning: The conf_limits_ncf() lower-limit clamp fired in 1 of the simple effect rows (observed F below the alpha_lower critical value of the central F-distribution); the corresponding lower_limit on partial_eta_squared is clamped to 0. See ?conf_limits_ncf for the meaning of the clamp.
#> effect focal_factor conditioning_factor
#> treatment | grade = 1 treatment grade
#> treatment | grade = 2 treatment grade
#> treatment | grade = 3 treatment grade
#> grade | treatment = Control grade treatment
#> grade | treatment = Bloomer grade treatment
#> conditioning_level F_value df_effect df_error p_value p_adjusted
#> 1 7.39 1 157 0.0073 0.0365
#> 2 5.26 1 157 0.0231 0.1154
#> 3 0.00438 1 157 0.9473 1.0000
#> Control 3.96 2 157 0.0211 0.1053
#> Bloomer 6.46 2 157 0.0020 0.0101
#> partial_eta_squared lower_limit upper_limit n_at_level
#> 0.0449 0.00322 0.119 52
#> 0.0324 0.000173 0.1 58
#> 2.79e-05 0 0.00907 53
#> 0.048 0.000596 0.117 131
#> 0.076 0.0112 0.155 32
#>
#> Confidence level: 95%
# Simulated 2 x 2 design with a known interaction pattern.
set.seed(113)
d <- simulate_ancova_factorial_data(
a = 2, b = 2,
mu_y = c(50, 60, 55, 50), # crossover at B = 2
mu_x = matrix(10, nrow = 4, ncol = 1),
sigma_y = 8, sigma_x = 3, rho_y_x = 0,
n = 30
)
fit_sim <- aov(y ~ A * B, data = d)
simple_effects_AB(fit_sim, conf_level = 0.95)
#> effect focal_factor conditioning_factor conditioning_level F_value
#> A | B = 1 A B 1 31.6
#> A | B = 2 A B 2 12.2
#> B | A = 1 B A 1 20.4
#> B | A = 2 B A 2 21.1
#> df_effect df_error p_value p_adjusted partial_eta_squared lower_limit
#> 1 116 < 0.0001 < 0.0001 0.214 0.0935
#> 1 116 0.0007 0.0007 0.0953 0.0179
#> 1 116 < 0.0001 < 0.0001 0.15 0.0482
#> 1 116 < 0.0001 < 0.0001 0.154 0.051
#> upper_limit n_at_level
#> 0.33 60
#> 0.201 60
#> 0.264 60
#> 0.268 60
#>
#> Confidence level: 95%