Necessary Sample Size to Reach Desired Power for a Welch (Unequal Variance) t Test Using a Publication Bias and Uncertainty Correction Procedure

ss_buc_welch_t returns the necessary per-group sample size to achieve a desired level of statistical power for a planned study using a Welch t test, the two-group comparison that does not assume equal variances, based on information obtained from a previous study. The effect from the previous study can be corrected for publication bias and/or uncertainty to provide a sample size that will achieve more accurate statistical power for a planned study, when compared to approaches that use a sample effect size at face value or rely on sample size only. The bias and uncertainty adjusted previous study noncentrality parameter is also returned, together with the planning effect size, the effect size the plan is built on, as a standardized mean difference.

Usage

ss_buc_welch_t(
  t_observed,
  n_1,
  n_2,
  sd_ratio = 1,
  sd_1,
  sd_2,
  var_1,
  var_2,
  alpha_prior = 0.05,
  alpha_planned = 0.05,
  assurance = 0.8,
  desired_power = 0.8
)

Arguments

t_observed: Observed Welch t value from a previous study used to plan sample size for a planned study. Either sign is accepted.
n_1, n_2: Group sample sizes of the previous study.
sd_ratio: Ratio of the second group's standard deviation to the first group's, in the previous study, assumed to hold in the planned study as well. A value of 1 means equal variances. Give this, or the two group standard deviations, or the two group variances, but not more than one of the three.
sd_1, sd_2: Group standard deviations of the previous study, an alternative to sd_ratio. Only their ratio enters the computation, so the unit does not matter.
var_1, var_2: Group variances of the previous study, a second alternative to sd_ratio.
alpha_prior: The significance level \(\alpha\) a prior result had to reach to be published (more generally, to reach you): the correction assumes you see only results with p below alpha_prior. Enter a number above 0 and at most 1. A value of 1, which may also be written "No Publication Filter", means no publication filter: every p value is at most 1, so any result would have been published whatever its p value, there is no publication bias to correct, and only the correction for uncertainty (set by assurance) remains, as for a pilot study. Anderson, Kelley, and Maxwell (2017, p. 1552) note that this level, their \(\alpha_P\), can be set equal to 1 when publication bias is thought not to apply. The string is read without regard to case, and the result echoes it as an alpha_prior of 1.
alpha_planned: Alpha level \(\alpha\) assumed for the planned study.
assurance: Desired level of assurance: the long run proportion of planned studies whose statistical power at the true effect reaches or surpasses desired_power, when each is planned from a prior study published under the rule alpha_prior describes. The adjusted noncentrality parameter is the lower limit of a one-sided confidence interval at this level, computed from the publication-truncated distribution of the prior statistic, so assurance = .5 plans from a median unbiased estimate (correcting for publication bias only), a larger value adds the correction for uncertainty, and a value below .5 is not recommended. The term follows Kelley and Maxwell (2003), who used assurance in sample size planning for the probability that a confidence interval will be no wider than desired. With alpha_prior = 1 and assurance = .80 the plan corresponds to safeguard power, which plans for the lower limit of a two-sided 60 percent confidence interval around the prior effect size rather than for the effect size itself (see BUCSS-package for how, and for where the two differ). Enter it as a proportion in (0, 1) or as a percentage greater than 1 (for example, 80 is read as .80); a value of exactly 1 is read as 1 percent. A percentage is echoed in the result's planning-input rows as the coerced proportion (80 is echoed as .80); the same applies to desired_power.
desired_power: Desired level of statistical power for the planned study: the probability that its test rejects the null hypothesis when the effect is present (Maxwell, Delaney, & Kelley, 2027). It must exceed alpha_planned: a test's power is never below its Type I error rate, so a value at or below alpha_planned is not allowable, and the function stops rather than return a plan. Enter it as a proportion in (0, 1) or as a percentage greater than 1; a value of exactly 1 is read as 1 percent (.01), which is then refused unless alpha_planned is below .01.

Value

An object of class bucss_plan: a data.frame with a character term column and a numeric value column, in the same shape as the sibling package DMAR's planners. The design results come first: the necessary per-group sample size for the planned study, named for its unit (necessary_n_per_group, necessary_n_per_cell, or necessary_N); for the per-group and per-cell designs, total_N, the implied total sample size; actual_power, the power of the plan returned, that is, of the returned sample size at the returned noncentrality parameter (see Details); ncp_adjusted, the publication bias and uncertainty adjusted prior study noncentrality parameter, on the scale of the t statistic (the noncentrality parameter of the prior study's noncentral t); and the planning effect size, smd_planning, a standardized mean difference (a magnitude) whose standardizer is the root mean square of the two groups' standard deviations (Bonett, 2008), \(\lambda\sqrt{(1/n_1 + r^2/n_2)/((1 + r^2)/2)}\), where \(\lambda\) is ncp_adjusted, \(n_1\) and \(n_2\) are the prior group sizes, and r is the ratio of standard deviations; with n per group in the planned study its noncentrality parameter is this value times \(\sqrt{n/2}\), as for the equal-variance test, and with a ratio of 1 it is Cohen's \(\delta\).

A planning effect size (the row whose name ends in _planning) is an adjusted effect size: the value, in the design's usual effect size metric, used to plan the sample size given the situation at hand (the prior study's result, the publication bias assumed through alpha_prior, and the uncertainty allowed for through assurance). It is ncp_adjusted restated in that metric and is what the plan is built on, not an estimate of the population effect to report: above an assurance of .5 it is deliberately conservative (a lower confidence limit), and at .5 it is the median unbiased estimate. Because it restates ncp_adjusted, actual_power is the ordinary power of a study of the returned sample size when the population effect is the planning effect size.

Rows echoing the planning inputs follow the design results, so the assumptions travel with the result. The design, the unit the sample size is counted in, the largest assurance this prior can support (the assurance ceiling), and the planned test's degrees of freedom (df_effect, and df_error where the test has a denominator) travel on attributes. print.bucss_plan shows the design, for the regression planners the predictor frame, the unit, and the ceiling beneath the table; tidy() and glance() give the compact and wide views, the latter with every attribute as a column.

Details

Researchers often use the sample effect size from a prior study as an estimate of the likely size of an expected future effect in sample size planning. However, sample effect size estimates should not usually be used at face value to plan sample size, due to both publication bias and uncertainty.

This planner is an approximation, and needs one assumption the others do not. A Welch t statistic (Welch, 1947) is not exactly distributed as a noncentral t: its denominator degrees of freedom are estimated from the data by the Welch-Satterthwaite formula (Satterthwaite, 1946; Welch, 1947), and they depend on the ratio of the two population standard deviations. ss_buc_welch_t treats the statistic as noncentral t on the Welch-Satterthwaite degrees of freedom implied by the prior study's group sizes and the standard deviation ratio you supply, which is the usual approximation in power analysis for this test, and computes the planned study's power the same way. A minimum planned sample size of 10 per group is imposed, because of estimation accuracy issues at smaller sample sizes. With fewer per group the approximation is not accurate enough: the real test estimates its two variances, and so its degrees of freedom, from too few observations for the noncentral t to describe it, and the approximation overstates its power, badly so for the smallest (a plan of 2 per group that it puts at .82 has real power near .50). From 10 per group, at an alpha_planned of .05, the approximation is accurate: across ratios from 1 to 5, the power of the real Welch test is within .006 of what this function reports for every plan it returns, and within .003 from about 12 per group at a target power of .80, and its Type I error stays between .047 and .054 against a nominal .05 across ratios from 1 to 3. At a stricter alpha_planned the approximation overstates the power of the smallest plans by more, at 10 per group by up to about .012 at .01, .016 at .005, and .025 at .001, and it is within .006 again from 14, 16, and 20 per group, respectively. When the target power is reached with fewer than 10 per group, the planner returns 10, whose power exceeds the target.

The correction of the prior study's statistic uses the same approximation and holds up well unless a prior group is very small. Simulating prior studies as raw data, published by the real Welch test, the corrected parameter fell at or below the truth in about .79 to .81 of them at an assurance of .80 when each prior group had 10 to 20 participants (a refusal counted as at or below). With 5 to 8 in the smaller prior group it ranged from about .78 to .86 when the true ratio was supplied, and fell as low as about .73 when the ratio was taken from the prior study's own standard deviations. One consequence of reading the prior study on the Welch-Satterthwaite degrees of freedom the supplied ratio implies, rather than on those its data produced: when the prior groups are small and unequal in size or in spread, including at the default ratio of 1, a prior result its paper reported as significant can fall short of the critical value here and be refused as nonsignificant. Such a result is typically close to the publication threshold, where it could support little assurance in any case.

The assumption that matters is the ratio itself. One sd_ratio describes both studies: the prior study, whose statistic is read under it, and the planned study. Getting it wrong costs far more than the distributional approximation does. For the example below, the plan built assuming equal variances (213 per group) delivers about .76 power rather than .80 if the ratio is in fact 2 in both studies, and about .43 if the prior study's groups were equally variable but the planned study's ratio is 2, with the mean difference held fixed in units of the first group's standard deviation. Re-running across a range of plausible ratios, and planning for the least favorable one you find credible, guards against the first kind of error. The second, a planned study more unequal than the prior one, is not something the prior result can speak to, and needs its own allowance. The required sample size does not always rise as the two standard deviations move apart: when the group assumed to be more variable is the smaller one it often falls, though not always (for the prior study of the example below it rises).

When the ratio is 1 and the prior groups were the same size, this planner returns what ss_buc_independent_t returns, except for the minimum of 10 per group imposed here. With unequal prior group sizes the two differ, and should: the equal-variance planner reduces the prior study to an equivalent equal-n design at the harmonic mean, while this one uses the actual Welch-Satterthwaite degrees of freedom, which are smaller when the groups are unbalanced. The same prior t is then less compelling, and the plan is usually larger when one prior group is more than about twice the size of the other; when the two sizes are close it is more often the same, and it can be smaller, because the equal-variance planner rounds the harmonic mean down and up to whole group sizes and returns the larger of the two plans.

The approach uses a likelihood function of a truncated noncentral F distribution, where the truncation occurs due to small effect sizes being unobserved due to publication bias. The numerator of the likelihood function is the density of a noncentral F distribution. The denominator is the power of the test, which serves to truncate the distribution. In the two-group case, this formula reduces to the density of a truncated noncentral t distribution. (See Taylor & Muller, 1996, Equation 2.1; Anderson, Kelley, & Maxwell, 2017, Equation 3; and Anderson & Maxwell, 2017, for more details.)

Assurance is the proportion of times that power will be at or above the desired level, if the experiment were to be reproduced many times. For example, assurance = .5 means that power will be above the desired level half of the time, but below the desired level the other half of the time. Selecting assurance = .5 (selecting the noncentrality parameter at the 50th percentile of the likelihood distribution) results in a median unbiased estimate of the population noncentrality parameter and does not correct for uncertainty. In order to correct for uncertainty, assurance > .5 can be selected, which corresponds to selecting the noncentrality parameter associated with the (1 - assurance) quantile of the likelihood distribution.

alpha_prior is the significance level a prior result had to reach to be published: the correction assumes that only results with p values below alpha_prior reach the literature, and so reach you. Different \(\alpha\) levels represent differing amounts of publication bias. For example, setting alpha_prior = .20 assumes that studies with p values less than .20 are published, whereas those with larger p values are not, which reflects less severe publication bias than the default of .05. If the previous study of interest has not been subjected to publication bias (e.g., a pilot study), set alpha_prior = 1, which may also be written alpha_prior = "No Publication Filter"; Anderson, Kelley, and Maxwell (2017, p. 1552) note that it can be set equal to 1 when publication bias is thought not to apply. Because no p value exceeds 1, this means no publication filter at all: any result would have been published whatever its p value, so there is no publication bias to correct, and only the correction for uncertainty, set by assurance, remains.

In some cases, the corrected noncentrality parameter for a given level of assurance will be estimated to be zero. This is an indication that, at the desired level of assurance, the previous study's effect cannot be accurately estimated due to high levels of uncertainty and bias. When this happens, subsequent sample size planning is not possible with the chosen specifications, and the function stops with an informative error rather than returning a result. The error reports the largest assurance the prior result can support, which is \(1 - p/\alpha\), where p is the prior study's p value and \(\alpha\) is alpha_prior; at any higher assurance the corrected noncentrality parameter is driven to zero. Two remedies are available. First, users can select a lower value of assurance (e.g., .8 instead of .95), at or below the reported ceiling. Second, users can reduce the influence of publication bias by setting alpha_prior at a value greater than .05, which raises the ceiling. Note that alpha_prior is not a property of the prior study, which is fixed, but the analyst's assumption about the publication threshold its literature faced. When the ceiling is at or below the recommended floor of .5, lowering assurance cannot help and alpha_prior is the only remaining lever. It is possible to correct for uncertainty only by setting alpha_prior = 1 and choosing the desired level of assurance. We encourage users to make the adjustments as minimal as possible.

The returned actual_power is the statistical power of the plan this function reports: the power of a study of the returned sample size when the effect is the returned bias and uncertainty adjusted noncentrality parameter. Because a sample size must be a whole number, actual_power is never below the desired_power that was requested and is usually a little above it.

The observed t may be entered with either sign; the publication rule the correction assumes is two-sided, so only the magnitude enters the computation. The planned study is assumed to have equal group sizes, which is the allocation that maximizes power when the variances are equal and is a reasonable default otherwise. It is not the optimal allocation when they are not: sampling in proportion to the standard deviations would need about 10 percent fewer participants in total at a ratio of 2, and about 20 percent fewer at a ratio of 3.

References

Anderson, S. F., & Kelley, K. (2024). Sample size planning for replication studies: The devil is in the design. Psychological Methods, 29, 844–867. doi:10.1037/met0000520

Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017). Sample-size planning for more accurate statistical power: A method adjusting sample effect sizes for publication bias and uncertainty. Psychological Science, 28, 1547–1562. doi:10.1177/0956797617723724

Anderson, S. F., & Maxwell, S. E. (2017). Addressing the "replication crisis": Using original studies to design replication studies with appropriate statistical power. Multivariate Behavioral Research, 52, 305–324. doi:10.1080/00273171.2017.1289361

Kelley, K., & Maxwell, S. E. (2003). Sample size for multiple regression: Obtaining regression coefficients that are accurate, not simply significant. Psychological Methods, 8, 305–321. doi:10.1037/1082-989X.8.3.305

Maxwell, S. E., Delaney, H. D., & Kelley, K. (2027). Designing experiments and analyzing data: A model comparison perspective (4th ed.). Routledge.

Taylor, D. J., & Muller, K. E. (1996). Bias in linear model power and sample size calculation due to estimating noncentrality. Communications in Statistics: Theory and Methods, 25, 1595–1610. doi:10.1080/03610929608831787

Bonett, D. G. (2008). Confidence intervals for standardized linear contrasts of means. Psychological Methods, 13, 99–109. doi:10.1037/1082-989X.13.2.99

Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2, 110–114. doi:10.2307/3002019

Welch, B. L. (1947). The generalization of 'Student's' problem when several different population variances are involved. Biometrika, 34, 28–35. doi:10.1093/biomet/34.1-2.28

Author

Ken Kelley (kkelley@nd.edu) and Samantha F. Anderson (samantha.f.anderson@asu.edu)

Examples

# A prior study with 40 and 55 participants whose second group was half
# again as variable as the first.
result <- ss_buc_welch_t(t_observed = 3, n_1 = 40, n_2 = 55,
  sd_ratio = 1.5, alpha_prior = .05, alpha_planned = .05,
  assurance = .80, desired_power = .80)
result
#>  term                  value
#>  necessary_n_per_group 222  
#>  total_N               444  
#>  actual_power          0.801
#>  ncp_adjusted          1.32 
#>  smd_planning          0.267
#>  t_observed            3    
#>  n_1                   40   
#>  n_2                   55   
#>  sd_ratio              1.5  
#>  alpha_prior           0.05 
#>  alpha_planned         0.05 
#>  assurance             0.8  
#>  desired_power         0.8  
#> 
#> Design: Welch (unequal variance) t test
#> Sample size unit: per group
#> Largest supportable assurance: .93

# The same study described by the group standard deviations, or by the
# group variances, rather than by their ratio.
ss_buc_welch_t(t_observed = 3, n_1 = 40, n_2 = 55, sd_1 = 8.4, sd_2 = 12.6)
#>  term                  value
#>  necessary_n_per_group 222  
#>  total_N               444  
#>  actual_power          0.801
#>  ncp_adjusted          1.32 
#>  smd_planning          0.267
#>  t_observed            3    
#>  n_1                   40   
#>  n_2                   55   
#>  sd_1                  8.4  
#>  sd_2                  12.6 
#>  alpha_prior           0.05 
#>  alpha_planned         0.05 
#>  assurance             0.8  
#>  desired_power         0.8  
#> 
#> Design: Welch (unequal variance) t test
#> Sample size unit: per group
#> Largest supportable assurance: .93
ss_buc_welch_t(t_observed = 3, n_1 = 40, n_2 = 55, var_1 = 70.56,
  var_2 = 158.76)
#>  term                  value
#>  necessary_n_per_group 222  
#>  total_N               444  
#>  actual_power          0.801
#>  ncp_adjusted          1.32 
#>  smd_planning          0.267
#>  t_observed            3    
#>  n_1                   40   
#>  n_2                   55   
#>  var_1                 70.6 
#>  var_2                 159  
#>  alpha_prior           0.05 
#>  alpha_planned         0.05 
#>  assurance             0.8  
#>  desired_power         0.8  
#> 
#> Design: Welch (unequal variance) t test
#> Sample size unit: per group
#> Largest supportable assurance: .93

# How sensitive is the plan to the assumed spread?
ss_buc_welch_t(t_observed = 3, n_1 = 40, n_2 = 55, sd_ratio = 1)
#>  term                  value
#>  necessary_n_per_group 213  
#>  total_N               426  
#>  actual_power          0.801
#>  ncp_adjusted          1.31 
#>  smd_planning          0.272
#>  t_observed            3    
#>  n_1                   40   
#>  n_2                   55   
#>  sd_ratio              1    
#>  alpha_prior           0.05 
#>  alpha_planned         0.05 
#>  assurance             0.8  
#>  desired_power         0.8  
#> 
#> Design: Welch (unequal variance) t test
#> Sample size unit: per group
#> Largest supportable assurance: .92
ss_buc_welch_t(t_observed = 3, n_1 = 40, n_2 = 55, sd_ratio = 2)
#>  term                  value
#>  necessary_n_per_group 236  
#>  total_N               472  
#>  actual_power          0.801
#>  ncp_adjusted          1.31 
#>  smd_planning          0.259
#>  t_observed            3    
#>  n_1                   40   
#>  n_2                   55   
#>  sd_ratio              2    
#>  alpha_prior           0.05 
#>  alpha_planned         0.05 
#>  assurance             0.8  
#>  desired_power         0.8  
#> 
#> Design: Welch (unequal variance) t test
#> Sample size unit: per group
#> Largest supportable assurance: .92

# Requesting more assurance than the prior result can support stops with an
# informative error naming the largest workable assurance (here near .93):
try(ss_buc_welch_t(t_observed = 3, n_1 = 40, n_2 = 55, sd_ratio = 1.5,
  assurance = .95))
#> Error : Sample size planning is not possible with these settings. After correcting the prior study's effect for publication bias and uncertainty at the requested level of assurance, the corrected noncentrality parameter is zero, which means there is not enough information in the prior result to plan the sample size as desired. With this prior result and 'alpha_prior', the largest 'assurance' that still yields a usable plan is about .93; re-run with 'assurance' at or below that value, or raise 'alpha_prior' to lift the ceiling. Re-running at a lower 'assurance' is not free: the plan it returns carries that lower assurance, not the one you first asked for, so the long run proportion of replications reaching your target power falls accordingly. 'alpha_prior' does not change the prior study, which is what it is; it is your assumption about the publication threshold the study's literature faced, so raising it toward 1 assumes less publication bias to undo (for example, 'alpha_prior = .10' assumes the literature's journals would have published a result significant at the .10 level, a more lenient policy than .05, and 'alpha_prior = 1' assumes no publication bias at all). See Anderson, Kelley, and Maxwell (2017) and Anderson and Kelley (2024).