Robust Variance Estimation in Meta-analysis with correlated effect sizes

Meta-analysis has grown greatly in popularity in the agricultural sciences over the past quarter-century for synthesizing the results from multiple independent studies. In a classical random-effects meta-analysis, there is one estimated effect size per study, such as a mean difference, log response ratio, log odds, slope, or correlation coefficient. Often, more than one effect size in a study may meet the criteria to be included in the dataset. For instance, in an analysis of the effects of a fungicide on plant disease control, three slightly different treatments (differences in application timing or dosage) may have been tested in some studies, all of which meet the selection criteria. Estimated effect sizes within studies are likely to be correlated, although the magnitude of the correlation may be unknown, different for each study, and almost certainly not recoverable directly from the published primary studies. One could average the effect sizes within studies or randomly select one of the treatments prior to a classical meta-analysis, although this results in a loss of information. Alternatively, one can conduct a meta-analysis that properly accounts for the multiple (correlated) estimated effect sizes in those studies.

A review of the literature across 10 disciplines reveals that there is more than one effect size in some of the studies in over 50% of published meta-analyses, but that the authors accounted for the multiple effect sizes in their analysis in less than half the cases (Wu et al. 2025 [Res. Synth. Meth.]). This may be due, in part, because specialized software, including commercial programs developed for meta-analysis, assume that each effect size comes from a separate study (at least as the default), and are thus independent. Assuming that estimated effect sizes are independent will result in artificially low estimated variances for estimated parameters (e.g., mean, coefficients of moderator variables, etc.). Starting with the work of Hedges et al. (2010 [Res. Synth. Meth.]), a recommended approach is to assume a common correlation of estimated effect sizes () to construct a within-study working variance-covariance matrix for each study (Ri), hold these matrices fixed in the analysis, and then use empirical (sandwich) estimation for the variances of estimated parameters. The classical sandwich estimate is known to be downward biased, so many bias-corrected estimators have been proposed (Gosho et al. [Int. Stat. Rev.]). Tipton (2014 [Psy. Meth.]) and Pustejovsky and Tipton (2018 [J. Bus. Econ. Stat.]) developed a new sandwich estimator and implemented it in the clubSandwich R package. However, it is not clear how the Tipton estimator compares with some of the other robust and non-robust estimators, especially those available in the GLIMMIX procedure of SAS. Using GLIMMIX with fixed Ri (based on a constant ) requires an easy “trick” to fix the residual matrix within each study (a method we have not seen previously reported).

I will compare results—such as Type I error rates and coverage percentages—for several robust and non-robust estimators when fitting different meta-analytical models to simulated datasets that are applicable in the plant sciences. The Tipton method has its greatest advantage when the number of studies is very small.