Quantifying Replicability and Consistency in Systematic Reviews

Iman Jaljuli*, Yoav Benjamini, Liat Shenhav, Orestis A. Panagiotou, Ruth Heller

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

5 Scopus citations


Systematic reviews and meta-analyses are important tools for synthesizing evidence from multiple studies. They serve to increase power and improve precision, in the same way that large studies can do, but also to establish the consistency of effects and replicability of results across studies. In this work we propose statistical tools to quantify replicability of effect signs (or directions) and their consistency. We suggest that these tools accompany the fixed-effect or random-effects meta-analysis, and we show that they convey important information for the assessment of the intervention under investigation. We motivate and demonstrate our approach and its implications by examples from systematic reviews from the Cochrane Library. Our tools make no assumptions on the distribution of the true effect sizes, so their inferential guarantees continue to hold even if the assumptions of the fixed-effect or random-effects models do not hold. We also develop a version of this tool under the fixed-effect assumption for cases where it is crucial and justified.

Original languageEnglish
Pages (from-to)372-385
Number of pages14
JournalStatistics in Biopharmaceutical Research
Issue number2
StatePublished - 2023


FundersFunder number
Baroness Ariane de Rothschild Women’s
US-Israel BSF2016746
National Science Foundation
Agency for Healthcare Research and Quality1R03HS025840
Seventh Framework Programme294519
Israel Science Foundation1049/16


    • Cochrane collaboration
    • Drug discovery
    • Heterogeneity
    • Meta-analysis
    • Partial conjunction analysis
    • r-value


    Dive into the research topics of 'Quantifying Replicability and Consistency in Systematic Reviews'. Together they form a unique fingerprint.

    Cite this