Optimal and maximin procedures for multiple testing problems

Saharon Rosset*, Ruth Heller, Amichai Painsky, Ehud Aharoni

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

4 Scopus citations

Abstract

Multiple testing problems (MTPs) are a staple of modern statistical analysis. The fundamental objective of MTPs is to reject as many false null hypotheses as possible (that is, maximize some notion of power), subject to controlling an overall measure of false discovery, like family-wise error rate (FWER) or false discovery rate (FDR). In this paper we provide generalizations to MTPs of the optimal Neyman-Pearson test for a single hypothesis. We show that for simple hypotheses, for both FWER and FDR and relevant notions of power, finding the optimal multiple testing procedure can be formulated as infinite dimensional binary programs and can in principle be solved for any number of hypotheses. We also characterize maximin rules for complex alternatives, and demonstrate that such rules can be found in practice, leading to improved practical procedures compared to existing alternatives that guarantee strong error control on the entire parameter space. We demonstrate the usefulness of these novel rules for identifying which studies contain signal in numerical experiments as well as in application to clinical trials with multiple studies. In various settings, the increase in power from using optimal and maximin procedures can range from 15% to more than 100%.

Original languageEnglish
Pages (from-to)1105-1128
Number of pages24
JournalJournal of the Royal Statistical Society. Series B: Statistical Methodology
Volume84
Issue number4
DOIs
StatePublished - Sep 2022

Funding

FundersFunder number
Israel Science Foundation2180/20

    Keywords

    • FDR
    • FWER
    • infinite linear programming
    • multiple comparisons
    • optimal testing
    • strong control

    Fingerprint

    Dive into the research topics of 'Optimal and maximin procedures for multiple testing problems'. Together they form a unique fingerprint.

    Cite this