Predicting counterfactuals from large historical data and small randomized trials

Nir Rosenfeld, Yishay Mansour, Elad Yom-Tov

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

When a new treatment is considered for use, whether a pharmaceutical drug or a search engine ranking algorithm, a typical question that arises is, will its performance exceed that of the current treatment? The conventional way to answer this counterfactual question is to estimate the effect of the new treatment in comparison to that of the conventional treatment by running a controlled, randomized experiment. While this approach theoretically ensures an unbiased estimator, it suffers from several drawbacks, including the difficulty in finding representative experimental populations as well as the cost of running randomized trials. Moreover, such trials neglect the huge quantities of available control-condition data, which in principle can be utilized for the harder task of predicting individualized effects. In this paper we propose a discriminative framework for predicting the outcomes of a new treatment from a large dataset of the control condition and data from a small (and possibly unrepresentative) randomized trial comparing new and old treatments. Our learning objective, which requires minimal assumptions on the treatments, models the relation between the outcomes of the different conditions. This allows us to not only estimate mean effects but also to generate individual predictions for examples outside the small randomized sample. We demonstrate the utility of our approach through experiments in three areas: search engine operation, treatments to diabetes patients, and market value estimation of houses. Our results demonstrate that our approach can reduce the number and size of the currently performed randomized controlled experiments, thus saving significant time, money and effort on the part of practitioners.

Original languageEnglish
Title of host publication26th International World Wide Web Conference 2017, WWW 2017 Companion
PublisherInternational World Wide Web Conferences Steering Committee
Pages602-609
Number of pages8
ISBN (Electronic)9781450349147
DOIs
StatePublished - 2017
Event26th International World Wide Web Conference, WWW 2017 Companion - Perth, Australia
Duration: 3 Apr 20177 Apr 2017

Publication series

Name26th International World Wide Web Conference 2017, WWW 2017 Companion

Conference

Conference26th International World Wide Web Conference, WWW 2017 Companion
Country/TerritoryAustralia
CityPerth
Period3/04/177/04/17

Fingerprint

Dive into the research topics of 'Predicting counterfactuals from large historical data and small randomized trials'. Together they form a unique fingerprint.

Cite this