A/B testing at SweetIM: The importance of proper statistical analysis

Slava Brodovsky, Saharon Rosset

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

A/B tests are an important aspect of modern applications, particularly on the internet, as they allow businesses to optimize their user or customer experience to maximize usage and ultimately profits. In this paper we review SweetIM A/B testing system, with focus on statistical aspects and methodology. We describe all parameters of the SweetIM A/B test environment including randomization mechanism, data collection and analysis. We also provide some interesting examples and case studies from real A/B tests that took place at SweetIM. Analyses of count data like number of searches or content sent per user are the most significant performance indicators for the majority of SweetIM tests. Accuracy of such analyses is a key-success factor and can increase the ROI of the tests dramatically. We expand the popular method of the analyses of counts based on Poisson distribution and show its inappropriateness when dealing with over dispersed Poisson. We propose to use Negative Binomial distribution as appropriate solution for over dispersion in search and content count data. We show that the conclusions from analyses of specific A/A and A/B tests run in this application with NB differ from those with an incorrect Poisson assumption.

Original languageEnglish
Title of host publicationProceedings - 11th IEEE International Conference on Data Mining Workshops, ICDMW 2011
Pages733-740
Number of pages8
DOIs
StatePublished - 2011
Event11th IEEE International Conference on Data Mining Workshops, ICDMW 2011 - Vancouver, BC, Canada
Duration: 11 Dec 201111 Dec 2011

Publication series

NameProceedings - IEEE International Conference on Data Mining, ICDM
ISSN (Print)1550-4786

Conference

Conference11th IEEE International Conference on Data Mining Workshops, ICDMW 2011
Country/TerritoryCanada
CityVancouver, BC
Period11/12/1111/12/11

Keywords

  • A/B testing
  • Negative binomial
  • Sypothesis testing

Fingerprint

Dive into the research topics of 'A/B testing at SweetIM: The importance of proper statistical analysis'. Together they form a unique fingerprint.

Cite this