Filtering with the crowd: Crowdscreen revisited

Benoît Groz, Ezra Levin, Isaac Meilijson, Tova Milo

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations


Filtering a set of items, based on a set of properties that can be verified by humans, is a common application of CrowdSourcing. When the workers are error-prone, each item is presented to multiple users, to limit the probability of misclassification. Since the Crowd is a relatively expensive resource, minimizing the number of questions per item may naturally result in big savings. Several algorithms to address this minimization problem have been presented in the CrowdScreen framework by Parameswaran et al. However, those algorithms do not scale well and therefore cannot be used in scenarios where high accuracy is required in spite of high user error rates. The goal of this paper is thus to devise algorithms that can cope with such situations. To achieve this, we provide new theoretical insights to the problem, then use them to develop a new efficient algorithm. We also propose novel optimizations for the algorithms of CrowdScreen that improve their scalability. We complement our theoretical study by an experimental evaluation of the algorithms on a large set of synthetic parameters as well as real-life crowdsourcing scenarios, demonstrating the advantages of our solution.

Original languageEnglish
Title of host publication19th International Conference on Database Theory, ICDT 2016
EditorsThomas Zeume, Wim Martens
PublisherSchloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing
ISBN (Electronic)9783959770026
StatePublished - 1 Mar 2016
Event19th International Conference on Database Theory, ICDT 2016 - Bordeaux, France
Duration: 15 Mar 201618 Mar 2016

Publication series

NameLeibniz International Proceedings in Informatics, LIPIcs
ISSN (Print)1868-8969


Conference19th International Conference on Database Theory, ICDT 2016


FundersFunder number
Seventh Framework Programme291071


    • Algorithms
    • CrowdSourcing
    • Filtering
    • Hypothesis Testing
    • Sprt


    Dive into the research topics of 'Filtering with the crowd: Crowdscreen revisited'. Together they form a unique fingerprint.

    Cite this