Break it down: A question understanding benchmark

Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, Jonathan Berant

Research output: Contribution to journalArticlepeer-review

106 Scopus citations

Abstract

Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer. In this work, we introduce a Question Decomposition Meaning Representation (QDMR) for questions. QDMR constitutes the ordered list of steps, expressed through natural language, that are necessary for answering a question. We develop a crowdsourcing pipeline, showing that quality QDMRs can be annotated at scale, and release the BREAK dataset, containing over 83K pairs of questions and their QDMRs. We demonstrate the utility of QDMR by showing that (a) it can be used to improve open-domain question answering on the HOTPOTQA dataset, (b) it can be determin-istically converted to a pseudo-SQL formal language, which can alleviate annotation in semantic parsing applications. Last, we use BREAK to train a sequence-to-sequence model with copying that parses questions into QDMR structures, and show that it substantially outperforms several natural baselines.

Original languageEnglish
Pages (from-to)183-198
Number of pages16
JournalTransactions of the Association for Computational Linguistics
Volume8
DOIs
StatePublished - 2020

Funding

FundersFunder number
European Union Horizons 2020 research and innovation programmeDELPHI 802800
Yandex Initiative for Machine Learning
Horizon 2020 Framework Programme802800
Horizon 2020 Framework Programme
European Research Council
Israel Science Foundation942/16, 978/17
Israel Science Foundation

    Fingerprint

    Dive into the research topics of 'Break it down: A question understanding benchmark'. Together they form a unique fingerprint.

    Cite this