Blockwise self-attention for long document understanding

Jiezhong Qiu; Hao Ma; Omer Levy; Wen Tau Yih; Sinong Wang; Jie Tang

Blockwise self-attention for long document understanding

Jiezhong Qiu, Hao Ma, Omer Levy, Wen Tau Yih, Sinong Wang, Jie Tang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics Findings of ACL
Subtitle of host publication	EMNLP 2020
Publisher	Association for Computational Linguistics (ACL)
Pages	2555-2565
Number of pages	11
ISBN (Electronic)	9781952148903
State	Published - 2020
Externally published	Yes
Event	Findings of the Association for Computational Linguistics, ACL 2020: EMNLP 2020 - Virtual, Online Duration: 16 Nov 2020 → 20 Nov 2020

Publication series

Name	Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020

Conference

Conference	Findings of the Association for Computational Linguistics, ACL 2020: EMNLP 2020
City	Virtual, Online
Period	16/11/20 → 20/11/20

Cite this

Qiu, J., Ma, H., Levy, O., Yih, W. T., Wang, S., & Tang, J. (2020). Blockwise self-attention for long document understanding. In Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020 (pp. 2555-2565). (Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020). Association for Computational Linguistics (ACL).

@inproceedings{39951a65be53432cbe3cc4c8b7f21654,

title = "Blockwise self-attention for long document understanding",

abstract = "We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.",

author = "Jiezhong Qiu and Hao Ma and Omer Levy and Yih, {Wen Tau} and Sinong Wang and Jie Tang",

note = "Publisher Copyright: {\textcopyright} 2020 Association for Computational Linguistics; null ; Conference date: 16-11-2020 Through 20-11-2020",

year = "2020",

language = "אנגלית",

series = "Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020",

publisher = "Association for Computational Linguistics (ACL)",

pages = "2555--2565",

booktitle = "Findings of the Association for Computational Linguistics Findings of ACL",

address = "ארצות הברית",

}

Qiu, J, Ma, H, Levy, O, Yih, WT, Wang, S & Tang, J 2020, Blockwise self-attention for long document understanding. in Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020. Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020, Association for Computational Linguistics (ACL), pp. 2555-2565, Findings of the Association for Computational Linguistics, ACL 2020: EMNLP 2020, Virtual, Online, 16/11/20.

Blockwise self-attention for long document understanding. / Qiu, Jiezhong; Ma, Hao; Levy, Omer et al.

Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020. Association for Computational Linguistics (ACL), 2020. p. 2555-2565 (Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Blockwise self-attention for long document understanding

AU - Qiu, Jiezhong

AU - Ma, Hao

AU - Levy, Omer

AU - Yih, Wen Tau

AU - Wang, Sinong

AU - Tang, Jie

PY - 2020

Y1 - 2020

N2 - We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.

AB - We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.

UR - http://www.scopus.com/inward/record.url?scp=85102540806&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85102540806

T3 - Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 2020

SP - 2555

EP - 2565

BT - Findings of the Association for Computational Linguistics Findings of ACL

PB - Association for Computational Linguistics (ACL)

Y2 - 16 November 2020 through 20 November 2020

ER -

Blockwise self-attention for long document understanding

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this