Blockwise Self-Attention for Long Document Understanding

Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, Jie Tang

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.
Original languageEnglish
Title of host publicationFindings of the Association for Computational Linguistics: EMNLP 2020
EditorsTrevor Cohn, Yulan He, Yang Liu
PublisherAssociation for Computational Linguistics
Pages2555-2565
Number of pages11
ISBN (Electronic)978-1-952148-90-3
DOIs
StatePublished - 1 Nov 2020
Externally publishedYes

Fingerprint

Dive into the research topics of 'Blockwise Self-Attention for Long Document Understanding'. Together they form a unique fingerprint.

Cite this