Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing

Tal Schuster, Ori Ram, Regina Barzilay, Amir Globerson

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

126 Scopus citations

Abstract

We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to their dynamic nature. To this end, we construct context-independent variants of the original monolingual spaces and utilize their mapping to derive an alignment for the context-dependent spaces. This mapping readily supports processing of a target language, improving transfer by context-aware embeddings. Our experimental results demonstrate the effectiveness of this approach for zero-shot and few-shot learning of dependency parsing. Specifically, our method consistently outperforms the previous state-of-the-art on 6 tested languages, yielding an improvement of 6.8 LAS points on average.

Original languageEnglish
Title of host publicationLong and Short Papers
PublisherAssociation for Computational Linguistics (ACL)
Pages1599-1613
Number of pages15
ISBN (Electronic)9781950737130
StatePublished - 2019
Event2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019 - Minneapolis, United States
Duration: 2 Jun 20197 Jun 2019

Publication series

NameNAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference
Volume1

Conference

Conference2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019
Country/TerritoryUnited States
CityMinneapolis
Period2/06/197/06/19

Funding

FundersFunder number
US-Israel Binational Science Foundation
Yandex Initiative in Machine Learning
Massachusetts Institute of Technology
Office of the Director of National Intelligence
Intelligence Advanced Research Projects ActivityFA8650-17-C-9116
United States-Israel Binational Science Foundation2012330

    Fingerprint

    Dive into the research topics of 'Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing'. Together they form a unique fingerprint.

    Cite this