Combining Static and Contextualised Multilingual Embeddings. 

Katharina Hämmerl, Jindřich Libovický, Alexander Fraser

Findings of ACL 2022

Static and contextual multilingual embeddings have complementary
strengths. Static embeddings, while less expressive than contextual
language models, can be more straightforwardly aligned across multiple
languages. We combine the strengths of static and contextual models to
improve multilingual representations. We extract static embeddings for
40 languages from XLM-R, validate those embeddings with cross-lingual
word retrieval, and then align them using VecMap. This results in
high-quality, highly multilingual static embeddings. Then we apply a
novel continued pre-training approach to XLM-R, leveraging the high
quality alignment of our static embeddings to better align the
representation space of XLM-R. We show positive results for multiple
complex semantic tasks. We release the static embeddings and the
continued pre-training code. Unlike most previous work, our continued
pre-training approach does not require parallel text.