Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems

Jindřich Libovický, Alexander Fraser

EMNLP 2020

Applying the Transformer architecture on the character level usually
requires very deep architectures that are difficult and slow to
train. These problems can be partially overcome by incorporating a
segmentation into tokens in the model. We show that by initially
training a subword model and then finetuning it on characters, we can
obtain a neural machine translation model that works at the character
level without requiring token segmentation. We use only the vanilla
6-layer Transformer Base architecture. Our character-level models
better capture morphological phenomena and show more robustness to
noise at the expense of somewhat worse overall translation
quality. Our study is a significant step towards high-performance and
easy to train character-based models that are not extremely large.