Local Byte Fusion for Neural Machine Translation

Collaborating with researchers from the University of Wisconsin-Madison, we put forward a fundamental tokenization method that we term Local Byte Fusion (LOBEF) for byte-based machine translation. The method outperforms other techniques, especially when it comes to multilingual translation.

Unlike the current dominant tokenization technique, subword tokenization, which has limitations on multilingual corpus, LOBEF utilizes byte n-gram and word boundaries to aggregate local semantic information. Thus, it has advantages on multilingual corpus with universal tokenization schema. In experiments, our method outperforms traditional byte-based models and subword techniques on multilingual translation, zero-shot cross-lingual transfer and domain adaptation.

LOBEF contains both n-gram Convolution Fusion (nCF) and Word-based Self-attention Fusion (WSF). In nCF, we use four convolutional layers to aggregate character level information, and in WSF, we use word boundaries with block-wise self attention to aggregate word level information. Our results indicate that byte based models outperform subword baselines on the Flores-101 dataset. Additionally, byte based models are smaller than comparable subword models and are 20% faster to train.