Q. In RoBERTa, I’m not sure if the model use Byte Pair Encoding (BPE) or Byte Pair Encoding (BPE) > Byte Level Tokenizer, are these techniques different or the same? Can someone explain? Thanks
Byte-level BPE is a subtype of BPE that uses bytes instead of characters as basic token component.
In the character-level BPE, the vocabulary is composed of sequences of characters that appear frequently together found by means of the BPE algorithm, but also includes all the individual characters seed in the training data. This allows to resort to character-by-character word composition for unseen words. This, however, poses two problems: 1) if at inference time we bump into unseen characters, the tokenizer will not be able to represent them, and 2) there are a lot of unicode characters, which needs a lot of room in the vocabulary to be represented, especially taking into account that a reasonable vocabulary size should be around 50k tokens.
In the byte-level BPE, the vocabulary is not created based on characters, but based on bytes. For that, characters are decomposed as the associated sequence of bytes in the unicode representation. This removes completely the possibility of unseen characters and saves space when representing individual elements (i.e. bytes) in the vocabulary.
In this article that studies the application of byte-level BPE to machine translation, you can find more details about this kind of BPE.