BytePairTokenizer

Bype-pair encoding tokenizer layer.
This BPE tokenizer provides the same functionality as the official GPT-2 tokenizer. Given the same vocabulary which maps tokens to ids, and merges which describes BPE merge rules, it should provide the same output as OpenAI implementation (https://github.com/openai/gpt-2/blob/master/src/encoder.py). Different from OpenAI, this implementation is graph-compatible, so you can use it within a tf.data pipeline.
If input is a batch of strings (rank > 0): By default, the layer will output a tf.RaggedTensor where the last dimension of the output is ragged. If sequence_length is set, the layer will output a dense tf.Tensor where all inputs have been padded or truncated to sequence_length. If input is a scalar string (rank == 0): By default, the layer will output a dense tf.Tensor with static shape [None]. If sequence_length is set, the output will be a dense tf.Tensor of shape [sequence_length].
Transform input tensors of strings into output tokens.
Transform tokens back into strings.
Get the tokenizer vocabulary as a list of strings tokens.
Get the integer size of the tokenizer vocabulary.
Convert a string token to an integer id.
Convert an integer id to a string token.