Qwen3_5Tokenizer

Tokenizer for Qwen3.5 models.
This tokenizer implements byte-pair encoding (BPE) for Qwen3.5 models.
Instantiate a keras_hub.models.Tokenizer from a model preset.
A preset is a directory of configs, weights and other file assets used to save and load a pre-trained model. The preset can be passed as one of:
For any Tokenizer subclass, you can run cls.presets.keys() to list all built-in presets available on the class.
This constructor can be called in one of two ways. Either from the base class like keras_hub.models.Tokenizer.from_preset(), or from a model class like keras_hub.models.GemmaTokenizer.from_preset(). If calling from the base class, the subclass of the returning object will be inferred from the config in the preset directory.