^j, UdZddlmZmZddlmZej eZdZ dZ dZ dZ dZ d Zd Ze d e d e d ede dediZeeefed<ej+Dcic]\}}|| c}}Zeeefed<GddeZdgZycc}}w)z Tokenization classes for CANINE.) AddedTokenPreTrainedTokenizer)loggingiiiiiiz[CLS]z[SEP]z[BOS]z[MASK]z[PAD]z [RESERVED]SPECIAL_CODEPOINTSSPECIAL_CODEPOINTS_BY_NAMEceZdZdZgdZeeeeeeeeeeee ddffd Z e de fdZ dZd edeefd Zd ede fd Zd e defdZdZxZS)CanineTokenizera Construct a CANINE tokenizer (i.e. a character splitter). It turns text into a sequence of characters, and then converts each character into its Unicode code point. [`CanineTokenizer`] inherits from [`PreTrainedTokenizer`]. Refer to superclass [`PreTrainedTokenizer`] for usage examples and documentation concerning parameters. Args: model_max_length (`int`, *optional*, defaults to 2048): The maximum sentence length the model accepts. ) input_idsattention_masktoken_type_idsFic  t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}i|_tj D]\} } | |j| <|jj D cic]\} } | |  c} } |_t|_t|j|_ t |0d||||||||dddd | ycc} } w)NF)lstriprstripT all_zeroscls_sep) bos_token eos_token sep_token cls_token pad_token mask_tokenadd_prefix_spacemodel_max_lengthtoken_type_ids_pattern%token_type_ids_include_special_tokensspecial_tokens_pattern) isinstancestrr_special_codepointsritems_special_codepoint_stringsUNICODE_VOCAB_SIZE_unicode_vocab_sizelen_num_special_tokenssuper__init__) selfrrrrrrrrkwargs codepointname __class__s y/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/canine/tokenization_canine.pyr)zCanineTokenizer.__init__GsyJTT]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir KUU_adJeZ 4Fku 46 1779 7OIt-6D $ $T * 7 483K3K3Q3Q3S; /iItO; '$6 #&t'?'?#@   !--#.26#,  ; s E(returnc|jSN)r%)r*s r/ vocab_sizezCanineTokenizer.vocab_sizexs'''ct|jDcic]}t||}}|j|j|Scc}wr2)ranger3chrupdateadded_tokens_encoder)r*ivocabs r/ get_vocabzCanineTokenizer.get_vocab|sB$)$//$:;qQ;; T../ res'B   H %     ( l &DcN  VhUmUmUo-p/)TdIo-pDcNpb)bJ  Q.qs B