^j d Z ddlmZmZmZmZmZmZ ddlm Z ddl
m Z ddl m
Z
e
j e ZddiZ G d d
e Zd
gZy ) z Tokenization classes for XGLM. )Regex Tokenizerdecoders normalizerspre_tokenizers
processors)Unigram )TokenizersBackend)loggingtokenizer_fileztokenizer.jsonc e Zd ZdZeZddgZeZ dde e
e e e f z dz de de de d e d
e d e d e dz d
e
f fd
Z xZS )
XGLMTokenizeraW
Construct a XGLM tokenizer (backed by HuggingFace's tokenizers library). Based on BPE.
This tokenizer inherits from [`TokenizersBackend`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
tokenizer_file (`str`, *optional*):
Path to a tokenizers JSON file containing the serialization of a tokenizer.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding.
vocab (`str`, `dict` or `list`, *optional*):
Custom vocabulary dictionary. If not provided, a minimal vocabulary is created.
merges (`list[tuple[str, str]]`, *optional*):
Custom merge rules for BPE. If not provided, merges are generated from the vocabulary.
add_prefix_space (`bool`, *optional*, defaults to `True`):
Whether to add a prefix space before encoding.
input_idsattention_maskNvocab bos_token eos_token sep_token cls_token unk_token pad_token_spm_precompiled_charsmapadd_prefix_spacec
d| _ t | j D cg c] } d| d
} } |
j dg xs g |
d< |
dxx | D
cg c] }
|
|
d vs |
c}
z
cc< | | _ ||| _ n7t | dft | dft | dft | dfg| _ t
t | j dd | _ t j t d d
g}|%|j d t j | t j | | j _ | rd nd
}t! j" d| | j _ t' j" d| | j _ t+ | X d||||||| d|
t/ j0 | j2 d| j2 d| j2 d
| j2 d| j4 | j6 f| j2 | j8 fg | j _ y c c} w c c}
w )N z additional_special_tokensg r
F)r unk_id
byte_fallbackz {2,} r alwaysneveru ▁) replacementprepend_scheme)r r r r r r r z $Az $A z $B)singlepairspecial_tokens ) num_madeup_wordsrangegetr _vocabstrr r
_tokenizerr Replacer insert PrecompiledSequence
normalizerr Metaspace
pre_tokenizerr decodersuper__init__r TemplateProcessingr r bos_token_id eos_token_idpost_processor)selfr r r r r r r r r kwargsi madeup_wordsword normalizers_r% __class__s u/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/xglm/tokenization_xglm.pyr9 zXGLMTokenizer.__init__> s5 !" 49$:O:O4PQq+aS*Q Q.4jj9TVX.Y._]_*+*+ )0
T@[9\-\D0
+ !1 DK Y %Y %Y %Y % DK $G$++aW\$]^ $++E(OSAB $ 0 ;#:#:;T#U V%0%9%9,%G"%57(6(@(@Ucq(r%"*"4"4We"f
-
*4)F)Fnn%S)NN#4'7q 8HL !2!23 !2!23 *
&U R0
s
H7 H<
H<
) N