^j3 d Z ddlZddlZddlmZ ddlmZ ddlmZ ej e
Z ddd Z d
Z
G d d e Zd gZy)
z Tokenization classes for PhoBERT N)copyfile )PreTrainedTokenizer)loggingz vocab.txtz bpe.codes)
vocab_file merges_filec x t }| d }| dd D ] }|j ||f |} t | }|S )z
Return set of symbol pairs in a word.
Word is represented as tuple of symbols (symbols being variable-length strings).
r N)setadd)wordpairs prev_charchars {/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/phobert/tokenization_phobert.py get_pairsr ! sO
EEQIQR
9d#$
JE L c
* e Zd ZdZeZ d fd Z ddee dee dz dee fdZ ddee dee dz de
dee f fd
Z ddee dee dz dee fd
Z e
d Zd Zd
Zd Zd Zd Zd Zddededz dee fdZd Z xZS )PhobertTokenizeraO
Construct a PhoBERT tokenizer. Based on Byte-Pair-Encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
bos_token (`st`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
c
|| _ || _ i | _ d| j t | < d| j t | < d| j t | < d| j t | < | j | | j j D ci c] \ } } | |
c} } | _ t |d 5 }
|
j j d d d }d d d D cg c] }t |j d d }}t t |t t | | _ i | _ t# | H d
||||||| d |
y c c} } w # 1 sw Y xY wc c}w ) Nr r
r utf-8encoding
) bos_token eos_token unk_token sep_token cls_token pad_token
mask_token )r r encoderstr
add_from_fileitemsdecoderopenreadsplittupledictziprangelen bpe_rankscachesuper__init__)selfr r r r r r! r r" r# kwargskv
merges_handlemergesmerge __class__s r r5 zPhobertTokenizer.__init__d sY %& '( S^$'( S^$'( S^$'( S^$ :&)- ););)=>A1>
+
0 ;M"'')//5cr:F ;9?@%
cr*+@@ c&%F *< =>
!
? ; ;@s %
E'#E-7#E9-E6N token_ids_0 token_ids_1returnc | | j g|z | j gz S | j g}| j g}||z |z |z |z |z S )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A PhoBERT sequence has the following format:
- single sequence: ` X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) cls_token_id sep_token_id)r6 r> r? clsseps r build_inputs_with_special_tokensz1PhobertTokenizer.build_inputs_with_special_tokens sg ( %%&48I8I7JJ J ! ![ 3&,{:S@@r already_has_special_tokensc |rt | ||d S |dgdgt | z z dgz S dgdgt | z z ddgz dgt | z z dgz S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)r> r? rG r
r )r4 get_special_tokens_maskr1 )r6 r> r? rG r= s r rI z(PhobertTokenizer.get_special_tokens_mask s & &72 '[]a 3
31#K 001QC7 7sqcC ,,-A61#K@P:PQUVTWWWr c | j g}| j g}|t ||z |z dgz S t ||z |z |z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. PhoBERT does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
r )rC rB r1 )r6 r> r? rE rD s r $create_token_type_ids_from_sequencesz5PhobertTokenizer.create_token_type_ids_from_sequences sm " ! ! s[(3./1#5 53$s*S0;>DEKKr c , t | j S N)r1 r% r6 s r
vocab_sizezPhobertTokenizer.vocab_size s 4<< r c B t | j fi | j S rM )r. r% added_tokens_encoderrN s r get_vocabzPhobertTokenizer.get_vocab s DLL>D$=$=>>r c $ | j v r j | S t | }t t |d d |d dz gz }t | }|s|S t | fd }| j
vrn|\ }}g }d}|t
| k r |j || } |j ||| | }|| |k( r6|t
| dz
k r%||dz |k( r|j ||z |dz
}n|j || |dz
}|t
| k rt | }|}t
| dk( rn t | }dj | }|d d }| j |< |S # t $ r |j ||d Y nw xY w)
Nr zc N j j | t d S )Ninf)r2 getfloat)pairr6 s r z&PhobertTokenizer.bpe.. s 1C1CD%PU,1W r )keyr r
r @@ ) r3 r- listr minr2 r1 indexextend
ValueErrorappendjoin)
r6 tokenr
r bigramfirstsecondnew_wordijs
` r bpezPhobertTokenizer.bpe s DJJ ::e$ $U|T$s)_R6(9'::;$L$WXFT^^+ "ME6HAc$i-
5!,A
OOD1I.A7e#CIM(9d1q5kV>S OOEFN3FA OODG,FA c$i- XHD4yA~!$9 : zz$ CRy
5 - " OODH-s E/ / FFc g }t j d| }|D ]: }|j t | j | j d < |S )zTokenize a string.z\S+\n? )refindallr` r] rk r, )r6 text split_tokenswordsrd s r _tokenizezPhobertTokenizer._tokenize sT
9d+ BE TXXe_%:%:3%? @ A Br c | j j || j j | j S )z0Converts a token (str) in an id using the vocab.)r% rV r )r6 rd s r _convert_token_to_idz%PhobertTokenizer._convert_token_to_id s, || t||'7'7'GHHr c N | j j || j S )z=Converts an index (integer) in a token (str) using the vocab.)r) rV r )r6 r_ s r _convert_id_to_tokenz%PhobertTokenizer._convert_id_to_token s || t~~66r c d dj | j dd j }|S )z:Converts a sequence of tokens (string) in a single string.rm r[ )rc replacestrip)r6 tokens
out_strings r convert_tokens_to_stringz)PhobertTokenizer.convert_tokens_to_string# s, XXf%--eR8>>@
r save_directoryfilename_prefixc t j j | st j d| d y t j j ||r|dz ndt d z }t j j ||r|dz ndt d z }t j j | j t j j | k7 r@t j j | j rt | j | nit j j | j s@t |d 5 }| j j }|j | d d d t j j | j t j j | k7 rt | j | ||fS # 1 sw Y lxY w)NzVocabulary path (z) should be a directory-ry r r wb)ospathisdirloggererrorrc VOCAB_FILES_NAMESabspathr isfiler r* sp_modelserialized_model_protowriter )r6 r r out_vocab_fileout_merge_fileficontent_spiece_models r save_vocabularyz PhobertTokenizer.save_vocabulary( sr ww}}^, LL,^,< ')
isinstancer& r* r' FileNotFoundError UnicodeError Exception readlinesr{ rfindra r1 r% ) r6 ffdfnfelineslineTmplineidxr
s r r' z PhobertTokenizer.add_from_fileE s a
c!S73 +r&&r*+
3G==?D**S/Cby !XYY:D!$T\\!2DLL
3+
%
c "A!D` abb
cs3 B7 B+B7 +B40B7 4B7 7 C C C )