^jJ/dZddlZddlZddlmZddlmZejeZ dddZ d Z Gd d eZ d gZ y) z Tokenization classes for BioGPT.N)PreTrainedTokenizer)loggingz vocab.jsonz merges.txt) vocab_file merges_filecbt}|d}|ddD]}|j||f|}|S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairs prev_charchars y/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/biogpt/tokenization_biogpt.py get_pairsrsF EEQIQR 9d#$  Lc eZdZdZeZddgZ dfd ZedZ dZ dZ dZ d Z dd Zd Zd Zd Z ddeedeedzdeefdZ ddeedeedzdedeeffd ZddededzdeefdZdZdZxZS)BioGptTokenizera: Construct an FAIRSEQ Transformer tokenizer. Moses tokenization followed by Byte-Pair Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc  ddl} d|_| |_i|_i|_ t |d5} tj| |_ ddd|jjD cic]\} } | |  c} } |_ t |d5} | jjddd}dddDcgc]}t|jdd }}tt!|t#t%||_i|_t+|Xd |||||d |y#t$r tdwxYw#1swYxYwcc} } w#1swYxYwcc}w) NrzqYou need to install sacremoses to use BioGptTokenizer. See https://pypi.org/project/sacremoses/ for installation.enutf-8encoding ) bos_token eos_token sep_token unk_token pad_token) sacremoses ImportErrorlangsmcache_moses_tokenizercache_moses_detokenizeropenjsonloadencoderitemsdecoderreadsplittupledictziprangelen bpe_rankscachesuper__init__)selfrrr"rr r!r#kwargsr% vocab_handlekv merges_handlemergesmerge __class__s rr;zBioGptTokenizer.__init__Zsb   %'"')$ *w / 3<99\2DL 3)-););)=>A1> + 0 ;M"'')//5cr:F ;8>?u% bq)*??c&%F *<=>       - M   3 3> ; ;?s.D1E 1 E#E#E'1E EE$c,t|jS)zReturns vocab size)r7r.r<s r vocab_sizezBioGptTokenizer.vocab_sizes4<<  rcBt|jfi|jSN)r4r.added_tokens_encoderrFs r get_vocabzBioGptTokenizer.get_vocabsDLL>D$=$=>>rc||jvr+|jj|}||j|<|j|j|dddS)Nr'TF)aggressive_dash_splits return_strescape)r)r(MosesTokenizertokenize)r<textr'moses_tokenizers rmoses_tokenizezBioGptTokenizer.moses_tokenizesc t11 1"gg44$4?O/>D & &t ,))$/88 %9  rc||jvr+|jj|}||j|<|j|j|S)NrM)r*r(MosesDetokenizer detokenize)r<tokensr'moses_detokenizers rmoses_detokenizez BioGptTokenizer.moses_detokenizesT t33 3 $ 8 8d 8 C 1BD ( ( .++D1<cNjj|tdS)Ninf)r8getfloat)pairr<s rz%BioGptTokenizer.bpe..s1C1CD%PU,1Wrkeyrr r z z ) r3r9rminr8r7indexextend ValueErrorappendjoin) r<tokenr r bigramfirstsecondnew_wordijs ` rbpezBioGptTokenizer.bpesU3BZ E"I$6#88 DJJ ::e$ $$6> !$WXFT^^+"ME6HAc$i- 5!,A OOD1I.A7e#CIM(9d1q5kV>SOOEFN3FAOODG,FAc$i- XHD4yA~!$9:xx~ : D  5 /"OODH-sE F?Fc|r|j}n|j||j}g}|D]=}|s|jt |j |jd?|S)zReturns a tokenized string.rf)r2rUr'rilistrt)r<rSbypass_tokenizer split_tokensrms r _tokenizezBioGptTokenizer._tokenizesm ::)>s)C$DE Frc|jj||jj|jS)z0Converts a token (str) in an id using the vocab.)r.r`r")r<rms r_convert_token_to_idz$BioGptTokenizer._convert_token_to_ids,||t||'7'7'GHHrcN|jj||jS)z=Converts an index (integer) in a token (str) using the vocab.)r0r`r")r<rhs r_convert_id_to_tokenz$BioGptTokenizer._convert_id_to_tokens||t~~66rc|Dcgc]$}|jddjdd&}}dj|j}|j||j}|Scc}w)z:Converts a sequence of tokens (string) in a single string.rfr])replacerlr2r[r')r<rYtrSs rconvert_tokens_to_stringz(BioGptTokenizer.convert_tokens_to_stringshDJJa!))C$,,VS9JJ&&($$VTYY7 Ks)A-N token_ids_0 token_ids_1returncV||jg|zS|jg}||z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BioGPT sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) sep_token_id)r<rrseps r build_inputs_with_special_tokensz0BioGptTokenizer.build_inputs_with_special_tokenssA&  %%&4 4  ![ 3&44ralready_has_special_tokensc|rt|||dS|'dgdgt|zzdgzdgt|zzSdgdgt|zzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr r)r:get_special_tokens_maskr7)r<rrrrDs rrz'BioGptTokenizer.get_special_tokens_masksy$ &72'[]a3   "31#K 001QC7A3[AQ;QR RsqcC ,,--rsave_directoryfilename_prefixc .tjj|stj d|dytjj ||r|dzndt dz}tjj ||r|dzndt dz}t|dd 5}|jtj|jd d d dzdddd}t|dd 5}t|jjdD]M\}} || k7rtjd|d| }|jdj |dz|dz }O ddd||fS#1swYxYw#1swY||fSxYw)NzVocabulary path (z) should be a directory-rrrwrrrTF)indent sort_keys ensure_asciirrc |dS)Nr r$)kvs rrcz1BioGptTokenizer.save_vocabulary..*s Y[\]Y^rrdzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rfr )ospathisdirloggererrorrlVOCAB_FILES_NAMESr+writer,dumpsr.sortedr8r/warning) r<rrr merge_filefrhwriter bpe_tokens token_indexs rsave_vocabularyzBioGptTokenizer.save_vocabularysww}}^, LL,^,<     M  s*)zzrz)FrI)NF)__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr;propertyrGrKrUr[rtryr{r}rrvintrboolrstrr3rrr __classcell__)rDs@rrr,s(T*$&67 * X!!? E *X I7GK595379t3C5 c52pu.9.379t3C.hl. c.6&c&C$J&Z_`cZd&8 rr)rr,rtokenization_pythonrutilsr get_loggerrrrrr__all__r$rrrsY' 6   H % Z)Zz  r