^j ' l d Z ddlZddlmZmZ ddlmZ ej e Z G d de Z
dgZ y)z"Tokenization class for model ByT5. N )
AddedTokenPreTrainedTokenizer)loggingc
N e Zd ZdZddgZ d d fd
Zed Zd Z dd e e
d
e e
dz d e de e
f fd
Z d
e e
de e
fdZ
d d e e
d
e e
dz de e
fdZ d d e e
d
e e
dz de e
fdZdede e fdZd Zd Zd Zd dededz dee fdZ xZS )
ByT5Tokenizera
Construct a ByT5 tokenizer. ByT5 simply uses raw bytes utf-8 encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 125):
Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are
accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are
indexed from the end of the vocabulary up to beginning ("" is the last token in the vocabulary
like in ByT5 preprocessing see
[here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)).
additional_special_tokens (`list[str]`, *optional*):
Additional special tokens used by the tokenizer.
input_idsattention_maskNreturnc 0 |dkD r | t | D cg c] }d| d
}}nK|dkD rF|Dt | dkD r6t t t d | }||k7 rt d| d| d t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}|||d
| _ t | j | _ d | _
t | 0 d
|||d|d | y c c}w )Nr z
c . t dt | v S )Nextra_id)boolstr)xs u/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/byt5/tokenization_byt5.pyz(ByT5Tokenizer.__init__..J s Ds1v9M4N zBoth extra_ids (z!) and additional_special_tokens (zm) are provided to ByT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensT)lstriprstrip)r ) eos_token unk_token pad_token extra_idsadditional_special_tokens )
rangelensetfilter
ValueError
isinstancer r _added_tokens_decoderoffset_utf_vocab_sizesuper__init__)
selfr r r r r kwargsi extra_tokens __class__s
r r+ zByT5Tokenizer.__init__< sI q= 6 >DI)DT(Uq:aS):(U %(U
] 8 DMfIgjkIk s6*NPi#j klLy( &yk1RSlRm n( ( HRR[]`GaJydCgp GQR[]`GaJydCgp GQR[]`GaJydCgp )2yY%O"$445 #
&?
' )Vs
Dc | j S N)r) )r, s r
vocab_sizezByT5Tokenizer.vocab_sizec s ###r c t | j | j z D ci c] }| j | | }}|j | j
|S c c}w r2 )r! r3 r( convert_ids_to_tokensupdateadded_tokens_encoder)r, r. vocabs r get_vocabzByT5Tokenizer.get_vocabg sW ;@ SWS^S^A^;_`a++A.1``
T../ as A token_ids_0 token_ids_1already_has_special_tokensc |rt | ||d S |dgt | z dgz S dgt | z dgz dgt | z z dgz S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)r: r; r< r r )r* get_special_tokens_maskr" )r, r: r; r< r0 s r r> z%ByT5Tokenizer.get_special_tokens_maskl sy $ &72 '[]a 3
C#k**qc1 1c+&&1#-!s;7G1GHA3NNr token_idsc t | dkD r7|d | j k( r%t j d| j d |S || j gz S )z.Do not add eos again if user already added it.r zThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)r" eos_token_idwarningswarnr )r, r? s r _add_eos_if_not_presentz%ByT5Tokenizer._add_eos_if_not_present s] y>A )B-43D3D"D MM,T^^,< =+ +
1 1 22 2r c t | j g}|t ||z dgz S t ||z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. ByT5 does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
r )rB r" )r, r: r; eoss r $create_token_type_ids_from_sequencesz2ByT5Tokenizer.create_token_type_ids_from_sequences sP ! {S()QC/ /;${2S89QC??r c X | j | }||S | j | }||z S )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
)rE )r, r: r; s r build_inputs_with_special_tokensz.ByT5Tokenizer.build_inputs_with_special_tokens s; & 22;? 66{CK , ,r textc ^ |j d D cg c]
}t | }}|S c c}w )zPTake as input a string and return a list of strings (tokens) for words/sub-wordsutf-8)encodechr)r, rK r. tokenss r _tokenizezByT5Tokenizer._tokenize s, "&++g"67Q#a&77
8s *c Z t | dk7 rd}|S t | | j z }|S )z0Converts a token (str) in an id using the vocab.r N)r" ordr( )r, tokentoken_ids r _convert_token_to_idz"ByT5Tokenizer._convert_token_to_id s4 u:?H 5zDKK/Hr c 6 t || j z
}|S )z=Converts an index (integer) in a token (str) using the vocab.)rO r( )r, indexrT s r _convert_id_to_tokenz"ByT5Tokenizer._convert_id_to_token s EDKK'( r c d}|D ]i }|| j v r | j | j d }n5|| j v r|j d }nt t | g }||z
}k |j dd }|S )z:Converts a sequence of tokens (string) in a single string.r rM ignore)errors)r' rN _added_tokens_encoderbytesrS decode)r, rP bstringrT
tok_stringstrings r convert_tokens_to_stringz&ByT5Tokenizer.convert_tokens_to_string s "E222 ! 7 7 > E Eg N
$444 "\\' 2
"CJ< 0
z !G " 9
r save_directoryfilename_prefixc y)Nr r )r, rd re s r save_vocabularyz ByT5Tokenizer.save_vocabulary s r )zzz} N)r N)NFr2 )__name__
__module__ __qualname____doc__model_input_namesr+ propertyr3 r9 listintr r> rE rH rJ r rQ rV rY rc tuplerg
__classcell__)r0 s @r r r sg @ %&67 "&
%
%
N $ $ puO 9O379t3COhlO
cO8 3c 3tCy 3 GK@ 9@379t3C@
c@. GK- 9-379t3C-
c-4c d3i
c C$J Z_`cZd r r ) rl rC tokenization_pythonr r utilsr
get_loggerri loggerr __all__r r r rx sA ) B
H %N' Nb
r