^jdZddlZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*e'jVe,Z-Gddej\Z/Gddej\Z0Gddej\Z1Gddej\Z2Gddej\Z3Gddej\Z4Gddej\Z5Gdd eZ6Gd!d"ej\Z7e&Gd#d$e"Z8Gd%d&e8Z9e&Gd'd(e8Z:e&d)*Gd+d,e8eZ;e&Gd-d.e8Z<e&d/*Gd0d1e8Z=e&Gd2d3e8Z>e&Gd4d5e8Z?gd6Z@y)7zPyTorch T5 model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringloggingtorch_compilable_check)T5Configc&eZdZdfd ZdZxZS) T5LayerNormct|tjt j ||_||_y)zc Construct a layernorm module in the T5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s m/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.pyr$zT5LayerNorm.__init__/s1 ll5::k#:; #c|jtjjdj dd}|tj ||j zz}|jjtjtjfvr%|j|jj}|j|zS)NT)keepdim) tor&float32powmeanrsqrtr)r(dtypefloat16bfloat16)r* hidden_statesvariances r.forwardzT5LayerNorm.forward7s !##EMM266q9>>r4>P% Ht?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r/)gư>)__name__ __module__ __qualname__r$r> __classcell__r-s@r.r!r!.s $ +r/r!c*eZdZdeffd ZdZxZS)T5DenseActDenseconfigc^t|tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_ yNFbias)r#r$rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr*rFr-s r.r$zT5DenseActDense.__init__Hsn ))FNNFKKeD))FKKeDzz&"5"56 &--.r/c|j|}|j|}|j|}t|jj t jr|j|jj jk7r`|jj jt jk7r/|j|jj j}|j |}|SN) rNrTrR isinstancerOr(r&Tensorr9int8r4r*r<s r.r>zT5DenseActDense.forwardOs . /  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r/r?r@rArr$r>rBrCs@r.rErEGs/x/ r/rEc*eZdZdeffd ZdZxZS)T5DenseGatedActDenserFct|tj|j|j d|_tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_yrH)r#r$rrKrLrMwi_0wi_1rOrPrQrRr rSrTrUs r.r$zT5DenseGatedActDense.__init__^s IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r/c,|j|j|}|j|}||z}|j|}t |j j tjr|j|j j jk7r`|j j jtjk7r/|j|j j j}|j |}|SrW) rTr`rarRrXrOr(r&rYr9rZr4)r*r< hidden_gelu hidden_linears r.r>zT5DenseGatedActDense.forwardfshhtyy78  -0 #m3  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r/r\rCs@r.r^r^]s/x/r/r^c*eZdZdeffd ZdZxZS) T5LayerFFrFct||jrt||_nt ||_t |j|j|_ tj|j|_ y)Nr,)r#r$ is_gated_actr^DenseReluDenserEr!rLlayer_norm_epsilon layer_normrrPrQrRrUs r.r$zT5LayerFF.__init__{s_    "6v">D "1&"9D %fnn&:S:STzz&"5"56 r/cr|j|}|j|}||j|z}|SrW)rlrjrR)r*r<forwarded_statess r.r>zT5LayerFF.forwards=??=9../?@% 5E(FF r/r\rCs@r.rfrfzs7x7r/rfc`eZdZ ddededzffd Zed dZd dZ d dZ xZ S) T5AttentionNrF layer_idxct||j|_||_|j|_|j |_|j |_|j|_|j|_ |j|_ |j|jz|_ ||_|9|jr-tj!d|j"j$dt'j(|j |jd|_t'j(|j |jd|_t'j(|j |jd|_t'j(|j|j d|_|jr/t'j2|j|j|_d|_y)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrI)r#r$ is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerLd_kvkey_value_proj_dim num_headsn_headsrQrR inner_dimrqlogger warning_oncer-r?rrKqkvo Embeddingrelative_attention_biasgradient_checkpointingr*rFrtrqr-s r.r$zT5Attention.__init__si  +++F(.4.S.S+/5/U/U,~~ "(++'' ** (?(??"     *4>>+B+B*CD,,  4<<eD4<<eD4<<eD4>>4<=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr1r) r4r&longabsmin zeros_likelogfloatmath full_likewhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges r._relative_position_bucketz%T5Attention._relative_position_buckets(,  A K !2Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $y0&/ II'--/); <hh|i/0 1Y& ( "UZZ. &" &+YY &8RT_bcTc(d& " EKK2CE_``r/c| |jjj}tj|tj |dddf|z}tj|tj |dddf}||z }|j ||j |j|j}|j|} | jgdjd} | S)z%Compute binned relative position biasN)r9device)rrr)r1rrr) rr(rr&arangerrrsrurvpermute unsqueeze) r* query_length key_lengthrpast_seen_tokenscontext_positionmemory_positionrrelative_position_bucketvaluess r. compute_biaszT5Attention.compute_biass >1188??F << EJJvVWXZ^W^_brr,,zFSTXZ[T[\+.>>#'#A#A #.;;== $B$  --.FG *44Q7 r/c |jdd}g|d|j} ||j|jnd} t | t j r| jn| } |du} |j|j| jdd} d} t |trA|jj|j} | r |j}n|j}n|}| r|n|}| rK|I| rG|j |jj"}|j |jj$}ng|jddd|j}|j'|j|jdd}|j)|j|jdd}|K|j+|||j\}}| r)t |trd|j|j<t j,| |jdd}||jd }|j.s`t j0d| jd|d|f|j2|j4 }|j6r6|j8r*d|_n"|j=|d||j2| }|#|ddddddd|jd f}||z}|}||z }t>j@jC|jEd jG|}t>j@jI||jH|j8 }t j,||}|jddjK}|jLg|d}|jO|}||f}|r||fz}|S)z Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). Nr2rrr1FTr)rr9)rrdim)ptraining)(shaperxget_seq_lengthrqrXr&rYcloner~view transposer is_updatedgetcross_attention_cacheself_attention_cachelayerskeysrrrupdatematmulrtzerosrr9rr requires_gradrr functionalsoftmaxrtype_asrR contiguousreshaper)r*r<maskkey_value_states position_biaspast_key_valuesoutput_attentionskwargs input_shape hidden_shaperis_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapescoresr causal_maskposition_bias_masked attn_weights attn_outputoutputss r.r>zT5Attention.forwards$))#2. BBbB$*A*AB M\Mh?99$..Ino7ABRTYT`T`7a+113gw.T9vvm,11,?II!QO  o': ;(3377GJ!'6'L'L$'6'K'K$#2 -?)] /"=*-44T^^DIIJ/66t~~FMMLP--cr2PBP8O8OPH/44X>HHANJ66.166x@JJ1aPL*+?+F+FzS_aeaoao+p( L%*_FY*ZAEO..t~~>lJ,@,@A,FG  #))"-J33 %  **1-{1~zJSYS`S`hnhtht! ..4==26M/ $ 1 1NJv}}Wg!2! "1a,Bj.>.>r.B,B#BC - ; ,&&}},,V\\^,DLLVT }},,\T\\TXTaTa,b ll<> !++Aq1<<> )k));;;; ff[)  . /Gr/FN)T )Nr)NNNNF) r?r@rArintr$ staticmethodrrr>rBrCs@r.rprpsX%* $ , ,: ,D- - ^([r/rpc>eZdZddedzffd Z ddZxZS)T5LayerSelfAttentionNrqct|t||||_t |j |j |_tj|j|_ y)Nrtrqrh) r#r$rp SelfAttentionr!rLrkrlrrPrQrRrs r.r$zT5LayerSelfAttention.__init__OsT ( 0KW` &fnn&:S:STzz&"5"56 r/c |j|}|j||||||} ||j| dz}|f| ddz} | S)N)rrr use_cacherrr)rlrrR) r*r<attention_maskrrrrrnormed_hidden_statesattention_outputrs r.r>zT5LayerSelfAttention.forwardWsp $}=-- '+/ . & 5Ea5H(II  "%5ab%99r/r)NNNFFr?r@rArr$r>rBrCs@r.rrNs*7SSWZ7r/rc<eZdZddedzffd Z ddZxZS)T5LayerCrossAttentionNrqct|t|d||_t |j |j |_tj|j|_ y)NFrrh) r#r$rpEncDecAttentionr!rLrkrlrrPrQrR)r*rFrqr-s r.r$zT5LayerCrossAttention.__init__psN *6u`ij%fnn&:S:STzz&"5"56 r/c |j|}|j||||||} ||j| dz} | f| ddz} | S)N)rrrrrrr)rlrrR) r*r<rrrrrrrr layer_outputrs r.r>zT5LayerCrossAttention.forwardvso $}=// -'+/ 0 %t||4DQ4G'HH /$4QR$88r/rW)NNNFrrCs@r.rros&7#*7r/rcFeZdZddedzffd Z ddZxZS)T5BlockNrqcpt||j|_tj|_|j j t||||jr&|j j t|||j j t|y)Nr)rq) r#r$rsr ModuleListlayerappendrrrfrs r.r$zT5Block.__init__s  ++]]_   E`lu v  ?? JJ  3FiP Q )F+,r/c  x|jd|||||| } | d}| dd} |jtjk(rtjtj |j tj|jjdz tj|jj}tj|| |}|jxr|du}|r|jd|||||| }|d}|jtjk(rtjtj |j tj|jjdz tj|jj}tj|| |}| |ddz} |jd|}|jtjk(rtjtj |j tj|jjdz tj|jj}tj|| |}|f}|| zS)Nr)rrrrrri)rmax)rrrrrr2) rr9r&r:risinfanyfinforclamprs)r*r<rrencoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasrrr return_dictrself_attention_outputsattention_outputs clamp_valuedo_cross_attentioncross_attention_outputsrs r.r>zT5Block.forwardsY"/A )'+/ " /q1 2126   %-- /++ M*..0 M//044t; M//044K "KK K<[YM!__R1Fd1R &3djjm!65; /"3 ' #4A6M""emm3#kkKK .224KK 3 34884?KK 3 3488 !& M |Q\ ] !24KAB4O O ' 2}5    %-- /++ M*..0 M//044t; M//044K "KK K<[YM " ' ' r/r) NNNNNNFFTrrCs@r.rrs7 -SSWZ -"#&*J r/rc`eZdZdZdeffd ZdejdejfdZxZ S)T5ClassificationHeadz-Head for sentence-level classification tasks.rFc(t|tj|j|j|_tj |j|_tj|j|j|_ y)N)r) r#r$rrKrLdenserPclassifier_dropoutrR num_labelsout_projrUs r.r$zT5ClassificationHead.__init__sZ YYv~~v~~> zzF$=$=>  &..&2C2CD r/r<returnc|j|}|j|}tj|}|j|}|j |}|SrW)rRrr&tanhrr[s r.r>zT5ClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r/) r?r@rA__doc__rr$r&rYr>rBrCs@r.rrs/7ExE U\\ellr/rceZdZUeed<dZdZdZdgZdgZ e dZ e jfdZdZxZS) T5PreTrainedModelrF transformerTrrOcvtjt}tjt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r&tensorrr)r*r input_mask dummy_inputss r.rzT5PreTrainedModel.dummy_inputss6LL. \\*- !*"&0 r/ct|||jj}t |t r$t j|j|dzy t |ttttfrt j|jjd|dzt|drE|jj s/t j|j"jd|dzt|drpt j|j$jd||jj&dzzt j(|j$j*y y t |t,rft|drYt j|j.jd|dzt j(|j.j*y y t |t0r9t j|j2jd||jj&dzzt|j2dr?|j2j*)t j(|j2j*t j|j4jd||jj&dzzt|j4drA|j4j**t j(|j4j*y y y t |t6r9t j|j8jd||jj&dzzt|j8dr?|j8j*)t j(|j8j*t j|j:jd||jj<dzzt|j:drA|j:j**t j(|j:j*y y y t |t>rt j|j@jd||jj&dzzt|j@dr?|j@j*)t j(|j@j*t j|jBjd||jj&dzzt|jBdr?|jBj*)t j(|jBj*t j|j:jd||jj<dzzt|j:drA|j:j**t j(|j:j*y y y t |tDrP|jj&}|jjF}|jjH}t j|jJjd|||zdzzt j|jLjd||dzzt j|jNjd||dzzt j|jPjd|||zdzz|jRr3t j|jTjd||dzzy y y ) zInitialize the weightsg?g)r7stdlm_head qa_outputs classifierrJN)+r# _init_weightsrFinitializer_factorrXr!init constant_r(T5ModelT5ForConditionalGenerationT5EncoderModelT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsrrrLzeros_rJT5ForTokenClassificationrrrrrErNrOrMr^r`rarprwryr~rrrrtr)r*modulefactorrLrxrzr-s r.rzT5PreTrainedModel._init_weightssD f%// fk * NN6==&3, 7   0.BX Y  LL--CVc\ Jvy)$++2Q2Q V^^22&3,Ov|, V..55CVPTP[P[PcPchlOlEmn F--223- 8 9v|, V..55CVc\R F--223- 4 5 LL,,3Ft{{GZGZ_cFc%>%@#qr'"$:&!  PQ Q&&'8D'@,O  r/)r?r@rAr__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyrr&no_gradrr3rBrCs@r.r r sa %&*#!" !F U]]_5p5pn!r/r c@eZdZfdZdZ ddZxZS)T5Stackc t||tj|j|j |_|j|_tjt|jDcgc]}t|t|dk(|c}|_ t|j |j|_tj"|j$|_|j)d|_ycc}w)NrrrhF)r#r$rr vocab_sizerL embed_tokensrsrrange num_layersrboolblockr!rkfinal_layer_normrPrQrR post_initr)r*rFir-s r.r$zT5Stack.__init__^s  LL):):FNNK ++]]]bcictct]u vXYWVa1fQR S v  !,FNN@Y@Y Zzz&"5"56  &+# ws7!Dc||_yrW)r@r*new_embeddingss r.set_input_embeddingszT5Stack.set_input_embeddingsns *r/c ||n|jj}||n|jj}| | n|jj} | | n|jj} |$|"|j rdnd} t d| d| d|&|j} |jd| d}n8||jdd} n"|j rdnd} t d| d| d |jr%|jr|rtjd d }|(|j t d |j|}| \}}|d ur|j st d|d|j rf|rr|p|jjr5tt!|jt!|j}n%t!|j}n|j sd}|jj r6t#|j||t%|tr |j&n|}nt)|j||}d}|j r|t)|j|||}| rdnd}|rdnd}|r|j rdnd}d}d}|j+|}|j,D]_}| r||fz}|||||||||||  }|d}|d}|j r | ||rdnd}|sA||dfz}|j sW||dfz}a|j/|}|j+|}| r||fz}| st1d|||||fDSt3|||||S)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer2zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fzz"T5Stack.forward..s = s)last_hidden_staterr< attentionscross_attentions)rFrroutput_hidden_statesrrsr/sizerrrr|r}r@is_encoder_decoderr r rrXrrrRrDrEtupler)r*rrrrrOrrrrXrrerr_msg_prefixr batch_size seq_lengthencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsrrr< layer_module layer_outputss r.r>zT5Stack.forwardqs"+!6IDKK+/??ZN*>*:.HXXvw  "#..*K!r;r?;I  &',,.s3K+/??ZN:>:J-XfWggtuv v  & &4==##p"    ( !_`` --i8M!, J  ?? #LTFRg!hii ??_4;;11&9$DKK8,dkk:Z'O'3$++&FO#O ;; ! !/{{+-o/BC!0 D D$ N7{{+-N +/' ??4@.G{{+5&; / +#7BD0d&7DOOrRV (,% ]3  JJ VL#$58H$H!(%/- /#"3' M*!,M *!,M#8#D0=CTaZ[0\- !/=3C2E!E??+?=QRCSBU+U(; V>--m<  ]3  1]4D D  "#%"(    9+++%1   r/) NNNNNNNNNN)r?r@rAr$rKr>rBrCs@r.r=r=]s3, + "#![ r/r=ceZdZdgZdddZdeffd ZdZdZe dd e jdzd e jdzd e jdzd e jdzd eee jdzdedzde j dzde j dzdedzdedzdedzdedzdee jezfdZxZS)rFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight shared.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightrFct||tj|j|j |_tj|}d|_ d|_ t||_ tj|}d|_ |j|_t||_|j!yNFT)r#r$rrr?rLr$copydeepcopyrsrr=encodernum_decoder_layersrBdecoderrFr*rFencoder_configdecoder_configr-s r.r$zT5Model.__init__s  ll6#4#4fnnE v.$)!#( ~. v.$(!$*$=$=!~.  r/c|jSrWr$r*s r.get_input_embeddingszT5Model.get_input_embeddings* {{r/c~||_|jj||jj|yrWr$rorKrqrIs r.rKzT5Model.set_input_embeddings--$  )).9 )).9r/Nrrrrencoder_outputsrrOdecoder_inputs_embedsrrrXrrc D| | n|jj} | | n|jj} ||j|||| | | }nI| rGt |t s7t |dt |dkDr|dndt |dkDr|dnd}|d}|j||||||| | | |  }| s||zSt|j|j|j|j|j|j|j|jS) aV input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, T5Model >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5Model.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for T5Model. >>> # This is not needed for torch's T5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```NrrrOrrXrrrr1rUr<rV rrrOrrrrrrXr)rUrdecoder_hidden_statesdecoder_attentionsrWencoder_last_hidden_staterencoder_attentions)rFrrrorXrlenrqrrUrr<rVrW)r*rrrrr}rrOr~rrrXrrr<decoder_outputss r.r>zT5Model.forward2sOD"+!6IDKKrBrCs@r.rrsx Q*&(7'6 x": .23759:>BF(,-159!%)-,0#'q ##d*q ))D0q !++d2 q !& 0 04 7 q uU%6%6784? q q ||d*q  %||d2q $;q  $;q #Tkq D[q  u  !$6 6q q r/rz: T5 Model with a `language modeling` head on top. ) custom_introceZdZdgZddddZdeffd ZdZdZe dd e jdzd e jdzd e jdzd e jdzd eee jdzdedzde jdzde jdzde jdzdedzdedzdedzdedzdee jezfdZde jfdZxZS)r rfrg)zlm_head.weightrirjrFct|||j|_t j |j |j|_tj|}d|_ d|_ t||_ tj|}d|_ |j|_t||_t j"|j|j d|_|j'y)NFTrI)r#r$rL model_dimrrr?r$rmrnrsrr=rorprBrqrKrrFrrs r.r$z#T5ForConditionalGeneration.__init__s  ll6#4#4fnnE v.$)!#( ~. v.$(!$*$=$=!~. yy1B1BO  r/c|jSrWrvrws r.rxz/T5ForConditionalGeneration.get_input_embeddingsryr/c~||_|jj||jj|yrWr{rIs r.rKz/T5ForConditionalGeneration.set_input_embeddingsr|r/Nrrrrr}rrOr~labelsrrrXrrc | | n|jj} | | n|jj} ||j|||| | | }nI| rGt |t s7t |dt |dkDr|dndt |dkDr|dnd}|d}| |||j| }|j||||||| | | |  }|d}|jjr||jdzz}|j|}d}| ^td }| j|j} ||jd |j!d | jd }| s|f|ddz|z}||f|zS|St#|||j$|j&|j(|j*|j,|j&|j( S) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, T5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer( ... "summarize: studies have shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model.generate(input_ids) >>> print(tokenizer.decode(outputs[0], skip_special_tokens=True)) >>> # studies have shown that owning a dog is good for you. ```Nrrrr1rrrr, ignore_indexr2 losslogitsrrrrWrrr)rFrrrorXrrr3rqscale_decoder_outputsrrrr4rrrYrrr<rVrWrU)r*rrrrr}rrOr~rrrrXrrr<rsequence_output lm_logitsrloss_fctoutputs r.r>z"T5ForConditionalGeneration.forwardsR"+!6IDKKrrBrCs@r.r r s Q*&*'6'6 x*: .23759:>=A(,26:>*.!%)-,0#'M ##d*M ))D0M !++d2 M !& 0 04 7 M uU\\23d: M M ((4/M  %0047M   4'M $;M  $;M #TkM D[M u  !O 3!M M ^)ELL)r/r ceZdZddiZdgZdeffd ZdZdZe dd e jdzd e jdzd e jdzd e dzd e dzde dzdee jezfdZxZS)r!rirgrqrFct||tj|j|j |_|}d|_d|_t||_ |jy)NF) r#r$rrr?rLr$rrZr=rorF)r*rFrsr-s r.r$zT5EncoderModel.__init__lsY  ll6#4#4fnnE #( ,1)~.  r/c|jSrWrvrws r.rxz#T5EncoderModel.get_input_embeddingsxryr/cH||_|jj|yrW)r$rorKrIs r.rKz#T5EncoderModel.set_input_embeddings{s$  )).9r/NrrrOrrXrrc h||n|jj}|j||||||}|S)aI input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). Example: ```python >>> from transformers import AutoTokenizer, T5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5EncoderModel.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```r)rFrro) r*rrrOrrXrrr}s r.r>zT5EncoderModel.forwardsHD&1%rBrCs@r.r!r!gs7I*4& x :.23726)-,0#',##d*,))D0,((4/ ,  $; , #Tk ,D[, u  !O 3,,r/r!z T5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c~eZdZdgZdeffd Ze ddejdzdejdzdejdzdejdzd e ejdzd ejdzd ejdzd ejdzd e dzde dzde dzde dzde ezfdZxZS)T5ForSequenceClassificationrfrFct||t||_t ||_|j yrW)r#r$rr rclassification_headrFrUs r.r$z$T5ForSequenceClassification.__init__s5  "6?#7#?  r/Nrrrrr}rOr~rrrrXrrc R| | n|jj} |d} |$|"td|jj| || t d|j |}|j|||||||| | | |  }|d}|j|jjj|j}ttj|jdj!dk(d|j"\}}}||ddf}t|j"d|zdk\d |j%|d |ddd ddf}|j'|}d}||j|j}|jj(|jj*dk(rd |j_nv|jj*dkDrL|j,tj.k(s|j,tj0k(rd |j_nd |j_|jj(d k(rSt3}|jj*dk(r&||j5|j5}n|||}n|jj(d k(rGt7}||j%d |jj*|j%d }n,|jj(d k(rt9}|||}| s|f|ddz}||f|zS|St;|||j<|j>|j@|jB|jD|jF|jH S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrrr}rOr~rrrXrrrz7All examples must have the same number of tokens.z3Each example must contain at least one token.r2 regressionsingle_label_classificationmulti_label_classificationr)%rFrNotImplementedErrorr-r?r/r3r eq eos_token_idr4rrr&unique_consecutivesumnumelrrr problem_typerr9rrrsqueezerrrrrrrWrrr)r*rrrrr}rOr~rrrrXrrrreos_maskr]_r+selectedsentence_representationrrrrs r.r>z#T5ForSequenceClassification.forwardsp`&1%>KbKbJcd   $)>)F  U !% 1 1) < "" )/#9+'"7/!5##  "!*<< 8 89<<_=S=ST  $ $X\\!_ 5 ; ; = B E &5%:%:" A{"8Q;/ NN1  +q 0 A #+-- B "LQPRTUX"V))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r/r)r?r@rArrr$rr&rrYlistrrCr[rr>rBrCs@r.rrsU+s)s&x.2.259:>:>26:>*.!%)-,0#'F ##d*F  t+F !++d2 F !& 0 04 7 F e//047 F ((4/F  %0047F   4'F $;F  $;F #TkF D[F  0 0F F r/rceZdZdeffd Ze d dejdzdejdzdejdzdejdzdedzd edzd edzd e eje zfd Z xZ S)r(rFc,t|||j|_t||_t j |j|_t j|j|j|_ |jyrW) r#r$rr!r rrPrrRrKr+rrFrUs r.r$z!T5ForTokenClassification.__init__Lsj   ++)&1zz&";";< ))F$6$68I8IJ r/NrrrOrrrXrrc ||n|jj}|j||||||} | d} |j| } |j | } d} |z T5ForTokenClassification.forwardWs4&1%rBrCs@r.r(r(Js x *..2-1&*)-,0#'6 <<$&6  t+6 ||d* 6  t# 6  $; 6 #Tk6 D[6  u|| 4 46 6 r/r(ceZdZdgZdddZdeffd ZdZdZe dd e jdzd e jdzd e jdzd e jdzd eee jdzde jdzde jdzde jdzde jdzdedzdedzdedzdedzdee jezfdZxZS)r"rfrgrhrFc$t|||j|_t j |j |j|_tj|}d|_ d|_ t||_ tj|}d|_ |j|_t||_|j"|_t j$|j&|j"|_|j+yrl)r#r$rLrrrr?r$rmrnrsrr=rorprBrqrrKr+rrFrrs r.r$zT5ForQuestionAnswering.__init__s  ll6#4#4fnnE v.$)!#( ~. v.$(!$*$=$=!~.  ++))F$6$68I8IJ r/c|jSrWrvrws r.rxz+T5ForQuestionAnswering.get_input_embeddingsryr/c~||_|jj||jj|yrWr{rIs r.rKz+T5ForQuestionAnswering.set_input_embeddingsr|r/Nrrrrr}start_positions end_positionsrOr~rrrXrrc | | n|jj} | | n|jj} ||d} | | | td|j |}| | n|jj} | | n|jj} ||j |||| | | }nI| rGt |ts7t|dt|dkDr|dndt|dkDr|dnd}|d}|j||| d||| | | | }|d}|j|}|jdd \}}|jd j}|jd j}d}||t|jdkDr*|jd j|j }t|jdkDr*|jd j|j }|jd}|j#d|}|j#d|}t%| }|||}|||}||zdz }| s||f|ddz|z}||f|zS|St'||||j(|j*|j,|j.|j0|j*|j, S)az input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. NFrrrrr1rrr2rr) r start_logits end_logitsrrrrWrrr)rFrrr/r3rorXrrrqrsplitrrrYr4rrrrrr<rVrWrU)r*rrrrr}rrrOr~rrrXrrr<rrrrr total_loss ignored_indexr start_lossend_lossrs r.r>zT5ForQuestionAnswering.forwards0\&1%)F  U !% 1 1) < !*!6IDKKrBrCs@r.r"r"s*r)s&'6'6 x,: .23759:>=A371526:>!%)-,0#'I ##d*I ))D0I !++d2 I !& 0 04 7 I uU\\23d: I ))D0I ''$.I ((4/I  %0047I $;I  $;I #TkI D[I u  !$G G!I I r/r")r!r rr r"rr()Ar rmrr&rtorch.nnrrrrNrr activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrconfiguration_t5r get_loggerr?r|Moduler!rEr^rfrprrrrr r=rr r!rr(r"__all__rPr/r.rs AA&!CC)J9.^^&   H %+"))+2bii,299: $"))D299BBII>W (W t299$_!_!_!Do o dT T T n x)!2Ox)  x)vD&DDN R "3R  R jC 0C C Lo .o o d r/