^j dZddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl mZmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZdd l m!Z!ddl"m#Z#ddl$m%Z%m&Z&ddl'm(Z(e&jRe*Z+GddejXZ-GddejXZ.GddejXZ/GddejXZ0GddejXZ1GddejXZ2GddejXZ3Gd d!eZ4Gd"d#ejXZ5Gd$d%ejXZ6Gd&d'ejXZ7Gd(d)ejXZ8e%Gd*d+e!Z9e%d,-Gd.d/e9Z:e%Gd0d1e9Z;e%d2-Gd3d4e9eZ<e%d5-Gd6d7e9Z=e%Gd8d9e9Z>e%Gd:d;e9Z?e%Gd<d=e9Z@gd>ZAy)?zPyTorch RemBERT model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)auto_docstringlogging) RemBertConfigc eZdZdZfdZ d dej dzdej dzdej dzdejdzded ejf d Z xZ S) RemBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c|t|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)ddy)N) padding_idxeps position_idsrF) persistent)super__init__r Embedding vocab_sizeinput_embedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandselfconfig __class__s w/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/rembert/modeling_rembert.pyr)zRemBertEmbeddings.__init__2s !||   v::H[H[ $&<<0N0NPVPkPk#l %'\\&2H2H&JeJe%f"f&A&AvG\G\]zz&"<"<=   ELL)G)GHOOPWXej  N input_idstoken_type_idsr$ inputs_embedspast_key_values_lengthreturnc||j}n|jdd}|d}||jdd|||zf}|:tj|tj|jj }||j |}|j|}||z} |j|} | | z } |j| } |j| } | S)Nr&rdtypedevice) sizer$r9zeroslongrJr.r2r0r3r7) r=rBrCr$rDrE input_shape seq_lengthr2 embeddingsr0s r@forwardzRemBertEmbeddings.forwardBs  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !"[[EJJtO`O`OgOghN   00;M $ : :> J"%:: "66|D)) ^^J/ \\*- rA)NNNNr) __name__ __module__ __qualname____doc__r)r9 LongTensor FloatTensorintTensorrQ __classcell__r?s@r@rr/sQ $.2260426&' ##d*((4/&&-  ((4/  !$  rArcVeZdZfdZdej dej fdZxZS) RemBertPoolerct|tj|j|j|_tj |_yN)r(r)rLinear hidden_sizedenseTanh activationr<s r@r)zRemBertPooler.__init__es9 YYv1163E3EF '')rA hidden_statesrFc\|dddf}|j|}|j|}|S)Nr)rbrd)r=refirst_token_tensor pooled_outputs r@rQzRemBertPooler.forwardjs6+1a40 #56  6 rArRrSrTr)r9rYrQrZr[s@r@r]r]ds#$ U\\ellrAr]c eZdZd fd Z d dej dej dzdej dzdedzdede f d Z xZ S) RemBertSelfAttentionNct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|_|j"|_||_y)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())r(r)ranum_attention_headshasattr ValueErrorrXattention_head_size all_head_sizerr`querykeyvaluer5attention_probs_dropout_probr7 is_decoder layer_idxr=r>ryr?s r@r)zRemBertSelfAttention.__init__ts0     : : :a ?PVXhHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F  &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$2D.- /"=*,33DNNCHHI.55dnnELLKQ--cr2QBQ8P8PQH055h?II!QOI**^499(CMMaQRSK*)=)D)DYP[]a]k]k)l& ;%*_FY*ZAEO..t~~>!<< Y5H5HR5PQ+dii8P8P.QQ  %/.@ --//0@b/I,,7 _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD o--rAr_NNNF rRrSrTr)r9rYrWr booltuplerQrZr[s@r@rkrkssz#048:>(,"' @.||@.))D0@. %0047 @.  @.  @. @.rArkcneZdZfdZdej dej dej fdZxZS)RemBertSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr") r(r)rr`rarbr3r4r5r6r7r<s r@r)zRemBertSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rAre input_tensorrFcr|j|}|j|}|j||z}|Sr_rbr7r3r=rers r@rQzRemBertSelfOutput.forward7 =1  ]3 }|'CD rArir[s@r@rr1> U\\RWR^R^rArceZdZd fd Z d dej dej dzdej dzdedzdedzde ej f d Z xZ S) RemBertAttentionNcft|t|||_t ||_y)Nry)r(r)rkr=routputrzs r@r)zRemBertAttention.__init__s( (9E '/ rArer{r|r}r~rFc n|j|||||}|j|d|}|f|ddz} | S)Nr{r|r}r~rr)r=r) r=rer{r|r}r~r self_outputsattention_outputoutputss r@rQzRemBertAttention.forwardsVyy )"7+/ !  ;;|A F#% QR(88rAr_rrr[s@r@rrs048:>(,). ||))D0 %0047     $;  u|| rArcVeZdZfdZdej dej fdZxZS)RemBertIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr_) r(r)rr`raintermediate_sizerbr hidden_actstrr intermediate_act_fnr<s r@r)zRemBertIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rArerFcJ|j|}|j|}|Sr_)rbrr=res r@rQzRemBertIntermediate.forwards& =1 00? rArir[s@r@rrs#9U\\ellrArcneZdZfdZdej dej dej fdZxZS) RemBertOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r(r)rr`rrarbr3r4r5r6r7r<s r@r)zRemBertOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rArerrFcr|j|}|j|}|j||z}|Sr_rrs r@rQzRemBertOutput.forwardrrArir[s@r@rrrrArceZdZd fd Z d dej dej dzdej dzdej dzdedzdedzd e ej fd Z d Z xZ S) RemBertLayerNcht||j|_d|_t |||_|j |_|j|_|jr,|j st|dt |||_ t||_ t||_ y)Nrz> should be used as a decoder model if cross attention is addedr)r(r)chunk_size_feed_forward seq_len_dimr attentionrxadd_cross_attentionrqcrossattentionr intermediaterrrzs r@r)zRemBertLayer.__init__s '-'E'E$)&)< ++#)#=#=  # #?? D6)g!hii"26Y"OD /7#F+ rArer{r|encoder_attention_maskr}r~rFc @|j||||}|d} |dd} |jr@|>t|dstd|d|j | ||||} | d} | | ddz} t |j |j|j| } | f| z} | S)N)r{r~r}rrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrxrprqrrfeed_forward_chunkrr) r=rer{r|rr}r~rself_attention_outputsrrcross_attention_outputs layer_outputs r@rQzRemBertLayer.forward%s"& )/+ "0"  2!4(, ??4@4!12 =dVDDD '+&9&9 5&; /"3 ':' # 7q9  7 ;;G0  # #T%A%A4CSCSUe  /G+rAcL|j|}|j||}|Sr_)rr)r=rintermediate_outputrs r@rzRemBertLayer.feed_forward_chunkQs,"//0@A{{#68HI rAr_)NNNNF) rRrSrTr)r9rYrWr rrrQrrZr[s@r@rrs ,$48:>;?(,).)||)))D0) %0047 ) !& 1 1D 8 )  ) $;) u|| )XrArceZdZfdZ ddej dej dzdej dzdej dzdedzdedzd ed ed ed e e zfd Z xZ S)RemBertEncoderc 2t|||_tj|j |j |_tjt|jDcgc]}t||c}|_ d|_ ycc}w)NrF)r(r)r>rr`r,raembedding_hidden_mapping_in ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r=r>ir?s r@r)zRemBertEncoder.__init__Xsq  +-99V5P5PRXRdRd+e(]]uU[UmUmOn#o!L1$E#op &+#$ps,BNrer{r|rr} use_cacher~output_hidden_states return_dictrFc n|jr%|jr|rtjdd}|r6|4t t |j t |j }|j|}|rdnd} |rdnd} |r|j jrdnd} t|jD]K\}}|r| |fz} |||||||}|d}|s#| |dfz} |j jsC| |dfz} M|r| |fz} | std||| | | fDSt||| | | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r>rrrc3$K|]}|| ywr_r).0vs r@ z)RemBertEncoder.forward..s = s)last_hidden_stater}re attentionscross_attentions) rtraininglogger warning_oncer r r>rr enumeraterrr)r=rer{r|rr}rr~rrrall_hidden_statesall_self_attentionsall_cross_attentionsr layer_module layer_outputss r@rQzRemBertEncoder.forward`s  & &4==##p" 01,dkk2RT`hlhshsTtuO88G "6BD$5b4%64;;;Z;Zr`d(4 VOA|#$58H$H!(%&! M*!,M &9]1=M;?(,!%"'%* D ||D ))D0D  %0047 D !& 1 1D 8 D  D $;D  D #D D  : :D rArcVeZdZfdZdej dej fdZxZS)RemBertPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r(r)rr`rarbrrrr transform_act_fnr3r4r<s r@r)z'RemBertPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrArerFcl|j|}|j|}|j|}|Sr_)rbrr3rs r@rQz&RemBertPredictionHeadTransform.forwards4 =1 --m< }5 rArir[s@r@rrs$UU\\ellrArcVeZdZfdZdej dej fdZxZS)RemBertLMPredictionHeadcnt|tj|j|j |_tj|j |j|_t|j|_ tj|j |j|_ yr)r(r)rr`raoutput_embedding_sizerbr+decoderr rrdr3r4r<s r@r)z RemBertLMPredictionHead.__init__sz YYv1163O3OP yy!=!=v?P?PQ  !2!23f&B&BH]H]^rArerFc|j|}|j|}|j|}|j|}|Sr_)rbrdr3rrs r@rQzRemBertLMPredictionHead.forwards@ =1  6 }5  ]3 rArir[s@r@rrs$_U\\ellrArcVeZdZfdZdej dej fdZxZS)RemBertOnlyMLMHeadcBt|t||_yr_)r(r)r predictionsr<s r@r)zRemBertOnlyMLMHead.__init__s 26:rAsequence_outputrFc(|j|}|Sr_)r)r=rprediction_scoress r@rQzRemBertOnlyMLMHead.forwards ,,_=  rArir[s@r@rrs#;!u||! !rArc2eZdZUeed<dZdZfdZxZS)RemBertPreTrainedModelr>rembertTct||t|trZt j |j tj|j jdjdyy)Nr&r%) r( _init_weightsrrinitcopy_r$r9r:rr;)r=moduler?s r@rz$RemBertPreTrainedModel._init_weightss[ f% f/ 0 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h 1rA) rRrSrTr__annotations__base_model_prefixsupports_gradient_checkpointingrrZr[s@r@r r s! !&*#iirAr a  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introcfeZdZdfd ZdZdZe ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd e dzd e dzde dzde dzde dzde e zfdZxZS) RemBertModelct||||_t||_t ||_|r t|nd|_|jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r(r)r>rrPrencoderr]pooler post_init)r=r>add_pooling_layerr?s r@r)zRemBertModel.__init__sM   +F3%f- /@mF+d  rAc.|jjSr_rPr.r=s r@get_input_embeddingsz!RemBertModel.get_input_embeddingss...rAc&||j_yr_r)r=rvs r@set_input_embeddingsz!RemBertModel.set_input_embeddingss*/'rANrBr{rCr$rDr|rr}rr~rrrFc 8| | n|jj} | | n|jj} | | n|jj} |jjr| | n|jj } nd} | | t d|#|j|||j}n!||jdd}n t d|\}}| |jn |j}|dn|j}|tj|||zf|}|&tj|tj|}|j|||||}t!|j|| }|t!|j||| }|j#|||||| | | | }|d}|j$|j%|nd}| s ||f|d dzSt'|||j(|j*|j,|j. S)NFzDYou cannot specify both input_ids and inputs_embeds at the same timer&z5You have to specify either input_ids or inputs_embedsr)rJrH)rBr$rCrDrE)r>rDr{)r>rDr{r|)r{r|rr}rr~rrr)r pooler_outputr}rerr)r>r~rrrxrrq%warn_if_padding_and_no_attention_maskrKrJget_seq_lengthr9onesrLrMrPrrrrr}rerr)r=rBr{rCr$rDr|rr}rr~rrrrN batch_sizerOrJrEembedding_outputencoder_outputsrrhs r@rQzRemBertModel.forwardsV"2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T&5&=?CaCaCc  !"ZZ*jCY6Y)ZdjkN  !"[[EJJvVN??%)'#9 + 3;;*)  " -%>{{.5&; & ",, )"7#9+/!5#'  *!,8< 8O O4UY #]3oab6II I;-'+;;)77&11,==   rA)T) NNNNNNNNNNNN)rRrSrTr)r!r#rr9rVrWr rrrrQrZr[s@r@rrsB /0.226260426:>;?(,!%)-,0#'] ##d*] ((4/] ((4/ ] &&- ] ((4/ ]  %0047] !& 1 1D 8] ] $;]  $;] #Tk] D[]  = =] ] rArcleZdZfdZdZdZe ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd e dzde dzde dzde e zfdZ xZS)RemBertForMaskedLMct|||jrtj dt |d|_t||_|jy)NznIf you want to use `RemBertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Fr r(r)rxrwarningrr rclsrr<s r@r)zRemBertForMaskedLM.__init__fsR     NN1  $FeD %f- rAcB|jjjSr_r2rrr s r@get_output_embeddingsz(RemBertForMaskedLM.get_output_embeddingsuxx##+++rAc:||jj_yr_r4r=new_embeddingss r@set_output_embeddingsz(RemBertForMaskedLM.set_output_embeddingsx'5$rANrBr{rCr$rDr|rlabelsr~rrrFc | | n|jj} |j|||||||| | |  } | d}|j|}d}|Ft }||j d|jj |j d}| s|f| ddz}||f|zS|St||| j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. N) r{rCr$rDr|rr~rrrr&rlosslogitsrer) r>rr r2rrr+rrer)r=rBr{rCr$rDr|rr<r~rrrrrr masked_lm_lossloss_fctrs r@rQzRemBertForMaskedLM.forward{s,&1%;?*.)-,0#'5 ##d*5 ((4/5 ((4/ 5 &&- 5 ((4/ 5  %00475 !& 1 1D 85   4'5  $;5 #Tk5 D[5   5 5 rAr-zS RemBERT Model with a `language modeling` head on top for CLM fine-tuning. c!eZdZfdZdZdZe ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd e dzd ejdzde dzde dzde dzde dzde ejzdeezfdZxZS)RemBertForCausalLMct|||jstj dt |d|_t||_|jy)NzOIf you want to use `RemBertForCausalLM` as a standalone, add `is_decoder=True.`Fr/r0r<s r@r)zRemBertForCausalLM.__init__sL    NNl m#FeD %f- rAcB|jjjSr_r4r s r@r5z(RemBertForCausalLM.get_output_embeddingsr6rAc:||jj_yr_r4r8s r@r:z(RemBertForCausalLM.set_output_embeddingsr;rANrBr{rCr$rDr|rr}r<rr~rrlogits_to_keeprFc | | n|jj} |j||||||||| | | |  }|d}t|tr t | dn|}|j |dd|ddf}d}| *|jd|| |jjd|}| s|f|ddz}||f|zS|St|||j|j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, RemBertForCausalLM, RemBertConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/rembert") >>> config = RemBertConfig.from_pretrained("google/rembert") >>> config.is_decoder = True >>> model = RemBertForCausalLM.from_pretrained("google/rembert", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```N) r{rCr$rDr|rr}rr~rrr)r@r<r+r)r?r@r}rerrr)r>rr rrXslicer2 loss_functionr+rr}rerr)r=rBr{rCr$rDr|rr}r<rr~rrrHrrre slice_indicesr@r?rs r@rQzRemBertForCausalLM.forwards)R&1%SV8W~ot4]k -=!(;<=  %4%%pVFt{{OeOepiopDY,F)-)9TGf$ Ev E0#33!//))$55   rA)NNNNNNNNNNNNNr)rRrSrTr)r5r:rr9rVrWr rrXrYrrrQrZr[s@r@rDrDsr ,6.226260426:>;?(,*.!%)-,0#'-.M ##d*M ((4/M ((4/ M &&- M ((4/ M  %0047M !& 1 1D 8M M   4'M $;M  $;M #TkM D[M ell*M " 2 2#M M rArDz RemBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c eZdZfdZe ddej dzdej dzdejdzdej dzdej dzdejdzd edzd edzd edzd e e zfd Z xZ S) RemBertForSequenceClassificationc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr_ r(r) num_labelsrr rr5classifier_dropout_probr7r`ra classifierrr<s r@r)z)RemBertForSequenceClassification.__init__$si   ++#F+ zz&"@"@A ))F$6$68I8IJ rANrBr{rCr$rDr<r~rrrFc >| | n|jj} |j|||||||| } | d} |j| } |j | } d}||jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt}|j dk(r&|| j|j}n|| |}n|jj dk(r=t}|| jd|j |jd}n,|jj dk(rt}|| |}| s| f| ddz}||f|zS|St!|| | j"| j$ S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr{rCr$rDr~rrr regressionsingle_label_classificationmulti_label_classificationr&rr>)r>rr r7rS problem_typerQrIr9rMrXrsqueezerrrrrer)r=rBr{rCr$rDr<r~rrrrrhr@r?rBrs r@rQz(RemBertForSequenceClassification.forward.s(&1%) r>rrrrKr r7rSrrrer)r=rBr{rCr$rDr<r~rrr num_choicesrrhr@reshaped_logitsr?rBrs r@rQz RemBertForMultipleChoice.forwardsX&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  ,, ))%'/!5#     ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rAr[) rRrSrTr)rr9rWrVrrrrQrZr[s@r@r]r]vs/337261526*.)-,0#'W $$t+W ))D0W ((4/ W ''$. W ((4/ W   4'W  $;W #TkW D[W  * *W W rAr]c eZdZfdZe ddej dzdej dzdejdzdej dzdej dzdejdzd edzd edzd edzd e e zfd Z xZ S)RemBertForTokenClassificationc0t|||j|_t|d|_t j |j|_t j|j|j|_ |jyNFr/rPr<s r@r)z&RemBertForTokenClassification.__init__sk   ++#FeD zz&"@"@A ))F$6$68I8IJ rANrBr{rCr$rDr<r~rrrFc | | n|jj} |j|||||||| } | d} |j| } |j | } d}|) r>rr r7rSrrrQrrer)r=rBr{rCr$rDr<r~rrrrrr@r?rBrs r@rQz%RemBertForTokenClassification.forwards$&1%rr rjsplitrZlenrKclamp_rrrer)r=rBr{rCr$rDrkrlr~rrrrrr@rorp total_loss ignored_indexrB start_lossend_lossrs r@rQz#RemBertForQuestionAnswering.forward,s&1%rsF AA&!CC)69   .6,0   H %1 1j BII V.299V.t  ryy8 "))  BII ?-?DM RYYM bRYY" bii "!!i_ii  u )u  u pL /L L ^ a /a  a H P '=P  P fc 5c c L> $:> > BJ "8J J Z rA