^j"dZddlmZddlmZddlZddlmZddlmZm Z m Z ddl m Z dd lmZdd lmZmZmZdd lmZdd lmZmZdd lmZddlmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3ddl4m5Z5ddl6m7Z7e0jpe9Z:GddejvZ< d\dejvdejzdejzdejzdejzdzde>dzde>d e)e.fd!Z?Gd"d#ejvZ@Gd$d%ejvZAGd&d'ejvZBGd(d)ejvZCGd*d+ejvZDGd,d-ejvZEGd.d/eZFGd0d1ejvZGGd2d3ejvZHGd4d5ejvZIGd6d7ejvZJGd8d9ejvZKGd:d;ejvZLGd<d=ejvZMe/Gd>d?e'ZNe/d@AeGdBdCe-ZOe/dDAGdEdFeNZPe/dGAGdHdIeNZQe/dJAGdKdLeNeZRe/GdMdNeNZSe/dOAGdPdQeNZTe/dRAGdSdTeNZUe/GdUdVeNZVe/GdWdXeNZWe/GdYdZeNZXgd[ZYy)]zPyTorch BERT model.)Callable) dataclassN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) BertConfigc eZdZdZfdZ d dej dzdej dzdej dzdejdzded ejf d Z xZ S) BertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j#dt%j*|j,j/t$j0dy) N) padding_idxeps position_idsr'F) persistenttoken_type_ids)dtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandzerosr/sizelongselfconfig __class__s q/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/bert/modeling_bert.pyr6zBertEmbeddings.__init__8s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  N input_idsr3r/ inputs_embedspast_key_values_lengthreturnc||j}n|jdd}|\}}||jdd|||zf}|t|drT|jj |j dd} t j| d|} | j ||}n:t j|t j|jj}||j|}|j|} || z} |j|} | | z} |j| } |j| } | S)Nr1r3rr')dimindex)r4device)rJr/hasattrr3rHshaperFgatherrIrKrYr;r?r=r@rD) rMrRr3r/rSrT input_shape batch_size seq_lengthbuffered_token_type_idsr? embeddingsr=s rPforwardzBertEmbeddings.forwardHsP  #..*K',,.s3K!, J  ,,Q0FVlIl0l-lmL  !t-.*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? J!W!&[ SWSdSdSkSk!l   00;M $ : :> J"%:: "66|D"55 ^^J/ \\*- rQ)NNNNr) __name__ __module__ __qualname____doc__r6rF LongTensor FloatTensorintTensorrb __classcell__rOs@rPr*r*5sQ $.2260426&' (##d*(((4/(&&- ( ((4/ ( !$ ( (rQr*modulequerykeyvalueattention_maskscalingrDkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)Nr1r rW)ptrainingr') rJrFmatmul transposer functionalsoftmaxrDry contiguous) rmrnrorprqrrrDrs attn_weights attn_outputs rPeager_attention_forwardrss**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$rQc eZdZd fd Z d dej dej dzdedzdee de ej f dZ xZ S) BertSelfAttentionNc@t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_|j&|_||_||_yNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()ru)r5r6r9num_attention_headsrZ ValueErrorrNriattention_head_size all_head_sizerrrLinearrnrorprBattention_probs_dropout_probrD is_decoder is_causal layer_idxrMrNrrrOs rPr6zBertSelfAttention.__init__sP     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9F|} t|tr |j} | j|| |j\}} tj|jjt } | |||| |f|j"sdn|j$j&|j(d|\} } | j*g|dj-} | | fS)Nr1r'rvrDrr)r[rrnviewr{rorp isinstancerself_attention_cacheupdaterr get_interfacerN_attn_implementationrryrDrxrrreshaper~)rMrrqrrsr] hidden_shape query_layer key_layer value_layercurrent_past_key_valuesattention_interfacerrs rPrbzBertSelfAttention.forwards$))#2. CCbC$*B*BC 5djj/44lCMMaQRS 0DHH]+00,?II!QO 4djj/44lCMMaQRS  &&5 #/+>?*9*N*N'&=%C%CI{\`\j\j%k "I{(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((rQFN)NN rcrdrer6rFrjrhr rr!tuplerbrkrls@rPrrsg#648(, ')||')))D0') ') +, ') u||  ')rQrceZdZd fd Z d dej dej dzdej dzdedzdee de ej f d Z xZ S) BertCrossAttentionNct||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_||_||_yr)r5r6r9rrZrrNrirrrrrrrnrorprBrrDrrrs rPr6zBertCrossAttention.__init__sC     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9FB**4>>:(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((rQr)NNN) rcrdrer6rFrjrhrrr!rrbrkrls@rPrrs#4;?376: 1)||1) %00471)))D0 1) -t3 1) +, 1) u|| 1)rQrcneZdZfdZdej dej dej fdZxZS)BertSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr-) r5r6rrr9denser@rArBrCrDrLs rPr6zBertSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rQr input_tensorrUcr|j|}|j|}|j||z}|SNrrDr@rMrrs rPrbzBertSelfOutput.forward%7 =1  ]3 }|'CD rQrcrdrer6rFrjrbrkrls@rPrr1> U\\RWR^R^rQrceZdZd fd Z d dej dej dzdej dzdej dzdedzdee d e ej fd Z xZ S) BertAttentionNct|||_|rtnt}|||||_t ||_y)Nrr)r5r6is_cross_attentionrrrMroutput)rMrNrrrattention_classrOs rPr6zBertAttention.__init__-s= "40B,HY#Fi9U $V, rQrrqrencoder_attention_maskrrsrUc |js|n|}|j|f|||d|\}}|j||}||fS)N)rrqr)rrMr) rMrrqrrrrsattention_outputrs rPrbzBertAttention.forward4sd04/F/FLb)2 * "7)+ *  * &, ;;'7G--rQ)FNFNNNNrrls@rPrr,s-48:>;?(, .||.))D0. %0047 . !& 1 1D 8 .  .+,. u|| .rQrcVeZdZfdZdej dej fdZxZS)BertIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r5r6rrr9intermediate_sizerr hidden_actstrr intermediate_act_fnrLs rPr6zBertIntermediate.__init__Js] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rQrrUcJ|j|}|j|}|Sr)rrrMrs rPrbzBertIntermediate.forwardRs& =1 00? rQrrls@rPrrIs#9U\\ellrQrcneZdZfdZdej dej dej fdZxZS) BertOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r5r6rrrr9rr@rArBrCrDrLs rPr6zBertOutput.__init__Ys` YYv779K9KL f&8&8f>S>STzz&"<"<= rQrrrUcr|j|}|j|}|j||z}|Srrrs rPrbzBertOutput.forward_rrQrrls@rPrrXrrQrceZdZd fd Z d dej dej dzdej dzdej dzdedzdee d ej fd Z d Z xZ S) BertLayerNct||j|_d|_t ||j ||_|j |_|j|_|jr.|j st|dt |d|d|_ t||_ t||_ y)Nr'rz> should be used as a decoder model if cross attention is addedFT)rrr)r5r6chunk_size_feed_forward seq_len_dimrr attentionadd_cross_attentionrcrossattentionr intermediaterr)rMrNrrOs rPr6zBertLayer.__init__gs '-'E'E$&v9J9JV_` ++#)#=#=  # #?? D6)g!hii"/##' #D  -V4 ( rQrrqrrrrsrUc |j||fd|i|\}}|} |jr:|8t|dstd|d|j|d||fd|i|\} }| } t |j |j|j| } | S)Nrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) rrrZrrrfeed_forward_chunkrr) rMrrqrrrrsself_attention_output_rcross_attention_output layer_outputs rPrbzBertLayer.forwardzs$24>>  $ ,$  $  q 1 ??4@4!12 =dVDDD )<(;(;%%& ) !0 )  ) % "A 6 0  # #T%A%A4CSCSUe rQcL|j|}|j||}|Sr)rr)rMrintermediate_outputrs rPrzBertLayer.feed_forward_chunks,"//0@A{{#68HI rQrr) rcrdrer6rFrjrhr rr!rbrrkrls@rPrrfs),48:>;?(, %||%))D0% %0047 % !& 1 1D 8 %  %+,% %NrQrceZdZfdZ d dej dej dzdej dzdej dzdedzdedzd e e d e ej e zfd Z xZS) BertEncoderc t|||_tjt |j Dcgc]}t||c}|_ycc}w)N)r) r5r6rNr ModuleListrangenum_hidden_layersrlayer)rMrNirOs rPr6zBertEncoder.__init__sF  ]]ERXRjRjLk#lqIf$B#lm #lsANrrqrrr use_cachersrUc t|jD]\}} | |||f||d|}t||r|SdS)N)rr)last_hidden_stater) enumeraterr) rMrrqrrrrrsr layer_modules rPrbzBertEncoder.forwardsn )4 OA|(%(> /   M 9+/8O  >B  rQ)NNNNN)rcrdrer6rFrjrhr boolrr!rrrbrkrls@rPrrsn48:>;?(,!% || ))D0  %0047  !& 1 1D 8    $; +,  u|| H H rQrcVeZdZfdZdej dej fdZxZS) BertPoolerct|tj|j|j|_tj |_yr)r5r6rrr9rTanh activationrLs rPr6zBertPooler.__init__s9 YYv1163E3EF '')rQrrUc\|dddf}|j|}|j|}|S)Nr)rr)rMrfirst_token_tensor pooled_outputs rPrbzBertPooler.forwards6+1a40 #56  6 rQrrls@rPrrs#$ U\\ellrQrcVeZdZfdZdej dej fdZxZS)BertPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r5r6rrr9rrrrr transform_act_fnr@rArLs rPr6z$BertPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrQrrUcl|j|}|j|}|j|}|Sr)rrr@rs rPrbz#BertPredictionHeadTransform.forwards4 =1 --m< }5 rQrrls@rPrrs$UU\\ellrQrc$eZdZfdZdZxZS)BertLMPredictionHeadct|t||_t j |j |jd|_t jtj|j|_ y)NT)bias) r5r6r transformrrr9r8decoder ParameterrFrIrrLs rPr6zBertLMPredictionHead.__init__s[ 4V<yy!3!3V5F5FTR LLV->->!?@ rQcJ|j|}|j|}|Sr)rr rs rPrbzBertLMPredictionHead.forwards$}5  ]3 rQrcrdrer6rbrkrls@rPrrsArQrcVeZdZfdZdej dej fdZxZS)BertOnlyMLMHeadcBt|t||_yr)r5r6r predictionsrLs rPr6zBertOnlyMLMHead.__init__s /7rQsequence_outputrUc(|j|}|Sr)r)rMrprediction_scoress rPrbzBertOnlyMLMHead.forwards ,,_=  rQrrls@rPrrs#8!u||! !rQrc$eZdZfdZdZxZS)BertOnlyNSPHeadclt|tj|jd|_yNrv)r5r6rrr9seq_relationshiprLs rPr6zBertOnlyNSPHead.__init__s'  " &*<*rr?rr)rMrRrqr3r/rSrrrrrsrTrIencoder_outputsrrs rPrbzBertModel.forwardps|  -t"; <YZ Z ;; ! !%.%:  @U@UII 0)4 8V8V$L $DlZ^ZeZeFfg!5  FUE`!?!?!Afg??%)'#9 + 261M1M)#9-"7+ 2N2 ..'$,,   )"7#9+%    *;;8< 8O O4UY ;-'+;;  rQc|jjrt|j|||}nt|j||}|t|j|||}||fS)N)rNrSrqr)rNrSrq)rNrSrqr)rNrrr)rMrqrrIrrs rPrMz!BertModel._create_attention_maskssu ;; ! !/{{.- / N7{{.-N " -%>{{.5&; & "555rQ)T) NNNNNNNNN)rcrdre_no_split_modulesr6rErGr%r&r"rFrjr rrr!rrrbrMrkrls@rPr;r;Js.*;7"/0 *..2.2,0-1596:(,!%? <<$&?  t+?  t+ ? llT) ? ||d* ?  %||d2? !& t 3? ? $;? +,?  u|| K K?  ? B6rQr;z Bert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. c`eZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde e de e jezfdZxZS)BertForPreTraining&bert.embeddings.word_embeddings.weightcls.predictions.biaszcls.predictions.decoder.weightzcls.predictions.decoder.biasct||t||_t ||_|j yr)r5r6r;r!rclsr@rLs rPr6zBertForPreTraining.__init__s4  f% '/ rQcB|jjjSrrWrr rDs rPget_output_embeddingsz(BertForPreTraining.get_output_embeddingsxx##+++rQc||jj_|j|jj_yrrWrr rrMnew_embeddingss rPset_output_embeddingsz(BertForPreTraining.set_output_embeddings,'5$$2$7$7!rQNrRrqr3r/rSlabelsnext_sentence_labelrsrUc |j|f||||dd|} | dd\} } |j| | \} } d}|u|st}|| jd|jj |jd}|| jdd|jd}||z}t || | | j| jS)am labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForPreTraining.from_pretrained("google-bert/bert-base-uncased") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` Trqr3r/rS return_dictNrvr1)r5r6r7rr") r!rWrrrNr8r4rr")rMrRrqr3r/rSrbrcrsoutputsrrrr total_lossloss_fctmasked_lm_lossnext_sentence_losss rPrbzBertForPreTraining.forwardsR$))  ))%'   *1!&48HH_m4\11  "5"A')H%&7&<& ,8*..2.2,0-1&*37A <<$&A  t+A  t+ A llT) A ||d* A  t#A #\\D0A +,A  u|| 7 7A A rQrRzP Bert Model with a `language modeling` head on top for CLM fine-tuning. ceZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde dzde dzde e jzdeedee jezfdZxZS)BertLMHeadModelrSrTrUct|||jstj dt |d|_t||_|jy)NzLIf you want to use `BertLMHeadModel` as a standalone, add `is_decoder=True.`FrA r5r6rloggerwarningr;r!rrWr@rLs rPr6zBertLMHeadModel.__init__BsL    NNi jf> "6* rQcB|jjjSrrYrDs rPrZz%BertLMHeadModel.get_output_embeddingsNr[rQc||jj_|j|jj_yrr]r^s rPr`z%BertLMHeadModel.set_output_embeddingsQrarQNrRrqr3r/rSrrrbrrlogits_to_keeprsrUc |d} |j|f||||||| | dd | } | j}t| tr t | dn| }|j |dd|ddf}d}|*|j d|||jjd| }t||| j| j| j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NFT) rqr3r/rSrrrrrf)logitsrbr8)r5ryrrr"r#r9)r!rrrislicerW loss_functionrNr8rrrr"r#)rMrRrqr3r/rSrrrbrrrwrsrgr slice_indicesryr5s rPrbzBertLMHeadModel.forwardUs.  I@I  A ))%'"7#9+ A  A  11 8B>SV8W~ot4]k -=!(;<=  %4%%pVFt{{OeOepiopD0#33!//))$55   rQ) NNNNNNNNNNr)rcrdrermr6rZr`r$r"rFrjr rrirr!rrrbrkrls@rProro7sP+S(> ,8*..2.2,0-1596:&*(,!%-.6 <<$&6  t+6  t+ 6 llT) 6 ||d* 6  %||d26 !& t 36  t#6 6 $;6 ell*6 +,6  u|| @ @6 6 rQroceZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde e de e jezfdZxZS)BertForMaskedLMrSrTrUct|||jrtj dt |d|_t||_|jy)NzkIf you want to use `BertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrqrrrLs rPr6zBertForMaskedLM.__init__sR     NN1  f> "6* rQcB|jjjSrrYrDs rPrZz%BertForMaskedLM.get_output_embeddingsr[rQc||jj_|j|jj_yrr]r^s rPr`z%BertForMaskedLM.set_output_embeddingsrarQNrRrqr3r/rSrrrbrsrUc >|j|f||||||dd| } | d} |j| } d} |Ft}|| jd|jj |jd} t | | | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rqr3r/rSrrrfrNr1r5ryrr") r!rWrrrNr8rrr")rMrRrqr3r/rSrrrbrsrgrrrjris rPrbzBertForMaskedLM.forwards($))   ))%'"7#9    "!* HH_5  ')H%&7&<& ,8*..2.2,0-1596:&*+ <<$&+  t++  t+ + llT) + ||d* +  %||d2+ !& t 3+  t#+ +,+  u|| ~ -+ + rQr~zT Bert Model with a `next sentence prediction (classification)` head on top. c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForNextSentencePredictionct||t||_t ||_|j yr)r5r6r;r!rrWr@rLs rPr6z&BertForNextSentencePrediction.__init__s4  f% "6* rQNrRrqr3r/rSrbrsrUc |j|f||||dd|}|d} |j| } d} |2t} | | jdd|jd} t | | |j |j S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForNextSentencePrediction.from_pretrained("google-bert/bert-base-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` Trer'Nr1rvr)r!rWrrrrr") rMrRrqr3r/rSrbrsrgrseq_relationship_scoresrkris rPrbz%BertForNextSentencePrediction.forwardsN$))  ))%'     "&((="9!  ')H!)*A*F*Fr1*Mv{{[]!_ *#*!//))   rQNNNNNN)rcrdrer6r$r"rFrjrr!rrrbrkrls@rPrrs *..2.2,0-1&*= <<$&=  t+=  t+ = llT) = ||d* =  t#= +,=  u|| : := = rQrz Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForSequenceClassificationcnt|||j|_||_t ||_|j |j n |j}tj||_ tj|j|j|_ |jyr)r5r6 num_labelsrNr;r!classifier_dropoutrCrrBrDrr9 classifierr@rMrNrrOs rPr6z&BertForSequenceClassification.__init__5s   ++ f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rQNrRrqr3r/rSrbrsrUc |j|f||||dd|}|d} |j| } |j| } d} ||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt} |j dk(r&| | j|j} n| | |} n|jjdk(r=t} | | jd|j |jd} n,|jjdk(rt} | | |} t| | |j |j" S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trer'N regressionsingle_label_classificationmulti_label_classificationr1r)r!rDrrN problem_typerr4rFrKrirsqueezerrrrrr") rMrRrqr3r/rSrbrsrgrryr5ris rPrbz%BertForSequenceClassification.forwardDs$$))  ))%'      ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   rQr)rcrdrer6r$r"rFrjrr!rrrbrkrls@rPrr.s *..2.2,0-1&*; <<$&;  t+;  t+ ; llT) ; ||d* ;  t#; +,;  u|| 7 7; ; rQrc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForMultipleChoicec*t||t||_|j |jn |j }t j||_t j|jd|_ |jy)Nr') r5r6r;r!rrCrrBrDrr9rr@rs rPr6zBertForMultipleChoice.__init__su  f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$6: rQNrRrqr3r/rSrbrsrUc ||jdn|jd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|j|f||||dd|} | d} |j | } |j | } | jd|} d} |t }|| |} t| | | j| jS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr'r1Trer) r[rrJr!rDrrrrr")rMrRrqr3r/rSrbrs num_choicesrgrryreshaped_logitsr5ris rPrbzBertForMultipleChoice.forwardsT-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  $))  ))%'      ]3 / ++b+6  ')HOV4D("!//))   rQr)rcrdrer6r$r"rFrjrr!rrrbrkrls@rPrrs *..2.2,0-1&*N <<$&N  t+N  t+ N llT) N ||d* N  t#N +,N  u|| 8 8N N rQrc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyNFrq)r5r6rr;r!rrCrrBrDrr9rr@rs rPr6z#BertForTokenClassification.__init__s   ++f> )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rQNrRrqr3r/rSrbrsrUc H|j|f||||dd|}|d} |j| } |j| } d} | ))F$6$68I8IJ rQNrRrqr3r/rSstart_positions end_positionsrsrUc |j|f||||dd|} | d} |j| } | jdd\} } | jdj } | jdj } d}||t |j dkDr|jd}t |j dkDr|jd}| j d}|jd|}|jd|}t|}|| |}|| |}||zdz }t|| | | j| j S) NTrerr'r1rw) ignore_indexrv)r5 start_logits end_logitsrr") r!rsplitrr~lenrJclamprrrr")rMrRrqr3r/rSrrrsrgrryrrrh ignored_indexri start_lossend_losss rPrbz BertForQuestionAnswering.forward.s$))  ))%'   "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   rQrl)rcrdrer6r$r"rFrjrr!rrrbrkrls@rPrr"s*..2.2,0-1/3-13 <<$&3  t+3  t+ 3 llT) 3 ||d* 3 ,3 ||d*3 +,3  u|| ; ;3 3 rQr) r~rrrRrrrrror;r )Nr)Zrfcollections.abcr dataclassesrrFrtorch.nnrrrr r& activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsr r!r"r# utils.genericr$r%utils.output_capturingr&configuration_bertr( get_loggerrcrsModuler*rjfloatrrrrrrrrrrrrrrrr r4r;rRror~rrrrr__all__r9rQrPrs$! AA&!CC)J9   G&6MMI5*   H %;RYY;H!% II% <<% % << % LL4' % T\ %%'(%8@) @)FI)I)X RYY .BII.: ryy   >*>B ")) @  "))" 299  !bii!&bii& 9299 9////2  7{7  7&  |6#|6 |6~ Y ,Y  Y x Q )?Q  Q hI )I I X I $7I  I X M $7M  M `^ /^ ^ B8 !48 8 v@ 2@ @ F rQ