^jddlmZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZm Z m!Z!m"Z"m#Z#dd l$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2ddl3m4Z4ddl5m6Z6e/jne8Z9GddejtZ; dZdejtdejxdejxdejxdejxdzde=dzde=de(e-fdZ>Gd d!ejtZ?Gd"d#ejtZ@Gd$d%ejtZAGd&d'ejtZBGd(d)ejtZCGd*d+ejtZDGd,d-eZEGd.d/ejtZFGd0d1ejtZGGd2d3ejtZHGd4d5ejtZIe.Gd6d7e&ZJe.d89Gd:d;eJZKe.d<9eGd=d>e,ZLGd?d@ejtZMe.dA9GdBdCeJZNGdDdEejtZOe.dF9GdGdHeJeZPe.GdIdJeJZQGdKdLejtZRe.dM9GdNdOeJZSe.dP9GdQdReJZTe.GdSdTeJZUe.GdUdVeJZVe.GdWdXeJZWgdYZXy)[)Callable) dataclassN)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) ErnieConfigceZdZdZfdZ d dej dzdej dzdej dzdej dzdejdzd ed ejfd Z xZ S) ErnieEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j#dt%j*|j,j/t$j0d|j2|_|j2r0tj|j4|j |_yy) N) padding_idxeps position_idsr&F) persistenttoken_type_ids)dtype)super__init__nn Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandzerosr.sizelong use_task_idtask_type_vocab_sizetask_type_embeddingsselfconfig __class__s s/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/ernie/modeling_ernie.pyr5zErnieEmbeddings.__init__<sX !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  "--   (* V5P5PRXRdRd(eD % N input_idsr2 task_type_idsr. inputs_embedspast_key_values_lengthreturnc|||j}n|jdd}|\}} ||jdd|| |zf}|t|drT|jj |j dd} t j| d|} | j || }n:t j|t j|jj}||j|}|j|} |j| j}|| z} |j|} | | z} |jrR|:t j|t j|jj}|j!|}| |z } |j#| } |j%| } | S)Nr0r2rr&)dimindex)r3device)rJr.hasattrr2rHshaperFgatherrIrKr]r;r?tor=rLrNr@rD)rPrUr2rVr.rWrX input_shape batch_size seq_lengthbuffered_token_type_idsr? embeddingsr=rNs rSforwardzErnieEmbeddings.forwardPs  #..*K',,.s3K!, J  ,,Q0FVlIl0l-lmL  !t-.*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? J!W!&[ SWSdSdSkSk!l   00;M $ : :> J&(()>)E)EF "%:: "66|D"55    $ % KuzzRVRcRcRjRj k #'#<#<]#K . .J^^J/ \\*- rT)NNNNNr) __name__ __module__ __qualname____doc__r5rF LongTensor FloatTensorintTensorrg __classcell__rRs@rSr)r)9sQf,.226150426&'3##d*3((4/3''$. 3 &&- 3 ((4/ 3!$3 3rTr)modulequerykeyvalueattention_maskscalingrDkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)Nr0rr[)ptrainingr&) rJrFmatmul transposer6 functionalsoftmaxrDr~ contiguous) rrrsrtrurvrwrDrx attn_weights attn_outputs rSeager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$rTc eZdZd fd Z d dej dej dzdedzdee de ej f dZ xZ S) ErnieSelfAttentionNc@t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_|j&|_||_||_yNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rz)r4r5r9num_attention_headsr^ ValueErrorrQrnattention_head_size all_head_sizerwr6LinearrsrtrurBattention_probs_dropout_probrD is_decoder is_causal layer_idxrPrQrrrRs rSr5zErnieSelfAttention.__init__sP     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9F|} t|tr |j} | j|| |j\}} tj|jjt } | |||| |f|j"sdn|j$j&|j(d|\} } | j*g|dj-} | | fS)Nr0r&r{rDrw)r_rrsviewrrtru isinstancer self_attention_cacheupdaterr get_interfacerQ_attn_implementationrr~rDr}rwreshaper)rPrrvrrxrb hidden_shape query_layer key_layer value_layercurrent_past_key_valuesattention_interfacerrs rSrgzErnieSelfAttention.forwards$))#2. CCbC$*B*BC 5djj/44lCMMaQRS 0DHH]+00,?II!QO 4djj/44lCMMaQRS  &&5 #/+>?*9*N*N'&=%C%CI{\`\j\j%k "I{(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((rTFN)NN rhrirjr5rFrormr rr tuplergrprqs@rSrrsg#648(, ')||')))D0') ') +, ') u||  ')rTrceZdZd fd Z d dej dej dzdej dzdedzdee de ej f d Z xZ S) ErnieCrossAttentionNct||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_||_||_yr)r4r5r9rr^rrQrnrrrwr6rrsrtrurBrrDrrrs rSr5zErnieCrossAttention.__init__sC     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9FB**4>>:(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((rTr)NNN) rhrirjr5rFrormr rr rrgrprqs@rSrrs#4;?376: 1)||1) %00471)))D0 1) -t3 1) +, 1) u|| 1)rTrcneZdZfdZdej dej dej fdZxZS)ErnieSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr,) r4r5r6rr9denser@rArBrCrDrOs rSr5zErnieSelfOutput.__init__2s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rTr input_tensorrYcr|j|}|j|}|j||z}|SNrrDr@rPrrs rSrgzErnieSelfOutput.forward87 =1  ]3 }|'CD rTrhrirjr5rFrorgrprqs@rSrr11> U\\RWR^R^rTrceZdZd fd Z d dej dej dzdej dzdej dzdedzdee d e ej fd Z xZ S) ErnieAttentionNct|||_|rtnt}|||||_t ||_y)Nrr)r4r5is_cross_attentionrrrProutput)rPrQrrrattention_classrRs rSr5zErnieAttention.__init__@s= "41C-I[#Fi9U %f- rTrrvrencoder_attention_maskrrxrYc |js|n|}|j|f|||d|\}}|j||}||fS)N)rrvr)rrPr) rPrrvrrrrxattention_outputrs rSrgzErnieAttention.forwardGsd04/F/FLb)2 * "7)+ *  * &, ;;'7G--rT)FNFNNNNrrqs@rSrr?s.48:>;?(, .||.))D0. %0047 . !& 1 1D 8 .  .+,. u|| .rTrcVeZdZfdZdej dej fdZxZS)ErnieIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r4r5r6rr9intermediate_sizerr hidden_actstrr intermediate_act_fnrOs rSr5zErnieIntermediate.__init__]s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rTrrYcJ|j|}|j|}|Sr)rrrPrs rSrgzErnieIntermediate.forwardes& =1 00? rTrrqs@rSrr\s#9U\\ellrTrcneZdZfdZdej dej dej fdZxZS) ErnieOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r4r5r6rrr9rr@rArBrCrDrOs rSr5zErnieOutput.__init__ls` YYv779K9KL f&8&8f>S>STzz&"<"<= rTrrrYcr|j|}|j|}|j||z}|Srrrs rSrgzErnieOutput.forwardrrrTrrqs@rSrrkrrTrceZdZd fd Z d dej dej dzdej dzdej dzdedzdee d ej fd Z d Z xZ S) ErnieLayerNct||j|_d|_t ||j ||_|j |_|j|_|jr.|j st|dt |d|d|_ t||_ t||_ y)Nr&rz> should be used as a decoder model if cross attention is addedFT)rrr)r4r5chunk_size_feed_forward seq_len_dimrr attentionadd_cross_attentionrcrossattentionr intermediaterr)rPrQrrRs rSr5zErnieLayer.__init__zs '-'E'E$'&:K:KW`a ++#)#=#=  # #?? D6)g!hii"0##' #D  .f5!&) rTrrvrrrrxrYc |j||fd|i|\}}|} |jr:|8t|dstd|d|j|d||fd|i|\} }| } t |j |j|j| } | S)Nrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) rrr^rrrfeed_forward_chunkrr) rPrrvrrrrxself_attention_output_rcross_attention_output layer_outputs rSrgzErnieLayer.forwards$24>>  $ ,$  $  q 1 ??4@4!12 =dVDDD )<(;(;%%& ) !0 )  ) % "A 6 0  # #T%A%A4CSCSUe rTcL|j|}|j||}|Sr)rr)rPrintermediate_outputrs rSrzErnieLayer.feed_forward_chunks,"//0@A{{#68HI rTrr) rhrirjr5rFrormr rr rgrrprqs@rSrrys*,48:>;?(, %||%))D0% %0047 % !& 1 1D 8 %  %+,% %NrTrcVeZdZfdZdej dej fdZxZS) ErniePoolerct|tj|j|j|_tj |_yr)r4r5r6rr9rTanh activationrOs rSr5zErniePooler.__init__s9 YYv1163E3EF '')rTrrYc\|dddf}|j|}|j|}|S)Nr)rr)rPrfirst_token_tensor pooled_outputs rSrgzErniePooler.forwards6+1a40 #56  6 rTrrqs@rSrrs#$ U\\ellrTrcVeZdZfdZdej dej fdZxZS)ErniePredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r4r5r6rr9rrrrr transform_act_fnr@rArOs rSr5z%ErniePredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrTrrYcl|j|}|j|}|j|}|Sr)rrr@rs rSrgz$ErniePredictionHeadTransform.forwards4 =1 --m< }5 rTrrqs@rSrrs$UU\\ellrTrc$eZdZfdZdZxZS)ErnieLMPredictionHeadct|t||_t j |j |jd|_t jtj|j|_ y)NT)bias) r4r5r transformr6rr9r8decoder ParameterrFrIrrOs rSr5zErnieLMPredictionHead.__init__s[ 5f=yy!3!3V5F5FTR LLV->->!?@ rTcJ|j|}|j|}|Sr)rrrs rSrgzErnieLMPredictionHead.forwards$}5  ]3 rTrhrirjr5rgrprqs@rSrrsArTrceZdZfdZ d dej dej dzdej dzdej dzdedzdedzd e e d e ej e zfd Z xZS) ErnieEncoderc t|||_tjt |j Dcgc]}t||c}|_ycc}w)N)r) r4r5rQr6 ModuleListrangenum_hidden_layersrlayer)rPrQirRs rSr5zErnieEncoder.__init__sF  ]]USYSkSkMl#mJv$C#mn #msANrrvrrr use_cacherxrYc t|jD]\}} | |||f||d|}t||r|SdS)N)rr)last_hidden_stater) enumerater r) rPrrvrrrr rxr  layer_modules rSrgzErnieEncoder.forwardsn )4 OA|(%(> /   M 9+/8O  >B  rT)NNNNN)rhrirjr5rFrormr boolrr rrrgrprqs@rSrrso48:>;?(,!% || ))D0  %0047  !& 1 1D 8    $; +,  u|| H H rTrcneZdZeZdZdZdZdZdZ dZ e e e dZej fdZxZS)ErniePreTrainedModelernieT)r attentionscross_attentionsct||t|tr t j |j yt|tryt j|jtj|jjdjdt j |jyy)zInitialize the weightsr0r/N)r4 _init_weightsrrinitzeros_rr)copy_r.rFrGr_rHr2)rPrrrRs rSrz"ErniePreTrainedModel._init_weightss f% f3 4 KK $  0 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .1rT)rhrirjr' config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrr_can_record_outputsrFno_gradrrprqs@rSrr sXL&*#N"&#(/ U]]_//rTra  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introceZdZdgZdfd ZdZdZeee dde jdzde jdzde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde dzde dzdeedee jezfdZdZxZS) ErnieModelrct||||_d|_t ||_t ||_|r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r4r5rQgradient_checkpointingr)rfrencoderrpooler post_init)rPrQadd_pooling_layerrRs rSr5zErnieModel.__init__3sU   &+#)&1#F+ ->k&)D  rTc.|jjSrrfr;rPs rSget_input_embeddingszErnieModel.get_input_embeddingsDs...rTc&||j_yrr/)rPrus rSset_input_embeddingszErnieModel.set_input_embeddingsGs*/'rTNrUrvr2rVr.rWrrrr rxrYc |du|duz r td|jjr| | n|jj} nd} | rd| b||jjr4t t |jt |jnt |j} | | jnd} |j|||||| } |j||| || \}}|j| f|||| | |d| }|d}|j|j|nd}t|||j S)  task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] Nz:You must specify exactly one of input_ids or inputs_embedsF)rQr)rUr.r2rVrWrX)rvrembedding_outputrr)rvrrrr r.)r pooler_outputr)rrQrr is_encoder_decoderr r get_seq_lengthrf_create_attention_masksr*r+rr)rPrUrvr2rVr.rWrrrr rxrXr6encoder_outputssequence_outputrs rSrgzErnieModel.forwardJs~0  -t"; <YZ Z ;; ! !%.%:  @U@UII 0)4 8V8V$L $DlZ^ZeZeFfg!5  FUE`!?!?!Afg??%)''#9+ 261M1M)#9-"7+ 2N2 ..'$,,   )"7#9+%    *!,8< 8O O4UY ;-'+;;  rTc|jjrt|j|||}nt|j||}|t|j|||}||fS)N)rQrWrvr)rQrWrv)rQrWrvr)rQrrr)rPrvrr6rrs rSr:z"ErnieModel._create_attention_maskssu ;; ! !/{{.- / N7{{.-N " -%>{{.5&; & "555rT)T) NNNNNNNNNN)rhrirj_no_split_modulesr5r1r3r$r%r!rFror rrr rrrgr:rprqs@rSr'r'$sO&"/0 *..2.2-1,0-1596:(,!%H <<$&H  t+H  t+ H ||d* H llT) H ||d*H  %||d2H !& t 3H H $;H +,H  u|| K KH  H T6rTr'z1 Output type of [`ErnieForPreTraining`]. ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e ej dzed<dZ e ej dzed<y)ErnieForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsrr) rhrirjrkrArFrm__annotations__rBrCrrrrTrSr@r@s~ &*D%  d ")26u((4/68<U..5<59M5**+d2926Je''(4/6rTr@c$eZdZfdZdZxZS)ErniePreTrainingHeadsct|t||_t j |j d|_yNr{)r4r5r predictionsr6rr9seq_relationshiprOs rSr5zErniePreTrainingHeads.__init__s4 08 " &*<*>> from transformers import AutoTokenizer, ErnieForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` Trvr2rVr.rW return_dictNr{r0)rArBrCrr) rrWrrrQr8r@rr)rPrUrvr2rVr.rWrbrcrxoutputsr<rrMrN total_lossloss_fctmasked_lm_lossnext_sentence_losss rSrgzErnieForPreTraining.forwards^$**   ))'%'    *1!&48HH_m4\11  "5"A')H%&7&<&SV8W~ot4]k -=!(;<=  %4%%pVFt{{OeOepiopD0#33!//))$55   rT) NNNNNNNNNNNr)rhrirjrmr5rZr`r#r!rFrolistrrnrr rrrgrprqs@rSrsrsPsp+T(> ,8*..2.2-1,0-1596:&*59!%-.= <<$&=  t+=  t+ = ||d* = llT) = ||d*=  %||d2= !& t 3=  t#= ell+d2= $;= ell*= +,=  u|| @ @= = rTrsceZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde jdzde e de e jezfdZxZS)ErnieForMaskedLMrQrRrSct|||jrtj dt |d|_t||_|jy)NzlIf you want to use `ErnieForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrurvrOs rSr5zErnieForMaskedLM.__init__sR     NN1   %@ #F+ rTcB|jjjSrrYr0s rSrZz&ErnieForMaskedLM.get_output_embeddingsr[rTc||jj_|j|jj_yrr]r^s rSr`z&ErnieForMaskedLM.set_output_embeddingsrarTNrUrvr2rVr.rWrrrbrxrYc @|j|f|||||||dd| } | d} |j| } d}| Ft}|| jd|jj | jd}t || | j| jS)as task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rvr2rVr.rWrrrfrNr0rAr}rr) rrWrrrQr8rrr)rPrUrvr2rVr.rWrrrbrxrgr<rMrjris rSrgzErnieForMaskedLM.forwards4$**   ))'%'"7#9    "!* HH_5  ')H%&7&<&>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` Trer&Nr0r{r)rrWrrrrr)rPrUrvr2rVr.rWrbrxrgrseq_relationship_scoresrkris rSrgz&ErnieForNextSentencePrediction.forwardsZ$**   ))'%'      "&((="9!  ')H!)*A*F*Fr1*Mv{{[]!_ *#*!//))   rTNNNNNNN)rhrirjr5r#r!rFrorr rrrgrprqs@rSrrs *..2.2-1,0-1&*D <<$&D  t+D  t+ D ||d* D llT) D ||d*D  t#D +,D  u|| : :D D rTrz Ernie Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cJeZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)ErnieForSequenceClassificationcnt|||j|_||_t ||_|j |j n |j}tj||_ tj|j|j|_ |jyr)r4r5 num_labelsrQr'rclassifier_dropoutrCr6rBrDrr9 classifierr,rPrQrrRs rSr5z'ErnieForSequenceClassification.__init__ms   ++ ' )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rTNrUrvr2rVr.rWrbrxrYc |j|f|||||dd|} | d} |j| } |j| } d} ||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt} |j dk(r&| | j|j} n| | |} n|jjdk(r=t} | | jd|j |jd} n,|jjdk(rt} | | |} t| | | j | j" S) a^ task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trer&N regressionsingle_label_classificationmulti_label_classificationr0r)rrDrrQ problem_typerr3rFrKrnrsqueezerrrrrr)rPrUrvr2rVr.rWrbrxrgrr}rAris rSrgz&ErnieForSequenceClassification.forward|s0$**   ))'%'       ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   rTr)rhrirjr5r#r!rFrorr rrrgrprqs@rSrrfs *..2.2-1,0-1&*B <<$&B  t+B  t+ B ||d* B llT) B ||d*B  t#B +,B  u|| 7 7B B rTrcJeZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)ErnieForMultipleChoicec*t||t||_|j |jn |j }t j||_t j|jd|_ |jy)Nr&) r4r5r'rrrCr6rBrDrr9rr,rs rSr5zErnieForMultipleChoice.__init__su  ' )/)B)B)NF % %TZTnTn zz"45 ))F$6$6: rTNrUrvr2rVr.rWrbrxrYc ||jdn|jd} |!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|j|f|||||dd|} | d} |j | } |j | } | jd| } d}|t }|| |}t|| | j| jS)a9 input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) task_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr&r0Trer) r_rrJrrDrrrrr)rPrUrvr2rVr.rWrbrx num_choicesrgrr}reshaped_logitsrAris rSrgzErnieForMultipleChoice.forwards`-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  $**   ))'%'       ]3 / ++b+6  ')HOV4D("!//))   rTr)rhrirjr5r#r!rFrorr rrrgrprqs@rSrrs *..2.2-1,0-1&*U <<$&U  t+U  t+ U ||d* U llT) U ||d*U  t#U +,U  u|| 8 8U U rTrcJeZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ee d e eje zfd Z xZ S)ErnieForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyNFru)r4r5rr'rrrCr6rBrDrr9rr,rs rSr5z$ErnieForTokenClassification.__init__.s   ++%@ )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rTNrUrvr2rVr.rWrbrxrYc J|j|f|||||dd|} | d} |j| } |j| } d} |rs*%! AA&!CC)J9   G&6MMI5,   H %JbiiJf!% II% <<% % << % LL4' % T\ %%'(%8@)@)FI)"))I)X bii .RYY.:   ")) >+>B ")) 299" BII   299 @/?//2  E6%E6 E6P  7 7  7& 9BII 9 ` .`  ` F!ryy! X +_X  X vP +P P f&ryy& P %9P  P f T %9T  T ne 1e e P? "6? ? DI 4I I X rT