^jdZddlZddlmZddlmZmZmZddlmZ ddl m Z m Z m Z ddlmZmZmZmZmZmZmZmZddlmZdd lmZdd lmZmZmZdd lm Z m!Z!dd l"m#Z#d dl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5ddl6m7Z7ejpe9Z:Gdde&Z;Gdde5Z<Gdde%Z=Gdde0Z>Gdde4Z?Gdde2Z@Gdde'ZAeGdd eZBGd!d"e3ZCGd#d$e,ZDGd%d&e+ZEGd'd(e1ZFGd)d*e(ZGGd+d,e*ZHGd-d.e.ZIGd/d0e)ZJGd1d2e/ZKGd3d4e-ZLgd5ZMy)6zPyTorch ERNIE model.N)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)Cache DynamicCacheEncoderDecoderCache),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)BertCrossAttentionBertEmbeddings BertEncoderBertForMaskedLMBertForMultipleChoiceBertForNextSentencePredictionBertForPreTrainingBertForPreTrainingOutputBertForQuestionAnsweringBertForSequenceClassificationBertForTokenClassification BertLayerBertLMHeadModelBertLMPredictionHead BertModel BertPoolerBertSelfAttention) ErnieConfigceZdZdZfdZ d dej dzdej dzdej dzdej dzdejdzd ed ejfd Z xZ S) ErnieEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|||j|_|jr0tj|j |j |_yy)N)super__init__ use_task_idnn Embeddingtask_type_vocab_size hidden_sizetask_type_embeddings)selfconfig __class__s r/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/ernie/modular_ernie.pyr3zErnieEmbeddings.__init__AsL  !--   (* V5P5PRXRdRd(eD % N input_idstoken_type_ids task_type_ids position_ids inputs_embedspast_key_values_lengthreturnc|||j}n|jdd}|\}} ||jdd|| |zf}|t|drT|jj |j dd} t j| d|} | j || }n:t j|t j|jj}||j|}|j|} |j| j}|| z} |j|} | | z} |jrR|:t j|t j|jj}|j!|}| |z } |j#| } |j%| } | S)Nr@rr-)dimindex)dtypedevice)sizerBhasattrr@expandshapetorchgatherzeroslongrKword_embeddingstoken_type_embeddingstoposition_embeddingsr4r9 LayerNormdropout)r:r?r@rArBrCrD input_shape batch_size seq_lengthbuffered_token_type_idsrU embeddingsrWr9s r=forwardzErnieEmbeddings.forwardHs  #..*K',,.s3K!, J  ,,Q0FVlIl0l-lmL  !t-.*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? J!W!&[ SWSdSdSkSk!l   00;M $ : :> J&(()>)E)EF "%:: "66|D"55    $ % KuzzRVRcRcRjRj k #'#<#<]#K . .J^^J/ \\*- r>)NNNNNr) __name__ __module__ __qualname____doc__r3rP LongTensor FloatTensorintTensorr_ __classcell__r<s@r=r0r0>sQf.226150426&'3##d*3((4/3''$. 3 &&- 3 ((4/ 3!$3 3r>r0c eZdZy)ErnieSelfAttentionNr`rarbr>r=rkrk~r>rkc eZdZy)ErnieCrossAttentionNrlrmr>r=rprprnr>rpc eZdZy) ErnieLayerNrlrmr>r=rrrrrnr>rrc eZdZy) ErniePoolerNrlrmr>r=rtrtrnr>rtc eZdZy)ErnieLMPredictionHeadNrlrmr>r=rvrvrnr>rvc eZdZy) ErnieEncoderNrlrmr>r=rxrxrnr>rxcneZdZeZdZdZdZdZdZ dZ e e e dZej fdZxZS)ErniePreTrainedModelernieT) hidden_states attentionscross_attentionsct||t|tr t j |j yt|tryt j|jtj|jjdjdt j |jyy)zInitialize the weightsrG)r-rGN)r2 _init_weights isinstancervinitzeros_biasr0copy_rBrParangerOrNr@)r:moduler<s r=rz"ErniePreTrainedModel._init_weightss f% f3 4 KK $  0 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .1r>)r`rarbr. config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrrkrp_can_record_outputsrPno_gradrrhris@r=rzrzsXL&*#N"&#(/ U]]_//r>rzceZdZdgZdfd Zeee ddejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd e dzd e dzde e deejezfdZxZS) ErnieModelrrct|||||_d|_t ||_t ||_|r t|nd|_ |jy)NF) r2r3r;gradient_checkpointingr0r^rxencoderrtpooler post_init)r:r;add_pooling_layerr<s r=r3zErnieModel.__init__sU v& &+#)&1#F+ ->k&)D  r>Nr?attention_maskr@rArBrCencoder_hidden_statesencoder_attention_maskpast_key_values use_cachekwargsrEc |du|duz r td|jjr| | n|jj} nd} | rd| b||jjr4t t |jt |jnt |j} | | jnd} |j|||||| } |j||| || \}}|j| f|||| | |d| }|d}|j|j|nd}t|||j S)  task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] Nz:You must specify exactly one of input_ids or inputs_embedsF)r;r)r?rBr@rArCrD)rrembedding_outputrr)rrrrrrB)last_hidden_state pooler_outputr) ValueErrorr; is_decoderris_encoder_decoderr r get_seq_lengthr^_create_attention_masksrrr r)r:r?rr@rArBrCrrrrrrDrencoder_outputssequence_output pooled_outputs r=r_zErnieModel.forwards~0  -t"; <YZ Z ;; ! !%.%:  @U@UII 0)4 8V8V$L $DlZ^ZeZeFfg!5  FUE`!?!?!Afg??%)''#9+ 261M1M)#9-"7+ 2N2 ..'$,,   )"7#9+%    *!,8< 8O O4UY ;-'+;;  r>)T) NNNNNNNNNN)r`rarb_no_split_modulesr3rrrrPrgrboolrrtupler r_rhris@r=rrs=%  *..2.2-1,0-1596:(,!%H <<$&H  t+H  t+ H ||d* H llT) H ||d*H  %||d2H !& t 3H H $;H +,H  u|| K KH  H r>rc eZdZy)ErnieForPreTrainingOutputNrlrmr>r=rrrnr>rcbeZdZdddZee ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd ee de eje zfdZ y)ErnieForPreTrainingcls.predictions.bias'ernie.embeddings.word_embeddings.weightzcls.predictions.decoder.biaszcls.predictions.decoder.weightNr?rr@rArBrClabelsnext_sentence_labelrrEc |j|f|||||dd| } | dd\} } |j| | \} }d}|u|st}|| jd|jj |jd}||jdd|jd}||z}t || || j| jS)a: task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForPreTraining.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` Trr@rArBrC return_dictNrrG)lossprediction_logitsseq_relationship_logitsr|r}) r{clsrviewr; vocab_sizerr|r})r:r?rr@rArBrCrrroutputsrrprediction_scoresseq_relationship_score total_lossloss_fctmasked_lm_lossnext_sentence_losss r=r_zErnieForPreTraining.forwards^$**   ))'%'    *1!&48HH_m4\11  "5"A')H%&7&<&NNNNNNNN) r`rarb_tied_weights_keysrrrPrgrrrrr_rmr>r=rrs(>*S *..2.2-1,0-1&*37H <<$&H  t+H  t+ H ||d* H llT) H ||d*H  t#H #\\D0H +,H  u|| 8 8H H r>rc eZdZee ddej dzdej dzdej dzdej dzdej dzdej dzdej dzd ej dzd ej dzd eej dzd edzd e ej zde e de ej e zfdZy)ErnieForCausalLMNr?rr@rArBrCrrrrrlogits_to_keeprrEc  | d} |j|f|||||||| | dd | }|j}t| tr t | dn| }|j |dd|ddf}d}| *|j d|| |jjd| }t|||j|j|j|jS)a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NFT) rr@rArBrCrrrrr)logitsrr)rrrr|r}r~rm)r{rrrfslicer loss_functionr;rr rr|r}r~)r:r?rr@rArBrCrrrrrrrrr| slice_indicesrrs r=r_zErnieForCausalLM.forwardfs:  I@J  A ))'%'"7#9+ A  A  11 8B>SV8W~ot4]k -=!(;<=  %4%%pVFt{{OeOepiopD0#33!//))$55   r>) NNNNNNNNNNNr)r`rarbrrrPrglistrrfrrrr r_rmr>r=rresM*..2.2-1,0-1596:&*59!%-.= <<$&=  t+=  t+ = ||d* = llT) = ||d*=  %||d2= !& t 3=  t#= ell+d2= $;= ell*= +,=  u|| @ @= = r>rceZdZdddZee ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd ejdzdee de eje zfdZ y)ErnieForMaskedLMrrrNr?rr@rArBrCrrrrrEc @|j|f|||||||dd| } | d} |j| } d}| Ft}|| jd|jj | jd}t || | j| jS)as task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rr@rArBrCrrrrNrGrrr|r}) r{rrrr;rr r|r})r:r?rr@rArBrCrrrrrrrrrs r=r_zErnieForMaskedLM.forwards4$**   ))'%'"7#9    "!* HH_5  ')H%&7&<&) NNNNNNNNN) r`rarbrrrrPrgrrrr r_rmr>r=rrs(>*S *..2.2-1,0-1596:&*2 <<$&2  t+2  t+ 2 ||d* 2 llT) 2 ||d*2  %||d22 !& t 32  t#2 +,2  u|| ~ -2 2 r>rc8eZdZee d dej dzdej dzdej dzdej dzdej dzdej dzdej dzd eed e ej e zfd Z y) ErnieForNextSentencePredictionNr?rr@rArBrCrrrEc |j|f|||||dd|} | d} |j| } d} |2t} | | jdd|jd} t | | | j | j S)a task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, ErnieForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> model = ErnieForNextSentencePrediction.from_pretrained("nghuyong/ernie-1.0-base-zh") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` Trr-NrGrr)r{rrrrr|r})r:r?rr@rArBrCrrrrseq_relationship_scoresrrs r=r_z&ErnieForNextSentencePrediction.forwardsZ$**   ))'%'      "&((="9!  ')H!)*A*F*Fr1*Mv{{[]!_ *#*!//))   r>NNNNNNN) r`rarbrrrPrgrrrrr_rmr>r=rrs*..2.2-1,0-1&*D <<$&D  t+D  t+ D ||d* D llT) D ||d*D  t#D +,D  u|| : :D D r>rc8eZdZee d dej dzdej dzdej dzdej dzdej dzdej dzdej dzd eed e ej e zfd Z y) ErnieForSequenceClassificationNr?rr@rArBrCrrrEc |j|f|||||dd|} | d} |j| } |j| } d} ||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt} |j dk(r&| | j|j} n| | |} n|jjdk(r=t} | | jd|j |jd} n,|jjdk(rt} | | |} t| | | j | j" S) a^ task_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trr-N regressionsingle_label_classificationmulti_label_classificationrGr)r{rY classifierr; problem_type num_labelsrJrPrSrfrsqueezerrrrr|r})r:r?rr@rArBrCrrrrrrrs r=r_z&ErnieForSequenceClassification.forward0s0$**   ))'%'       ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   r>r) r`rarbrrrPrgrrrrr_rmr>r=rr/s*..2.2-1,0-1&*B <<$&B  t+B  t+ B ||d* B llT) B ||d*B  t#B +,B  u|| 7 7B B r>rc8eZdZee d dej dzdej dzdej dzdej dzdej dzdej dzdej dzd eed e ej e zfd Z y) ErnieForMultipleChoiceNr?rr@rArBrCrrrEc ||jdn|jd} |!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|j|f|||||dd|} | d} |j | } |j | } | jd| } d}|t }|| |}t|| | j| jS)a9 input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) task_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Task type embedding is a special embedding to represent the characteristic of different tasks, such as word-aware pre-training task, structure-aware pre-training task and semantic-aware pre-training task. We assign a `task_type_id` to each task and the `task_type_id` is in the range `[0, config.task_type_vocab_size-1] position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr-rGTrr) rOrrLr{rYrrrr|r})r:r?rr@rArBrCrr num_choicesrrrreshaped_logitsrrs r=r_zErnieForMultipleChoice.forwardxs`-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  $**   ))'%'       ]3 / ++b+6  ')HOV4D("!//))   r>r) r`rarbrrrPrgrrrrr_rmr>r=rrws*..2.2-1,0-1&*U <<$&U  t+U  t+ U ||d* U llT) U ||d*U  t#U +,U  u|| 8 8U U r>rc8eZdZee d dej dzdej dzdej dzdej dzdej dzdej dzdej dzd eed e ej e zfd Z y) ErnieForTokenClassificationNr?rr@rArBrCrrrEc J|j|f|||||dd|} | d} |j| } |j| } d} |r) r`rarbrrrPrgrrrrr_rmr>r=rrs*..2.2-1,0-1&*. <<$&.  t+.  t+ . ||d* . llT) . ||d*.  t#. +,.  u|| 4 4. . r>rcXeZdZee d dej dzdej dzdej dzdej dzdej dzdej dzdej dzd ej dzd eed e ej e zfd Z y)ErnieForQuestionAnsweringNr?rr@rArBrCstart_positions end_positionsrrEc |j|f|||||dd| } | d} |j| } | jdd\} }| jdj } |jdj }d}||t |j dkDr|jd}t |j dkDr|jd}| j d}|jd|}|jd|}t|}|| |}|||}||zd z }t|| || j| j S) rTrrr-rG)rHN) ignore_indexr)r start_logits end_logitsr|r}) r{ qa_outputssplitr contiguouslenrLclamprrr|r})r:r?rr@rArBrCrrrrrrrrr ignored_indexr start_lossend_losss r=r_z!ErnieForQuestionAnswering.forwards*$**   ))'%'    "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   r>r) r`rarbrrrPrgrrrrr_rmr>r=rrs*..2.2-1,0-1/3-1< <<$&<  t+<  t+ < ||d* < llT) < ||d*< ,< ||d*< +,<  u|| ; ;< < r>r) rrrrrrrrrrz)NrcrPtorch.nnr5rrrrr cache_utilsrr r modeling_outputsr r r rrrrrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrrutils.output_capturingrbert.modeling_bertrrrrr r!r"r#r$r%r&r'r(r)r*r+r,configuration_ernier. get_loggerr`loggerr0rkrprrrtrvrxrzrrrrrrrrrr__all__rmr>r=r s AA&CC   .&@@I5&-   H %=n=@ *  ,    *  0  ; /?//2[ [ | 8 P ,P f@ @ F: : zG %BG TE %BE PX 2X v1 "<1 h? 8? D r>