^jCdZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZmZmZdd lmZdd lmZdd lmZmZmZdd lmZddlmZddlmZmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(Gdde"Z)eGddeZ*Gdde Z+GddeZ,eGdde*Z-ed Gd!d"e*e Z.Gd#d$eZ/Gd%d&e$Z0Gd'd(e!Z1gd)Z2y)*zPyTorch PLBART model.N)nn)CrossEntropyLoss)initialization)Cache)GenerationMixin)BaseModelOutputSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)merge_with_config_defaults)capture_outputs)BartClassificationHead BartDecoder BartEncoderBartForCausalLMBartScaledWordEmbedding)'BigBirdPegasusForSequenceClassification)shift_tokens_right) PLBartConfigc eZdZy)PLBartScaledWordEmbeddingN__name__ __module__ __qualname__t/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/plbart/modular_plbart.pyrr/r$rcFeZdZUeed<dZdZddgZdZdZ dZ fdZ xZ S)PLBartPreTrainedModelconfigmodelTPLBartDecoderLayerPLBartEncoderLayerct||t|tr t j |j yyN)super _init_weights isinstancePLBartForConditionalGenerationinitzeros_final_logits_bias)selfmodule __class__s r%r0z#PLBartPreTrainedModel._init_weights=s2 f% f< = KK00 1 >r$) r r!r"r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attnr0 __classcell__r8s@r%r(r(3s< &*#-/CDN22r$r(c eZdZy) PLBartEncoderNrr#r$r%rCrCCr&r$rCc eZdZy) PLBartDecoderNrr#r$r%rErEGr&r$rEceZdZdddZdeffd ZdZdZee e dde jdzd e jdzd e jdzd e jdzd ee jdzd edzde jdzde jdzdedzdeedee jezfdZxZS) PLBartModelz shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightr)cJt|||j|j}}|jrt j |jnd}t||j|||_ t||_ t||_ |jy)Ng?) embed_scale)r/__init__ pad_token_id vocab_sizescale_embeddingmathsqrtd_modelrsharedrCencoderrEdecoder post_init)r6r) padding_idxrLrIr8s r%rJzPLBartModel.__init__Rsz  "("5"5v7H7HZ 393I3Idii/s / FNNKepq $V, $V,  r$c|jSr.)rQ)r6s r%get_input_embeddingsz PLBartModel.get_input_embeddings^s {{r$c~||_|j|j_|j|j_yr.)rQrR embed_tokensrS)r6values r%set_input_embeddingsz PLBartModel.set_input_embeddingsas) $(KK !$(KK !r$N input_idsattention_maskdecoder_input_idsdecoder_attention_maskencoder_outputspast_key_values inputs_embedsdecoder_inputs_embeds use_cachekwargsreturnc |"| t||jj}||jd |||d| }nGt |t s7t |dt |dkDr|dndt |dkDr|dnd}|jd |||d|||| d| } t| j| j| j| j| j|j|j|jS) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. N)r\r]rbrrr)last_hidden_state hidden_states attentions)r\r]encoder_hidden_statesencoder_attention_maskrarbrd)rhradecoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_staterkencoder_attentionsr#)rr)rKrRr1r lenrSr rhrarirjro) r6r\r]r^r_r`rarbrcrdredecoder_outputss r%forwardzPLBartModel.forwardfs3P  $)>)F 29dkk>V>V W   "/;t||0#-+0 0O O_=-"1!"4474H14Loa0RV14_1E1I?1-tO '$,,  '1"1!"4#1+/    "-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r$) NNNNNNNNN)r r!r"_tied_weights_keysrrJrWr[rrrtorch LongTensorTensorlist FloatTensorrboolr rtupler rtr@rAs@r%rGrGKsP(7'6 | 0  .226596::>(,26:>!%J ##d*J ((4/J !++d2 J !& t 3 J e//047 J J ((4/J  %0047J $;J +,J  u|| 1 1J  J r$rGzv The PLBART Model with a language modeling head. Can be used for code-to-text, text-to-code and code-to-code. ) custom_introceZdZdZdgZddiZdeffd Z dded edzd e d e jffd Z ded dfd Z eee ddej$dzdej$dzdej$dzdej&dzdeej*dzdedzdej*dzdej*dzdej&dzde dzdeed eej&ezfdZdej&fdZxZS)r2r*r5zlm_head.weightzmodel.shared.weightr)cxt||t||_|j dt j d|jjjftj|j|jjjd|_ |jy)Nr5rF)bias)r/rJrGr*register_bufferrvzerosrQnum_embeddingsrLinearrPlm_headrT)r6r)r8s r%rJz'PLBartForConditionalGeneration.__init__s~   (  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  r$Nnew_num_tokenspad_to_multiple_of mean_resizingrfczt||||}|j|jjd|S)Nr)r/resize_token_embeddings_resize_final_logits_biasweightshape)r6rrrnew_embeddingsr8s r%rz6PLBartForConditionalGeneration.resize_token_embeddingss?8I[]jk &&~'<'<'B'B1'EFr$c6|jjd}||kr|jddd|f}nStjd||z f|jj}tj |j|gd}|j d|y)Nr)device)dimr5)r5rrvrrcatr)r6rold_num_tokensnew_bias extra_biass r%rz8PLBartForConditionalGeneration._resize_final_logits_biass//55b9 ^ +--a..@AHa.)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;r$r\r]r^r_r`rarbrclabelsrdrec T| $|"| t| |jj}|j|f|||||||| d| } |j | j } | |j j| jz} d}| Ft}|| jd|jj| jd}t|| | j| j| j| j | j"| j$| j& S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Mask-filling: ```python >>> from transformers import AutoTokenizer, PLBartForConditionalGeneration >>> model = PLBartForConditionalGeneration.from_pretrained("uclanlp/plbart-base") >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> # en_XX is the language symbol id for English >>> TXT = " Is 0 the Fibonacci number ? en_XX" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt").input_ids >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['first', 'same', 'highest', 'result', 'number'] ``` N)r]r^r`r_rarbrcrdr) losslogitsrarmrnrorprkrq)rr)rKr*rrhr5torrviewrLr rarmrnrorprkrq)r6r\r]r^r_r`rarbrcrrdreoutputs lm_logitsmasked_lm_lossloss_fcts r%rtz&PLBartForConditionalGeneration.forwards2@   (-B-J$6vt{{?W?W$X!&0djj  ' )/+#9+'"7 '  ' LL!:!:;  6 6 9 9):J:J KK   ')H%innR9O9O&PRXR]R]^`RabN#33")"?"?&99$55&-&G&G")"?"?&99  r$cBt||jjSr.)rr)rK)r6rs r%%prepare_decoder_input_ids_from_labelszDPLBartForConditionalGeneration.prepare_decoder_input_ids_from_labels>s!&$++*B*BCCr$)NT) NNNNNNNNNN)r r!r"r:_keys_to_ignore_on_load_missingrurrJintr{r Embeddingrrrrrrvrwrxryrzrr rr|r rtrr@rAs@r%r2r2s  ':&;#/|ae!7:TzY] <<< .226596::>(,26:>&*!%_ ##d*_ ((4/_ !++d2 _ !& t 3 _ e//047 _ _ ((4/_  %0047_  t#_ $;_ +,_  u||  ._  _ BDELLDr$r2c eZdZy)PLBartClassificationHeadNrr#r$r%rrBr&r$rceZdZfdZxZS)PLBartForSequenceClassificationc :tjdi|y)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr#r/rt super_kwargsr8s r%rtz'PLBartForSequenceClassification.forwardGs4 ','r$)r r!r"rtr@rAs@r%rrFs ((r$rc2eZdZeefdZxZS)PLBartForCausalLMc :tjdi|y)aF labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PLBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> model = PLBartForCausalLM.from_pretrained("uclanlp/plbart-base") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nr#rrs r%rtzPLBartForCausalLM.forwardes2 ','r$)r r!r"rrrtr@rAs@r%rrds((r$r)rr2rrGr()3__doc__rNrvrtorch.nnrrr3 cache_utilsr generationrmodeling_outputsr r r modeling_utilsr processing_utilsr utilsrrr utils.genericrutils.output_capturingrbart.modeling_bartrrrrr(bigbird_pegasus.modeling_bigbird_pegasusrmbart.modeling_mbartrconfiguration_plbartrrr(rCrErGr2rrr__all__r#r$r%rs %& ) .&II75_5. 7  2O 2 2 K  K g 'g g T DD%:ODD  DDN 5 (&M(<((: r$