^jjdZddlZddlZddlmZddlZddlZddlmZddl m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZm Z m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/ddl0m1Z1m2Z2ddl3m4Z4e-jje6Z7dejpde9de9fdZ:GddejvZ< ddzd"e>d#e'e)fd$Z?Gd%d&ejzZ@Gd'd(eZAGd)d*eZBe*Gd+d,e%ZCGd-d.eCZDGd/d0eCZEe*Gd1d2eCZFe*d34Gd5d6eCeZGGd7d8eCZHGd9d:eCeZIgd;ZJy)=zPyTorch PEGASUS model.N)Callable)nn)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilinglogging)merge_with_config_defaults)OutputRecordercapture_outputs) PegasusConfig input_ids pad_token_iddecoder_start_token_idc|j|j}|ddddfj|ddddf<||dddf<| td|j |dk(||S)z1 Shift input ids one token to the right. Nr!rz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)r#r$r%shifted_input_idss w/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/pegasus/modeling_pegasus.pyshift_tokens_rightr/9s}"++IOO<(CRC0668ae4adLMM""#4# )D92D> 9D> input_ids_shapepast_key_values_length position_idsc|F|dd\}}tj|||ztj|jj}t ||S)z3`input_ids_shape` is expected to be [bsz x seqlen].Nr>)r?device)rFarangelongrArWr9forward)r;rSrTrUbszseq_lenr<s r.rZz,PegasusSinusoidalPositionalEmbedding.forward_s]  *2A.LC <<&(>(HPUPZPZcgcncncucuLw|,,r0N)rN) __name__ __module__ __qualname____doc__intr:rRrFno_gradSizeTensorrZ __classcell__r<s@r.r2r2JsNEcE#ECRVJEbfE U]]_pt -$zz -CF -Z_ZfZfimZm -  - -r0r2modulequerykeyvalueattention_maskscalingdropoutkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)Nr'r>rrLptrainingr!) sizerFmatmul transposer functionalsoftmaxrnru contiguous) rhrirjrkrlrmrnro attn_weights attn_outputs r.eager_attention_forwardr~ms**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r0ceZdZdZ ddedededededed edzd edzffd Z dd e jd e jdzde dzde jdzde e dee je jdzff dZxZS)PegasusAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsrn is_decoderbias is_causalconfig layer_idxc t |||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ ||_ |9|jr-tjd|jjdt!j"||||_t!j"||||_t!j"||||_t!j"||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rqzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.r)r9r:rrrnhead_dimrr+rmrrrlogger warning_oncer<r^rLineark_projv_projq_projout_proj) r;rrrnrrrrrr<s r.r:zPegasusAttention.__init__s$ "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $""     *4>>+B+B*CD,,  ii 94@ ii 94@ ii 94@  )YTB r0 hidden_stateskey_value_statespast_key_valuesrlror6c |du}|jdd}g|d|j}|j|j|j dd} d} |St |t rA|jj|j} |r |j} n|j} n|} |r|n|} |rK|I| rG j|jj} | j|jj}n|j| } |j!| }g| jddd|j}| j|j dd} |j|j dd}|K j#| ||j\} }|r)t |t rd|j|j<t%j&|j(j*t,}||| | ||f|j.sdn |j0|j2d|\}}|j4g|dj7}|j9|}||fS) z#Input shape: Batch x Time x ChannelNr'r!r>FT)rnrm)r)rrviewrx isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdater get_interfacer_attn_implementationr~rurnrmreshaper{r)r;rrrrlrois_cross_attention input_shape hidden_shape query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapeattention_interfacer}r|s r.rZzPegasusAttention.forwardsd.T9$))#2. 88b8$--8 {{=166|DNNqRST   &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML^4J;;~6LF--cr2FBF FH#2<(?(M(M KK , ,.E) %8      % $}}C$,,LL %  % ! \*k));;;;FFH mmK0 L((r0)rFTFNNNNN)r^r_r`rarbfloatboolr"r:rFrer rrtuplerZrfrgs@r.rrsG  '+ $%C%C%C %C  %C  %C%C$%C:%CT15(,.2 H)||H) ,,-H) H)  t+ H)-.H) u||U\\D00 1H)r0rc~eZdZdeffd Zdej dej deedej fdZ xZ S)PegasusEncoderLayerrcht||j|_t |j|j |j ||_tj|j|_ |j|_ t|j|_|j|_tj |j|j"|_tj |j"|j|_tj|j|_y)N)rrrnr)r9r:d_modelrrencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrnractivation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normr;rr<s r.r:zPegasusEncoderLayer.__init__s )nn44,,   %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r0rrlror6c |}|j|}|jd||d|\}}tjj ||j|j }||z}|}|j |}|j|j|}tjj ||j|j }|j|}tjj ||j|j }||z}|jtjk(rEtj|jjdz }tj || |}|S)a> Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. )rrlrsi)minmax)rrrryrnrurrrrrr?rFfloat16finforclamp)r;rrlroresidual_ clamp_values r.rZzPegasusEncoderLayer.forwardsT!11-@ )4>> ')   q  --mt||VZVcVc-d  =0  --m< **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0   %-- /++m&9&9:>>EK!KK K<[YMr0) r^r_r`r"r:rFrerrrZrfrgs@r.rrsL=}=$"||" "+, "  "r0rceZdZddededzffd Z ddejdejdzdejdzdejdzd edzd e dzd e e d ejfd Z xZ S)PegasusDecoderLayerNrrc t||j|_t |j|j |j dd|||_|j|_t|j|_ |j|_ tj|j|_t |j|j |j d|||_tj|j|_tj$|j|j&|_tj$|j&|j|_tj|j|_y)NT)rrrnrrrr)rnrrr)r9r:rrrdecoder_attention_headsrrrnrrrrrrr encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrrr)r;rrr<s r.r:zPegasusDecoderLayer.__init__:s  )nn44,, ~~ #F$>$>?"(";";$&LL$@!, NN  * *,,  (*||DNN'C$99T^^V-C-CD99V33T^^D " T^^ <r0rrlencoder_hidden_statesencoder_attention_maskr use_cacheror6c |}|j|}|jd|||d|\}} tjj ||j|j }||z}|h|}|j |}|jd||||d|\}} tjj ||j|j }||z}|}|j|}|j|j|}tjj ||j|j }|j|}tjj ||j|j }||z}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states )rrrlrs)rrrlrr) rrrryrnrurrrrrrr) r;rrlrrrrrorrs r.rZzPegasusDecoderLayer.forwardYs*!11-@ *4>> '+)    q  --mt||VZVcVc-d  =0  ! ,$H 88GM0t00 +!65 /      M1MM11-4<?Z>[[_`a.I +C KK / / KK       4''..0D0D0R0R0TU  ,r0r6c|jSz8 Returns the position embeddings matrix rr;s r.get_position_embeddingsz&PegasusEncoder.get_position_embeddings###r0roc F|du|duz r td||j||jz}|jdd}|j |}||z}t j j||j|j}t|j||}t|jD]F\}} d} |jr&tjg} | |jkrd} | r=| ||fi|}H|j!|}t#|S)Nz:You must specify exactly one of input_ids or inputs_embedsr'rs)r inputs_embedsrlFT)last_hidden_state)r+rrr)rrryrnrur r enumeraterrFrandrrr) r;r#rlrror embed_posridx encoder_layerto_dropdropout_probabilitys r.rZzPegasusEncoder.forwards3  -t"; <YZ Z   --i84;K;KKM#))#2. ((5 % 1  --mt||VZVcVc-d 2;;')  #,DKK"8  CG}}&+jjn#&7"G -!"!!   6 +  r0r)r^r_r`rarr_can_record_outputsr"r:rbrrrr rr rrrrrZrfrgs@r.rrs-& }2-c-0$$   - +, -  -  - r0rceZdZdZeeeddeedddZdeffd Z d e fd Z d e jfd Zeee dd eed efdZxZS)PegasusDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`PegasusDecoderLayer`] Args: config: PegasusConfig embed_tokens (nn.Embedding): output embedding r!r)index layer_namer)rrcross_attentionsrc t|||j|_|j|_|j |_|j|_|jrtj|jnd|_ tj|j |j|j |_t%|j|j|j |_tj(t+|j,Dcgc]}t/||c}|_tj2|j|_d|_|j9ycc}w)Nr)rF)r9r:rndecoder_layerdroprr$r5rmax_target_positionsrrrrrrrrrr2rrrDdecoder_layersrrrrrr)r;rir<s r.r:zPegasusDecoder.__init__1s  ~~ 11!..$*$B$B!8>8N8N499V^^4TWLL):):FNNDL\L\]C  * * NN     mmW\]c]r]rWs$tRS%81%M$tu ,,v~~6&+#  %usE3rctjd|d||j_t |jj|jj |j |_tj|jj|jj|jj|jyrrrs r.rz)PegasusDecoder.resize_position_embeddingsGrr0r6c|jSr r r s r.r z&PegasusDecoder.get_position_embeddings_rr0roc t|du|duz r td||j|}||jz}|rd|b||jjr4t t |jt |jnt |j}|jdd\} } ||jnd} tj| |j| z} |1ts'| | z} tj| | |j}t|t r |jn|}t!|j|||}t#|j|||}|j%| | f| | }||z}t&j(j+||j*|j, }t/|j0D]E\}}|j,r%tj2g}||j4kr7||||f|||d |}G|j7|}t9|| S) NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time)rr'rrW)rrrlr)rrrlr)rUrs)rrr)rr)r+rrris_encoder_decoderr r rvget_seq_lengthrFrXrWronesrrrr rrryrnrurrrrrr)r;r#rlrrrrrro batch_size seq_lengthrTrUmask_seq_lengthself_attn_cache causal_mask positionsrr decoder_layerrs r.rZzPegasusDecoder.forwardesQ  -t"; <st t   --i8M&(8(88  0)4 8V8V$L $DlZ^ZeZeFfg!5  "/!3!3!5cr!: JETE`!?!?!Afg||J}7K7KLOee  !*B*D4zAO"ZZ OML`L`aN/+>?  0 0   );;')+  ";;;'1"7 " ((*j)ACYht(u % 1  --mt||VZVcVc-d "+DKK"8  C}}&+jjn#&7)%(> /# M " 6 8++  r0)NNNNNNN)r^r_r`rarrrrr"r:rbrrrr rr rrrrrZrfrgs@r.rr"s-$%5Q;W*+;1Q_` },-c-0$$  "#S +,S  3S  S r0rceZdZdddZdeffd ZdZdZdefdZ d e e jfd Z ee dd ej"d zd ej"d zdej"d zdej"d zde ej$d zded zdej"d zdej"d zded zdeed e ezfdZxZS) PegasusModelz shared.weight)zdecoder.embed_tokens.weightzencoder.embed_tokens.weightrct|||j|j}}t j ||j ||_t||_ t||_ |jyr]) r9r:r$rrrrsharedrencoderrdecoderr)r;rr5rr<s r.r:zPegasusModel.__init__sa  "("5"5v7H7HZ ll:v~~{K %f- %f-  r0c|jSr])r5r s r.get_input_embeddingsz!PegasusModel.get_input_embeddingss {{r0c~||_|j|j_|j|j_yr])r5r6rr7r;rks r.set_input_embeddingsz!PegasusModel.set_input_embeddingss) $(KK !$(KK !r0rc||j_|jj||jj|yrN)rrr6rr7rs r.rz'PegasusModel.resize_position_embeddingss7/J + //0KL //0KLr0r6cj|jj|jjfSr )r6r r7r s r.r z$PegasusModel.get_position_embeddingss) 446 8\8\8^__r0Nr#rldecoder_input_idsdecoder_attention_maskencoder_outputsrrdecoder_inputs_embedsrroc ||jd |||d| }nGt|ts7t|dt|dkDr|dndt|dkDr|dnd}|jd |||d|||| d| } t | j | j| j| j| j|j |j|jS) aE decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pegasus uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, PegasusModel >>> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-large") >>> model = PegasusModel.from_pretrained("google/pegasus-large") >>> inputs = tokenizer("Studies have been shown that owning a dog is good for you", return_tensors="pt") >>> decoder_inputs = tokenizer("Studies show that", return_tensors="pt") >>> outputs = model(input_ids=inputs.input_ids, decoder_input_ids=decoder_inputs.input_ids) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 4, 1024] ```N)r#rlrrr!r>)rrrr#rlrrrrr)rrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentionsr) r6rrlenr7rrrrrr) r;r#rlr@rArBrrrCrrodecoder_outputss r.rZzPegasusModel.forwards`  "*dll#-+ OO_=-"1!"4474H14Loa0RV14_1E1I?1-tO'$,,  '1"1!"4#1+/    "-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r0) NNNNNNNNN)r^r_r`_tied_weights_keysr"r:r9r<rbrrrrr rrrFrerHr rrrrrZrfrgs@r.r3r3sZ(7'6 } 0 McM"`r||)<` *..2156:;?(,-159!%R <<$&R  t+R !<<$. R !& t 3 R u001D8 R R ||d*R  %||d2R $;R +,R  # #R R r0r3zY The PEGASUS Model with a language modeling head. Can be used for summarization. ) custom_introc"eZdZdZdgZddiZdeffd Z dded edzd e d e jffd Z ded dfd Z defdZd ee jfdZee ddej(dzdej(dzdej(dzdej(dzdeej*dzdedzdej(dzdej(dzdej(dzde dzdeed eezfdZdej(fdZxZS) rrrlm_head.weightzmodel.shared.weightrcxt||t||_|j dt j d|jjjftj|j|jjjd|_ |jy)Nrr!Fr)r9r:r3rregister_bufferrFzerosr5num_embeddingsrrrlm_headrrs r.r:z(PegasusForConditionalGeneration.__init__Ss  !&)  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  r0Nnew_num_tokenspad_to_multiple_of mean_resizingr6czt||||}|j|jjd|S)Nr)r9resize_token_embeddings_resize_final_logits_biasrAr))r;rUrVrWnew_embeddingsr<s r.rYz7PegasusForConditionalGeneration.resize_token_embeddings\s?8I[]jk &&~'<'<'B'B1'EFr0c6|jjd}||kr|jddd|f}nStjd||z f|jj}tj |j|gd}|j d|y)Nr'r!r'rrr)rr)rFrRrWcatrQ)r;rUold_num_tokensnew_bias extra_biass r.rZz9PegasusForConditionalGeneration._resize_final_logits_biascs//55b9 ^ +--a..@AHa.)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;r0rc||j_|jjj ||jj j |yr>)rrrr6rr7rs r.rz:PegasusForConditionalGeneration.resize_position_embeddingslsC/J + 556QR 556QRr0c|jjj|jjjfSr )rr6r r7r s r.r z7PegasusForConditionalGeneration.get_position_embeddings}s5 ""::P>P>h>h>jkkr0r#rlr@rArBrrrClabelsrroc z| R| rtjdd} |7|5t| |jj|jj }|j |f|||||||| d| } |j| j|jz} d}| Ft}|| jd|jj| jd}t|| | j| j| j | j"| j$| j&| j( S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pegasus uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Summarization: ```python >>> from transformers import AutoTokenizer, PegasusForConditionalGeneration >>> model = PegasusForConditionalGeneration.from_pretrained("google/pegasus-xsum") >>> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-xsum") >>> ARTICLE_TO_SUMMARIZE = ( ... "PG&E stated it scheduled the blackouts in response to forecasts for high winds " ... "amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were " ... "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow." ... ) >>> inputs = tokenizer(ARTICLE_TO_SUMMARIZE, max_length=1024, return_tensors="pt") >>> # Generate Summary >>> summary_ids = model.generate(inputs["input_ids"]) >>> tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "California's largest electricity provider has turned off power to hundreds of thousands of customers." ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rlr@rBrArrrCrr') losslogitsrrFrGrrHrrI)rwarningr/rr$r%rrTrrrrrrrrFrGrrHrrI)r;r#rlr@rArBrrrCrcrrooutputs lm_logitsmasked_lm_lossloss_fcts r.rZz'PegasusForConditionalGeneration.forwardsBr  klI (-B-J$6DKK44dkk6X6X%!'1djj  ' )/+#9+'"7 '  ' LL!:!:;d>T>TT   ')H%innR9O9O&PRXR]R]^`RabN#33")"?"?&99$55&-&G&G")"?"?&99  r0clt||jj|jjSr])r/rr$r%)r;rcs r.%prepare_decoder_input_ids_from_labelszEPegasusForConditionalGeneration.prepare_decoder_input_ids_from_labelss%!&$++*B*BDKKDfDfggr0)NT) NNNNNNNNNN)r^r_r`r_keys_to_ignore_on_load_missingrLr"r:rbrrrrYrZrrr rrrFrerHr rrrrZrmrfrgs@r.rrGs  ':&;#/}ae!7:TzY] <<<ScS"lr||)<l *..2156:;?(,-159&*!%] <<$&]  t+] !<<$. ] !& t 3 ] u001D8 ] ] ||d*]  %||d2]  t#] $;] +,]   ] ] ~hELLhr0rc(eZdZdZfdZdZxZS)PegasusDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cdt||t||_|j yr])r9r:rr7rrs r.r:zPegasusDecoderWrapper.__init__s&  %f-  r0c&|j|i|Sr])r7)r;argsros r.rZzPegasusDecoderWrapper.forwardst||T,V,,r0)r^r_r`rar:rZrfrgs@r.rprps  -r0rpceZdZddiZfdZdZdZdejfdZ de fd Z e e dd ejd zd ej d zd ej"d zdej"d zded zdej"d zdejd zded zde ej zdeedeezfdZxZS)PegasusForCausalLMrOz!model.decoder.embed_tokens.weightc tj|}d|_d|_t||t ||_tj|j|jd|_ |jy)NTFr)copydeepcopyrr(r9r:rprrr hidden_sizerrTrrs r.r:zPegasusForCausalLM.__init__sfv& $)!  *62 yy!3!3V5F5FUS  r0cB|jjjSr]rr7rr s r.r9z'PegasusForCausalLM.get_input_embeddings szz!!...r0c:||jj_yr]r{r;s r.r<z'PegasusForCausalLM.set_input_embeddings s*/ 'r0r6cJ|jjjSr )rr7r r s r.r z*PegasusForCausalLM.get_position_embeddingsszz!!99;;r0rcp||j_|jjj |yr>)rrrr7rrs r.rz-PegasusForCausalLM.resize_position_embeddingss*/J + 556QRr0Nr#rlrrrrrcrlogits_to_keeproc |jjd|||||||d| } | d} t| tr t | dn| } |j | dd| ddf}d}|a|j |j}t}||jd|jj|jd}t||| j| j| j| j S)aJ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PegasusForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-large") >>> model = PegasusForCausalLM.from_pretrained("google/pegasus-large") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```rErNr')rerfrrrrr)rr7rrbslicerTrrWrrrrrrrrr)r;r#rlrrrrrcrrrorhr slice_indicesrfrerks r.rZzPegasusForCausalLM.forward%sN>PTZZ=O=O > )"7#9+' >  >   8B>SV8W~ot4]k mA}a,?@A  YYv}}-F')HFKKDKK,B,BCV[[QS_UD0#33!//))$55   r0) NNNNNNNNr)r^r_r`rLr:r9r<rrr rbrrrrF LongTensorrerHr rrrrrrZrfrgs@r.rurusM= /0<< ScS .2.2:>;?(,26*.!%-.A ##d*A  t+A  %0047 A !& 1 1D 8 A  A ((4/A   4'A $;A ell*A +,A  2 2A A r0ru)rurr3r)Nr)Krarwrcollections.abcrnumpyrBrFrtorch.nnrrr activationsr cache_utilsr r r generationr masking_utilsr rmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrutils.output_capturingrr configuration_pegasusr" get_loggerr^rrerbr/rr2Modulerr~rrrrrrr3rrpru__all__rr0r.rs $ %&!CC)JB9G&8E0   H % %,, c [^ "-2<<-R!% II% <<% % << % LL4' % T\ %%'(%:r)ryyr)l545rZ4Zz2_22$v +v rY +Y xE )E E P Yh&