^jD\dZddlZddlZddlmZddlZddlmZddlmZm Z m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2ddl3m4Z4m5Z5ddl6m7Z7e/jpe9Z:dejvdeeHeZNe,d?<Gd@dAeHZOe,GdBdCeHZPGdDdEeHZQe,dF<GdGdHeHeZRgdIZSy)KzPyTorch BART model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilingloggingtorch_compilable_check)merge_with_config_defaults)OutputRecordercapture_outputs) BartConfig input_ids pad_token_iddecoder_start_token_idc|j|j}|ddddfj|ddddf<||dddf<| td|j |dk(||S)z1 Shift input ids one token to the right. Nr&rz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)r(r)r*shifted_input_idss q/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/bart/modeling_bart.pyshift_tokens_rightr4:s}"++IOO<(CRC0668ae4adLMM""#4#forwardr=)r@r(rBrCbszseq_lenrAs r3rMz&BartLearnedPositionalEmbedding.forwardUs  $??2A.LC <<&(>(HPUPZPZcgcncncucufS"o (11!4Lw|dkk9::r5)rN) __name__ __module__ __qualname____doc__intr?rGTensorrM __classcell__rAs@r3r7r7JsWFsF3Fmq ; ;?B ;V[VbVbeiVi ; ;r5r7c `eZdZdZd dededededzffd Zdejffd Z xZ S) BartScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. r8r9 padding_idx embed_scaleNc6t||||||_yN)r>r?r[)r@r8r9rZr[rAs r3r?z BartScaledWordEmbedding.__init__js  D&r5r(c<t|||jzSr])r>rMr[)r@r(rAs r3rMzBartScaledWordEmbedding.forwardnswy)D,<,<<L==((6??([L,,|U3K''1-88:K  $$r5ceZdZdZ ddedededededed edzd edzffd Z dd e jd e jdzde dzde jdzde e dee je jdzff dZxZS) BartAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsrh is_decoderbias is_causalconfig layer_idxc t |||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ ||_ |9|jr-tjd|jjdt!j"||||_t!j"||||_t!j"||||_t!j"||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rkzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.r)r>r?r|r}rhhead_dimrr0rgr~rrlogger warning_oncerArPrLineark_projv_projq_projout_proj) r@r|r}rhr~rrrrrAs r3r?zBartAttention.__init__s$ "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $""     *4>>+B+B*CD,,  ii 94@ ii 94@ ii 94@  )YTB r5 hidden_stateskey_value_statespast_key_valuesrfrireturnc |du}|jdd}g|d|j}|j|j|j dd} d} |St |t rA|jj|j} |r |j} n|j} n|} |r|n|} |rK|I| rG j|jj} | j|jj}n|j| } |j!| }g| jddd|j}| j|j dd} |j|j dd}|K j#| ||j\} }|r)t |t rd|j|j<t%j&|j(j*t,}||| | ||f|j.sdn |j0|j2d|\}}|j4g|dj7}|j9|}||fS) z#Input shape: Batch x Time x ChannelNr,r&r<FT)rhrg)r.rrviewrs isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdater get_interfacer_attn_implementationryrprhrgreshapervr)r@rrrrfriis_cross_attention input_shape hidden_shape query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapeattention_interfacerxrws r3rMzBartAttention.forwardsd.T9$))#2. 88b8$--8 {{=166|DNNqRST   &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML^4J;;~6LF--cr2FBF FH#2<(?(M(M KK , ,.E) %8      % $}}C$,,LL %  % ! \*k));;;;FFH mmK0 L((r5)rFTFNNNNN)rPrQrRrSrTraboolr'r?rGrUr rrtuplerMrVrWs@r3r{r{sG  $( $%C%C%C %C  %C  %C%CT!%C:%CT15(,.2 H)||H) ,,-H) H)  t+ H)-.H) u||U\\D00 1H)r5r{ceZdZd dededzffd Zdejdejdee dejfd Z xZ S) BartEncoderLayerNrrcjt||j|_t |j|j |j |||_tj|j|_ |j|_ t|j|_|j|_tj |j|j"|_tj |j"|j|_tj|j|_y)N)r|r}rhrr)r>r?d_modelr|r{encoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrhr activation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normr@rrrAs r3r?zBartEncoderLayer.__init__s &nn44,,  %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r5rrfrirc F|}|j|fd|i|\}}tjj||j|j}||z}|j |}|}|j |j|}tjj||j|j}|j|}tjj||j|j}||z}|j|}|jtjk(rhtj|jsEtj |jj"dz }tj$|| |}|S)Nrfrni)minmax)rrrtrhrprrrrrrrErGfloat16isfiniteallfinforclamp)r@rrfriresidual_ clamp_values r3rMzBartEncoderLayer.forwardsh !)4>>  )   q  --mt||VZVcVc-d  =0 11-@  **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0 --m<   %-- /}8U8Y8Y8[++m&9&9:>>EK!KK K<[YMr5r]) rPrQrRr'rTr?rG FloatTensorrrrUrMrVrWs@r3rrs[=z=cDj=&(())+,   r5rceZdZddededzffd Z ddejdejdzdejdzdejdzd edzd e dzd e e d ejfd Z xZ S)BartDecoderLayerNrrc t||j|_t |j|j |j dd|||_|j|_t|j|_ |j|_ tj|j|_t |j|j |j d|||_tj|j|_tj$|j|j&|_tj$|j&|j|_tj|j|_y)NT)r|r}rhr~rrr)rhr~rr)r>r?rr|r{decoder_attention_headsrrrhr rrrrrr encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrrrrs r3r?zBartDecoderLayer.__init__8s  &nn44,, ~~ #F$>$>?"(";";$&LL$@!) NN  * *,,  (*||DNN'C$99T^^V-C-CD99V33T^^D " T^^ <r5rrfencoder_hidden_statesencoder_attention_maskr use_cacherirc |}|j|f||d|\}} tjj||j|j}||z}|j |}|h|}|j |f|||d|\}} tjj||j|j}||z}|j|}|}|j|j|}tjj||j|j}|j|}tjj||j|j}||z}|j|}|S)N)rrfrn)rrfr) rrrtrhrprrrrrrrr) r@rrfrrrrrirrs r3rMzBartDecoderLayer.forwardWs!*4>>  +)    q  --mt||VZVcVc-d  =0 11-@  ! ,$H0t00 !65 /      M1MM11-4<r?rrdenseDropoutrhr)r@rrrrrAs r3r?zBartClassificationHead.__init__sD YYy)4 zzN3  )[9 r5rrc|j|}|j|}tj|}|j|}|j |}|Sr])rhrrGtanhr)r@rs r3rMzBartClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r5r`rWs@r3rrsL7 : : : :  :U\\ellr5rcheZdZUeed<dZdZddgZddgZdgZ dZ dZ dZ dZ fd Zed ZxZS) BartPreTrainedModelrmodelTzencoder.versionzdecoder.versionrrrct||t|tr t j |j yyr])r> _init_weightsrBartForConditionalGenerationinitzeros_final_logits_bias)r@rbrAs r3rz!BartPreTrainedModel._init_weightss2 f% f: ; KK00 1 r?rhencoder_layerdrop layerdroprr)rZmax_position_embeddingsmax_source_positionsscale_embeddingmathsqrtrY vocab_size embed_tokensr7embed_positionsr ModuleListrangeencoder_layersrrrlayernorm_embeddinggradient_checkpointing post_init)r@rr|r[irAs r3r?zBartEncoder.__init__s  ~~ 11NN !..$*$B$B!.4.D.Ddii *# 3   y$*:*:   >  * *  mmTYZ`ZoZoTp$qq%5f%J$qr #%<< #: &+#  %rs-D?Nr(rf inputs_embedsrirc X|du|duz r td||j|}|j|dddddf}|j|j}||z}|j |}t jj||j|j}t|j||}t|jD]F\}}d} |jr&tjg} | |j krd} | r=|||fi|}Ht#|S)Nz:You must specify exactly one of input_ids or inputs_embedsr,rn)rr'rfFT)last_hidden_state)r0rrtorFr#rrtrhrprr enumeraterrGrandrr) r@r(rfr'ri embed_posridx encoder_layerto_dropdropout_probabilitys r3rMzBartEncoder.forwards7  -t"; <YZ Z   --i8M((q!Rx)@A LL!5!56 % 1 00?  --mt||VZVcVc-d 2;;')  #,DKK"8  CG}}&+jjn#&7"G -!"!!  +  r5r)rPrQrRrSrr{_can_record_outputsr'r?r#r%rrG LongTensorrUrrrrrMrVrWs@r3rrs*# z4 .2.226 * ##d**  t+* ((4/ * +, *  *  * r5rcFeZdZdZeeeddeedddZdeffd Z e e e dd e jd zd e jd zd e j d zd e jd zded zde j d zded zdeedefdZxZS) BartDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`BartDecoderLayer`] Args: config: BartConfig embed_tokens (nn.Embedding): output embedding r&r)index layer_namer)rrcross_attentionsrc t|||j|_|j|_|j |_|j|_|jrtj|jnd}t|j|j|j ||_t!|j|j|_t%j&t)|j*Dcgc]}t-||c}|_t%j0|j|_d|_|j7ycc}wr)r>r?rhdecoder_layerdroprr)rZrmax_target_positionsrrrrrYrrr7rrr r!decoder_layersrrrr#r$r%)r@rr[r&rAs r3r?zBartDecoder.__init__7s  ~~ 11!..$*$B$B!393I3Idii/s 3   v~~t/?/?[  >  * * NN mmTYZ`ZoZoTp$qq%5f%J$qr #%<<#? &+#  %rs?ENr(rfrrrr'rrirc |du|duz r td||j|}|rd|b||jjr4t t |jt |jnt |j}|j dd\} } ||jnd} tj| |j| z} |1ts'| | z} tj| | |j}t|tr |jn|}t|j|||}t!|j|||}|j#|| | }|j%|j}||z}|j'|}t(j*j-||j,|j. }t1|j2D]E\}}|j.r%tj4g}||j6kr7||||f|||d |}Gt9|| S) NzJYou must specify exactly one of decoder_input_ids or decoder_inputs_embeds)rr,rr)rr'rfr)rr'rfr)rCrn)rrr)r)r)r0rris_encoder_decoderr r rqget_seq_lengthrGrHrFr onesrrrrrr*r#rrtrhrpr+rr,rr)r@r(rfrrrr'rri batch_size seq_lengthrBrCmask_seq_lengthself_attn_cache positionsrr. decoder_layerr1s r3rMzBartDecoder.forwardOsP  -t"; <ij j   --i8M 0)4 8V8V$L $DlZ^ZeZeFfg!5  "/!3!3!5cr!: JETE`!?!?!Afg||J}7K7KLOee  !*B*D4zAO"ZZ OML`L`aN/+>?  0 0   ,;;')+   ";;;'1"7 " ((4JYe(f LL!5!56 % 1 00?  --mt||VZVcVc-d "+DKK"8  C}}&+jjn#&7)%(> /# M "9++  r5)NNNNNNN)rPrQrRrSrr$r{r2r'r?r#r%rrGr3rUrr rrrrrMrVrWs@r3r5r5(s*$]! T*=n] z0 .2.2:>:>(,26!%R ##d*R  t+R  %0047 R !& 0 04 7 R  R ((4/R $;R +,R  3R  R r5r5cjeZdZdddZdeffd ZdZdZee dde jdzd e jdzd e jdzd e jdzd e e jdzd edzde jdzde jdzdedzdeedeezfdZxZS) BartModelz shared.weight)zdecoder.embed_tokens.weightzencoder.embed_tokens.weightrcJt|||j|j}}|jrt j |jnd}t||j|||_ t||_ t||_ |jy)Nr_r)r>r?r)rrrrrrYsharedrencoderr5decoderr%)r@rrZrr[rAs r3r?zBartModel.__init__s|  "("5"5v7H7HZ 393I3Idii/s -j&..+cno "6* "6*  r5c|jSr])rJrs r3get_input_embeddingszBartModel.get_input_embeddingss {{r5c~||_|j|j_|j|j_yr])rJrKrrLr@res r3set_input_embeddingszBartModel.set_input_embeddingss) $(KK !$(KK !r5Nr(rfdecoder_input_idsdecoder_attention_maskencoder_outputsrr'decoder_inputs_embedsrrirc N|D|B| tdt||jj|jj}||j d |||d| }nGt |ts7t|dt|dkDr|dndt|dkDr|dnd}|jd |||d|||| d| } t| j| j| j| j| j|j|j|j S)  decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. NzIf no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.)r(rfr'rr&r<)r)rrr(rfrrrr'r)r)rdecoder_hidden_statesdecoder_attentionsr8encoder_last_hidden_staterencoder_attentionsr )r0r4rr)r*rKrrlenrLrr)rrrr8) r@r(rfrRrSrTrr'rUrridecoder_outputss r3rMzBartModel.forwardsdP  $)>)F  U !34;;33T[[5W5W!   "/;t||0#-+0 0O O_=-"1!"4474H14Loa0RV14_1E1I?1-tO FRT\\ F '1"1!"4#1+/ F  F "-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r5 NNNNNNNNN)rPrQrR_tied_weights_keysr'r?rNrQrrrGr3rUlistrr rrrrrrMrVrWs@r3rHrHs?(7'6 z 0 .2.259:>:>(,26:>!%T ##d*T  t+T !++d2 T !& 0 04 7 T e//047 T T ((4/T  %0047T $;T +,T  # #T T r5rHzV The BART Model with a language modeling head. Can be used for summarization. ) custom_introceZdZdZddiZdgZdeffd Z dded edzd e d e jffd Z ded dfd Z ee ddej"dzdej$dzdej"dzdej"dzdeej(dzdedzdej(dzdej(dzdej"dzde dzdeed eezfdZdej$fdZxZS)rrlm_head.weightzmodel.shared.weightrrcxt||t||_|j dt j d|jjjftj|j|jjjd|_ |jy)Nrr&Fr)r>r?rHrregister_bufferrGzerosrJr8rrrlm_headr%r@rrAs r3r?z%BartForConditionalGeneration.__init__(s  v&  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  r5Nnew_num_tokenspad_to_multiple_of mean_resizingrczt||||}|j|jjd|S)Nr)r>resize_token_embeddings_resize_final_logits_biasrJr.)r@rjrkrlnew_embeddingsrAs r3rnz4BartForConditionalGeneration.resize_token_embeddings1s?8I[]jk &&~'<'<'B'B1'EFr5c6|jjd}||kr|jddd|f}nStjd||z f|jj}tj |j|gd}|j d|y)Nr,r&rrlr)rr.rGrgrFcatrf)r@rjold_num_tokensnew_bias extra_biass r3roz6BartForConditionalGeneration._resize_final_logits_bias8s//55b9 ^ +--a..@AHa.)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;r5r(rfrRrSrTrr'rUlabelsrric | R| rtjdd} |7|5t| |jj|jj }|j |f|||||||| d| } |j| d} | |jj| jz} d}| a| j| j} t}|| jd|jj| jd}t|| | j| j | j"| j$| j&| j(| j* S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example summarization: ```python >>> from transformers import AutoTokenizer, BartForConditionalGeneration >>> model = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn") >>> ARTICLE_TO_SUMMARIZE = ( ... "PG&E stated it scheduled the blackouts in response to forecasts for high winds " ... "amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were " ... "scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow." ... ) >>> inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors="pt") >>> # Generate Summary >>> summary_ids = model.generate(inputs["input_ids"], num_beams=2, min_length=0, max_length=20) >>> tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] 'PG&E scheduled the blackouts in response to forecasts for high winds amid dry conditions' ``` Mask filling example: ```python >>> from transformers import AutoTokenizer, BartForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-base") >>> model = BartForConditionalGeneration.from_pretrained("facebook/bart-base") >>> TXT = "My friends are but they eat too many carbs." >>> input_ids = tokenizer([TXT], return_tensors="pt")["input_ids"] >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['not', 'good', 'healthy', 'great', 'very'] ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rfrRrTrSrr'rUrrr, losslogitsrrYrZr8r[rr\)rwarningr4rr)r*rrhrr*rFrrrrrrYrZr8r[rr\)r@r(rfrRrSrTrr'rUrvrrioutputs lm_logitsmasked_lm_lossloss_fcts r3rMz$BartForConditionalGeneration.forwardAsjh  klI (-B-J$6DKK44dkk6X6X%!'1djj  ' )/+#9+'"7 '  ' LL,  6 6 9 9):J:J KK   YYy//0F')H%innR9O9O&PRXR]R]^`RabN#33")"?"?&99$55&-&G&G")"?"?&99  r5clt||jj|jjSr])r4rr)r*)r@rvs r3%prepare_decoder_input_ids_from_labelszBBartForConditionalGeneration.prepare_decoder_input_ids_from_labelss%!&$++*B*BDKKDfDfggr5)NT NNNNNNNNNN)rPrQrRrr`_keys_to_ignore_on_load_missingr'r?rTrr EmbeddingrnrorrrGr3rUrarr rrrrrMrrVrWs@r3rrs  /(;&;#zae!7:TzY] <<<.2.259:>:>(,26:>*.!%{ ##d*{  t+{ !++d2 { !& 0 04 7 { e//047 { { ((4/{  %0047{   4'{ $;{ +,{   { { zhELLhr5rz Bart model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cheZdZdeffd Zee ddejdzdejdzdejdzdejdzde ejdzd ejdzd ejdzd ejdzd e dzd e edeezfdZxZS)BartForSequenceClassificationrc t||fi|t||_t |j |j |j |j|_|jyr]) r>r?rHrrr num_labelsclassifier_dropoutclassification_headr%)r@rrirAs r3r?z&BartForSequenceClassification.__init__sZ *6*v& #9 NN NN     % % $   r5Nr(rfrRrSrTr'rUrvrrirc |d} |$|"td|jj|j|f||||||| d| } | d} |j |j j j| j} ttj| jdjdk(d| | ddf}t|jd| jdzdk\d|j| j!dd | j!d ddd ddf}|j#|}d}||j|j}|j j$|j j&dk(rd |j _nv|j j&dkDrL|j(tj*k(s|j(tj,k(rd |j _nd |j _|j j$d k(rSt/}|j j&dk(r&||j1|j1}n|||}n|j j$d k(rGt3}||jd |j j&|jd }n,|j j$d k(rt5}|||}t7||| j8| j:| j<| j>| j@| jB| jD S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for rfrRrSrTr'rUrrr&z7All examples must have the same number of tokens.z3Each example must contain at least one token.r, regressionsingle_label_classificationmulti_label_classificationrx)#NotImplementedErrorrArPreqr eos_token_idr*rFr"rGunique_consecutivesumnumelr.rrqr problem_typerrErIrTrsqueezerrrrrYrZr8r[rr\)r@r(rfrRrSrTr'rUrvrrir|reos_maskselectedsentence_representationrzryrs r3rMz%BartForSequenceClassification.forwards R  I  !:%J4>>KbKbJcd '1djj  ' )/#9+'"7 '  '   << 8 89<<]=Q=QR  $ $X\\!_ 5 ; ; = B E !1- NN1 !4!4Q!7 71 < A #+-- 0B0B10Er=K]K]^`Ka"bcdfhjkck"l))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./.#33")"?"?&99$55&-&G&G")"?"?&99  r5r_)rPrQrRr'r?rrrGr3rUrarrrrrrrMrVrWs@r3rrs, z .2.259:>:>26:>*.!%l ##d*l  t+l !++d2 l !& 0 04 7 l e//047 l ((4/l  %0047l   4'l $;l +,l  0 0l l r5rceZdZfdZee ddejdzdejdzdejdzdejdzde ejdzdejdzd ejdzd ejdzd ejdzd e dzd e e deezfdZxZS)BartForQuestionAnsweringct||d|_|j|_t||_t j |j|j|_|jyr;) r>r?rrHrrr hidden_size qa_outputsr%ris r3r?z!BartForQuestionAnswering.__init__Ks[   ++v& ))F$6$68I8IJ r5Nr(rfrRrSrTstart_positions end_positionsr'rUrrirc D||d} |j|f|||||| | d| } | d} |j| }|jdd\}}|jdj }|jdj }d}||t |j dkDr|jd}t |j dkDr|jd}|j d}|jd|}|jd|}t|}|||}|||}||zd z }t|||| j| j| j| j| j| j| j S) rWNFrrr&r,rl) ignore_indexr<) ry start_logits end_logitsrrYrZr8r[rr\)rrsplitrrvr]rqrrrrrYrZr8r[rr\)r@r(rfrRrSrTrrr'rUrrir|sequence_outputrzrr total_loss ignored_indexr start_lossend_losss r3rMz BartForQuestionAnswering.forwardWsN  &=+DI&0djj  ' )/#9+'"7 '  ' "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J2%!#33")"?"?&99$55&-&G&G")"?"?&99  r5r)rPrQrRr?rrrGrUr3rarrrrrrrMrVrWs@r3rrIs< *..259:>:>371526:>!%W <<$&W  t+W !++d2 W !& 0 04 7 W e//047 W ))D0W ''$.W ((4/W  %0047W $;W +,W  4 4W W r5rc(eZdZdZfdZdZxZS)BartDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cdt||t||_|j yr])r>r?r5rLr%ris r3r?zBartDecoderWrapper.__init__s&  "6*  r5c&|j|i|Sr])rL)r@argsris r3rMzBartDecoderWrapper.forwardst||T,V,,r5)rPrQrRrSr?rMrVrWs@r3rrs  -r5rzu BART decoder with a language modeling head on top (linear layer with weights tied to the input embeddings). c\eZdZddiZfdZdZdZee dde jdzde jdzd e jdzd e jdzd e dzd e jdzd e jdzdedzdee jzdeedeezfdZxZS)BartForCausalLMrdz!model.decoder.embed_tokens.weightcd|_d|_t| |t ||_t j|j|jd|_ |jy)NTFr) r~r>r>r?rrrrrrrhr%ris r3r?zBartForCausalLM.__init__sX $)!  '/ yy!3!3V5F5FUS  r5cB|jjjSr]rrLrrs r3rNz$BartForCausalLM.get_input_embeddingsszz!!...r5c:||jj_yr]rrPs r3rQz$BartForCausalLM.set_input_embeddingss*/ 'r5Nr(rfrrrr'rvrlogits_to_keeprirc |jjd|||||||d| } | d} t| tr t | dn| } |j | dd| ddf}d}|a|j |j}t}||jd|jj|jd}t||| j| j| j| j S)a@ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("facebook/bart-base") >>> model = BartForCausalLM.from_pretrained("facebook/bart-base") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```rXrNr,)ryrzrrrr8r )rrLrrTslicerhr*rFrrrrrrrrr8)r@r(rfrrrr'rvrrrir|r slice_indicesrzryrs r3rMzBartForCausalLM.forwardsL>PTZZ=O=O > )"7#9+' >  >   8B>SV8W~ot4]k mA}a,?@A  YYv}}-F')HFKKDKK,B,BCV[[QS_UD0#33!//))$55   r5) NNNNNNNNr)rPrQrRr`r?rNrQrrrGr3rUrr rrTrrrrrMrVrWs@r3rrs1 = /0.2.2:>;?(,26*.!%-.A ##d*A  t+A  %0047 A !& 1 1D 8 A  A ((4/A   4'A $;A ell*A +,A  2 2A A r5r)rrrrrHrrr)Nr)TrSrrcollections.abcrrGrtorch.nnrrrr r activationsr cache_utilsr r r generationr masking_utilsrrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr r!r" utils.genericr#utils.output_capturingr$r%configuration_bartr' get_loggerrPrrUrTr4rr7rYModuleraryr{rrrrrrrr5rHrrrrr__all__r r5r3rs $ AA&!CC)JB9G&8E*   H % %,, c [^  ;R\\;6 =bll =(!% II% <<% % << % LL4' % T\ %%'(%8r)BIIr)j010fO1OdRYY0/: -  - V %V r| %| ~q #q q h `h#6`h  `hF | $7|  | ~f 2f f R -, - Y )?Y  Y x r5