^j dZddlZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZdd lmZmZmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZmZm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2ddl3m4Z4m5Z5ddl6m7Z7e-r e/jpe9Z:dejvdeeHZNGd?d@eHZOGdAdBeHeZPgdCZQy)EzPyTorch MBART model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torch_flex_attn_availableis_torchdynamo_compilingloggingtorch_compilable_check)merge_with_config_defaults)OutputRecordercapture_outputs) MBartConfig input_ids pad_token_idcf|j}| td|j|dk(||j|j ddz j d}|j d|j}|ddddfj|ddddf<||dddf<|S)z Shift input ids one token to the right, and wrap the last non pad token (the token) Note that MBart does not have a single `decoder_start_token_id` in contrast to other Bart-like models. Nz1self.model.config.pad_token_id has to be defined.ir'dimr)clone ValueError masked_fill_nesum unsqueezegathersqueeze)r)r*prev_output_tokens index_of_eosdecoder_start_tokenss s/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/mbart/modeling_mbart.pyshift_tokens_rightr;@s #*LMM##$6$$> M&)),7;;;BQFQQRTUL-44Q EMMO 21crc6 : @ @ Bq!"u3q!t cveZdZdZdedeffd Z d dejdedejdzffd ZxZ S) MBartLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcNd|_t| ||jz|yN)offsetsuper__init__)selfr?r@ __class__s r:rFz(MBartLearnedPositionalEmbedding.__init__Zs$  $++5}Er<Nr)past_key_values_length position_idsc$|a|jdd\}}tj|||ztj|jj j |d}n|jd}t|%||jzS)z3`input_ids' shape is expected to be [bsz x seqlen].NrC)dtypedevicer.r) shapetorcharangelongweightrMexpandr4rEforwardrD)rGr)rIrJbszseq_lenrHs r:rTz'MBartLearnedPositionalEmbedding.forward`s  $??2A.LC <<&(>(HPUPZPZcgcncncucufS"o (11!4Lw|dkk9::r<)rN) __name__ __module__ __qualname____doc__intrFrOTensorrT __classcell__rHs@r:r>r>UsWFsF3Fmq ; ;?B ;V[VbVbeiVi ; ;r<r>c `eZdZdZd dededededzffd Zdejffd Z xZ S) MBartScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. r?r@ padding_idx embed_scaleNc6t||||||_yN)rErFrb)rGr?r@rarbrHs r:rFz!MBartScaledWordEmbedding.__init__vs  D&r<r)c<t|||jzSrd)rErTrb)rGr)rHs r:rTz MBartScaledWordEmbedding.forwardzswy)D,<,<<L==((6??([L,,|U3K''1-88:K  $$r<ceZdZdZ ddedededededed edzd edzffd Z dd e jd e jdzde dzde jdzde e dee je jdzff dZxZS)MBartAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsro is_decoderbias is_causalconfig layer_idxc t |||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ ||_ |9|jr-tjd|jjdt!j"||||_t!j"||||_t!j"||||_t!j"||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rrzInstantiating a decoder z without passing `layer_idx` is not recommended and will lead to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.r)rErFrrrohead_dimrr0rnrrrlogger warning_oncerHrWrLineark_projv_projq_projout_proj) rGrrrorrrrrrHs r:rFzMBartAttention.__init__s$ "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $""     *4>>+B+B*CD,,  ii 94@ ii 94@ ii 94@  )YTB r< hidden_stateskey_value_statespast_key_valuesrmrpreturnc |du}|jdd}g|d|j}|j|j|j dd} d} |St |t rA|jj|j} |r |j} n|j} n|} |r|n|} |rK|I| rG j|jj} | j|jj}n|j| } |j!| }g| jddd|j}| j|j dd} |j|j dd}|K j#| ||j\} }|r)t |t rd|j|j<t%j&|j(j*t,}||| | ||f|j.sdn |j0|j2d|\}}|j4g|dj7}|j9|}||fS) z#Input shape: Batch x Time x ChannelNr.r'rCFT)rorn)rNrrviewrx isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdater get_interfacer_attn_implementationr~rurornreshaper{r)rGrrrrmrpis_cross_attention input_shape hidden_shape query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapeattention_interfacer}r|s r:rTzMBartAttention.forwardsd.T9$))#2. 88b8$--8 {{=166|DNNqRST   &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML^4J;;~6LF--cr2FBF FH#2<(?(M(M KK , ,.E) %8      % $}}C$,,LL %  % ! \*k));;;;FFH mmK0 L((r<)rFTFNNNNN)rWrXrYrZr[rhboolr(rFrOr\r rrtuplerTr]r^s@r:rrsG  %) $%C%C%C %C  %C  %C%Cd"%C:%CT15(,.2 H)||H) ,,-H) H)  t+ H)-.H) u||U\\D00 1H)r<rc~eZdZdeffd Zdej dej deedej fdZ xZ S)MBartEncoderLayerrcht||j|_t |j|j |j ||_tj|j|_ |j|_ t|j|_|j|_tj |j|j"|_tj |j"|j|_tj|j|_y)N)rrror)rErFd_modelrrencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normror activation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normrGrrHs r:rFzMBartEncoderLayer.__init__s 'nn44,,   %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r<rrmrprc |}|j|}|jd||d|\}}tjj ||j|j }||z}|}|j |}|j|j|}tjj ||j|j }|j|}tjj ||j|j }||z}|jtjk(rEtj|jjdz }tj || |}|S)a> Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. )rrmrsi)minmax)rrrryrorurrrrrrLrOfloat16finforclamp)rGrrmrpresidual_ clamp_values r:rTzMBartEncoderLayer.forward$sT!11-@ )4>> ')   q  --mt||VZVcVc-d  =0  --m< **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0   %-- /++m&9&9:>>EK!KK K<[YMr<) rWrXrYr(rFrOr\rrrTr]r^s@r:rrsL={=$"||" "+, "  "r<rceZdZddededzffd Z ddejdejdzdejdzdejdzd edzd e dzd e e d ejfd Z xZ S)MBartDecoderLayerNrrc t||j|_t |j|j |j dd|||_|j|_t|j|_ |j|_ tj|j|_t |j|j |j d|||_tj|j|_tj$|j|j&|_tj$|j&|j|_tj|j|_y)NT)rrrorrrr)rorrr)rErFrrrdecoder_attention_headsrrror rrrrrr encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrrr)rGrrrHs r:rFzMBartDecoderLayer.__init__Js  'nn44,, ~~ #F$>$>?"(";";$&LL$@!* NN  * *,,  (*||DNN'C$99T^^V-C-CD99V33T^^D " T^^ <r<rrmencoder_hidden_statesencoder_attention_maskr use_cacherprc |}|j|}|jd|||d|\}} tjj ||j|j }||z}|h|}|j |}|jd||||d|\}} tjj ||j|j }||z}|}|j|}|j|j|}tjj ||j|j }|j|}tjj ||j|j }||z}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states )rrrmrs)rrrmrr) rrrryrorurrrrrrr) rGrrmrrrrrprrs r:rTzMBartDecoderLayer.forwardis*!11-@ *4>> '+)    q  --mt||VZVcVc-d  =0  ! ,$H 88GM0t00 +!65 /      M1MM11-4<embed_positionsr ModuleListrangeencoder_layersrrrrlayernorm_embedding layer_normgradient_checkpointing post_init)rGrrrbrrHs r:rFzMBartEncoder.__init__s  ~~ 11NN !..$*$B$B!.4.D.Ddii *# 4   y$*:*:   ?  * *  mmfNcNcHd$e1%6v%>$ef  #%<< #: ,,v~~6&+# %fs-E(cn|jr)t|jddr|jyyy)NrF)rgetattrrgradient_checkpointing_enablerGs r:._backward_compatibility_gradient_checkpointingz;MBartEncoder._backward_compatibility_gradient_checkpointings1  / /GDKKIach4i  . . 05j /r<Nr)rm inputs_embedsrprc h|du|duz r td||j|}|j|d}||j|jz}|j |}t jj||j|j}t|j||}t|jD]F\}}d} |jr&tjg} | |j krd} | r=|||fi|}H|j#|}t%|S) a  Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embeds).r.rs)rrrmFT)last_hidden_state)r0rrtorMrrryrorurr enumeraterrOrandrrr) rGr)rmrrp embed_posridx encoder_layerto_dropdropout_probabilitys r:rTzMBartEncoder.forward s5>  -t"; <YZ Z   --i8M((w)?@ % ]5I5I(JJ 00?  --mt||VZVcVc-d 2;;')  #,DKK"8  CG}}&+jjn#&7"G -!"!!   6 ??r<r)rWrXrYrZrr%r_can_record_outputsr(rFrr$r&rO LongTensorr\ FloatTensorrrrrrTr]r^s@r:rrs+$^1U {81  .2.226 @@##d*@@ t+@@((4/ @@ +, @@  @@ @@r<rcBeZdZdZeeeddeedddZdeffd Z e e dd e jd zd e jd zd e jd zd e jd zded zde jd zded zdeedeezfdZxZS) MBartDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MBartDecoderLayer`] Args: config: MBartConfig embed_tokens (nn.Embedding): output embedding r'rrr)rrcross_attentionsrc t|||j|_|j|_|j |_|j|_|jrtj|jnd}t|j|j|j ||_t!|j|j|_t%j&t)|j*Dcgc]}t-||c}|_||_t%j2|j|_t%j2|j|_d|_|j;ycc}w)Nrfr)rF)rErFrodecoder_layerdroprr*rar max_target_positionsr r r rr`rrr>rrrrdecoder_layersrrrrrrrr)rGrrbirHs r:rFzMBartDecoder.__init__^s   ~~ 11!..$*$B$B!393I3Idii/s 4   v~~t/?/?[  ?  * * NN mmUZ[a[p[pUq$rPQ%6v%K$rs  #%<<#? ,,v~~6&+# %ss?FNr)rmrrrrrrprc |du|duz r td||j|}|rd|b||jjr4t t |jt |jnt |j}|j dd\} } ||jnd} tj| |j| z} |1ts'| | z} tj| | |j}t|tr |jn|}t|j|||}t!|j|||}|j#|| | } || j%|jz}|j'|}t(j*j-||j,|j. }t1|j2D]E\}}|j.r%tj4g}||j6kr7||||f|||d |}G|j9|}t;|| S) a( Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time)rr.rr)rrrmr)rrrmr)rJrs)rrr)rr)r0rris_encoder_decoderr r rvget_seq_lengthrOrPrMr!onesrrrrrr rrryrorur!rr"rrr)rGr)rmrrrrrrp batch_size seq_lengthrIrJmask_seq_lengthself_attn_cache causal_maskrr$ decoder_layerr's r:rTzMBartDecoder.forwardxsZn  -t"; <st t   --i8M 0)4 8V8V$L $DlZ^ZeZeFfg!5  "/!3!3!5cr!: JETE`!?!?!Afg||J}7K7KLOee  !*B*D4zAO"ZZ OML`L`aN/+>?  0 0   );;')+  ";;;'1"7 " ++I7M\h+i %  8L8L(MM 00?  --mt||VZVcVc-d "+DKK"8  C}}&+jjn#&7)%(> /# M " 6 8++  r<)NNNNNNN)rWrXrYrZrr%rr(r(rFr$r&rOr)r\r*r rrrrrrTr]r^s@r:r,r,Os+$^1U*>~^ {4 .2.2:>:>(,26!%} ##d*}  t+}  %0047 } !& 0 04 7 }  } ((4/} $;} +,}  : :}  } r<r,ceZdZdddZdeffd ZdZdZee dde jdzd e jdzd e jdzd e jdzd e e e jdzd edzde jdzde jdzdedzdedzdeedee e jzfdZxZS) MBartModelz shared.weight)zdecoder.embed_tokens.weightzencoder.embed_tokens.weightrcJt|||j|j}}|jrt j |jnd}t||j|||_ t||_ t||_ |jy)Nrfr)rErFr*rr r r rr`sharedrencoderr,decoderr)rGrrarrbrHs r:rFzMBartModel.__init__s|  "("5"5v7H7HZ 393I3Idii/s .z6>>;dop #F+ #F+  r<c|jSrd)r@rs r:get_input_embeddingszMBartModel.get_input_embeddingss {{r<c~||_|j|j_|j|j_yrd)r@rArrBrGrls r:set_input_embeddingszMBartModel.set_input_embeddingss) $(KK !$(KK !r<Nr)rmdecoder_input_idsdecoder_attention_maskencoder_outputsrrdecoder_inputs_embedsr return_dictrprc T| | n|jj} |"| t||jj}||jd |||| d| }nI| rGt |t s7t |dt|dkDr|dndt|dkDr|dnd}|jd |||d|||| | d| } | s| |zSt| j| j| j| j| j|j|j|jS) a5 decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) MBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 25004 for *en_XX*, and 25003 for *de_DE*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. N)r)rmrrLrr'rC)rrr)r)rmrrrrrrL)rrdecoder_hidden_statesdecoder_attentionsr-encoder_last_hidden_staterencoder_attentionsr)rrLr;r*rArrlenrBrrrrrr-) rGr)rmrHrIrJrrrKrrLrpdecoder_outputss r:rTzMBartModel.forwardsaJ&1%)F 29dkk>V>V W   "*dll#-+'   OO_!M-"1!"4474H14Loa0RV14_1E1I?1-tO'$,,  '1"1!"4#1+/#    "_4 4!-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r< NNNNNNNNNN)rWrXrY_tied_weights_keysr(rFrDrGrrrOr)r\rr*r rrrrrTr]r^s@r:r>r>s`(7'6 { 0 .2.259:>BF(,26:>!%#'S ##d*S  t+S !++d2 S !& 0 04 7 S uU%6%6784? S S ((4/S  %0047S $;S D[S +,S  eE$5$56 6S S r<r>z The MBART Model with a language modeling head. Can be used for summarization, after fine-tuning the pretrained models. ) custom_introceZdZdZdgZddiZdeffd Z dded edzd e d e jffd Z ded dfd Z e ddej dzdej"dzdej dzdej dzdeeej&dzdedzdej&dzdej&dzdej dzde dzde dzdeed eeej&zfdZdej"fdZxZS)rrrlm_head.weightzmodel.shared.weightrcxt||t||_|j dt j d|jjjftj|j|jjjd|_ |jy)Nrr'Fr)rErFr>rregister_bufferrOzerosr@r?rrrlm_headrrs r:rFz&MBartForConditionalGeneration.__init__xs  '  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  r<Nnew_num_tokenspad_to_multiple_of mean_resizingrczt||||}|j|jjd|S)Nr)rEresize_token_embeddings_resize_final_logits_biasrRrN)rGr]r^r_new_embeddingsrHs r:raz5MBartForConditionalGeneration.resize_token_embeddingss?8I[]jk &&~'<'<'B'B1'EFr<c6|jjd}||kr|jddd|f}nStjd||z f|jj}tj |j|gd}|j d|y)Nr.r'rr,r)rrNrOr[rMcatrZ)rGr]old_num_tokensnew_bias extra_biass r:rbz7MBartForConditionalGeneration._resize_final_logits_biass//55b9 ^ +--a..@AHa.)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;r<r)rmrHrIrJrrrKlabelsrrLrpc | | n|jj} | =| rtjdd} |"| t | |jj }|j |f|||||||| | d | } |j| d|jz}d}| Ft}||jd|jj| jd}| s|f| ddz}||f|zS|St||| j| j| j| j | j"| j$| j& S) u6 decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) MBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 25004 for *en_XX*, and 25003 for *de_DE*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Translation: ```python >>> from transformers import AutoTokenizer, MBartForConditionalGeneration >>> model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-en-ro") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-en-ro") >>> example_english_phrase = "42 is the answer" >>> inputs = tokenizer(example_english_phrase, return_tensors="pt") >>> # Translate >>> generated_ids = model.generate(**inputs, num_beams=4, max_length=5) >>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] '42 este răspuns' ``` Mask filling example: ```python >>> from transformers import AutoTokenizer, MBartForConditionalGeneration >>> model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25") >>> # de_DE is the language symbol id for German >>> TXT = " Meine Freunde sind nett aber sie essen zu viel Kuchen. de_DE" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt")["input_ids"] >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['nett', 'sehr', 'ganz', 'nicht', 'so'] ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F) rmrHrJrIrrrKrrLrr.r' losslogitsrrNrOr-rPrrQ)rrLrwarningr;r*rr\rrrrrrrNrOr-rPrrQ)rGr)rmrHrIrJrrrKrirrLrpoutputs lm_logitsmasked_lm_lossloss_fctoutputs r:rTz%MBartForConditionalGeneration.forwardsv`&1%BF(,26:>*.!%#'z ##d*z  t+z !++d2 z !& 0 04 7 z uU%6%6784? z z ((4/z  %0047z   4'z $;z D[z +,z  5!2!23 3z z xDELLDr<rz MBart model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cheZdZdeffd Zee ddejdzdejdzdejdzdejdzde ejdzd ejdzd ejdzd ejdzd e dzd e edeezfdZxZS)MBartForSequenceClassificationrc t||fi|t||_t |j |j |j |j|_|jyrd) rErFr>rrr num_labelsclassifier_dropoutclassification_headr)rGrrprHs r:rFz'MBartForSequenceClassification.__init__sZ *6*' #: NN NN     % % $   r<Nr)rmrHrIrJrrKrirrprc |d} |$|"td|jj|j|f||||||| d| } | d} |j |j j j| j} ttj| jdjdk(d| | ddf}t|jd| jdzdk\d|j| j!dd | j!d ddd ddf}|j#|}d}||j|j}|j j$|j j&dk(rd |j _nv|j j&dkDrL|j(tj*k(s|j(tj,k(rd |j _nd |j _|j j$d k(rSt/}|j j&dk(r&||j1|j1}n|||}n|j j$d k(rGt3}||jd |j j&|jd }n,|j j$d k(rt5}|||}t7||| j8| j:| j<| j>| j@| jB| jD S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for rmrHrIrJrrKrrr'z7All examples must have the same number of tokens.z3Each example must contain at least one token.r. regressionsingle_label_classificationmulti_label_classificationrk)#NotImplementedErrorrHrWreqr eos_token_idr rMr#rOunique_consecutiver3numelrNrrvr} problem_typer{rLrQr[rr6rrrrrNrOr-rPrrQ)rGr)rmrHrIrJrrKrirrproreos_maskselectedsentence_representationrmrlrrs r:rTz&MBartForSequenceClassification.forward&s T  I  !:%J4>>KbKbJcd '1djj  ' )/#9+'"7 '  '   << 8 89<<]=Q=QR  $ $X\\!_ 5 ; ; = B E !1- NN1 !4!4Q!7 71 < A #+-- 0B0B10Er=K]K]^`Ka"bcdfhjkck"l))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./.#33")"?"?&99$55&-&G&G")"?"?&99  r<) NNNNNNNNN)rWrXrYr(rFrrrOr)r\listr*rrrrrrTr]r^s@r:ryrys, { .2.259:>:>26:>*.!%l ##d*l  t+l !++d2 l !& 0 04 7 l e//047 l ((4/l  %0047l   4'l $;l +,l  0 0l l r<ryceZdZfdZee ddejdzdejdzdejdzdejdzde ejdzdejdzd ejdzd ejdzd ejdzd e dzd e e deezfdZxZS)MBartForQuestionAnsweringct||d|_|j|_t||_t j |j|j|_|jyrB) rErFr{r>rrr hidden_size qa_outputsrrs r:rFz"MBartForQuestionAnswering.__init__s[   ++' ))F$6$68I8IJ r<Nr)rmrHrIrJstart_positions end_positionsrrKrrprc D||d} |j|f|||||| | d| } | d} |j| }|jdd\}}|jdj }|jdj }d}||t |j dkDr|jd}t |j dkDr|jd}|j d}|jd|}|jd|}t|}|||}|||}||zd z }t|||| j| j| j| j| j| j| j S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. NFrrr'r.r,) ignore_indexrC) rl start_logits end_logitsrrNrOr-rPrrQ)rrsplitr6r{rRrvrrrrrNrOr-rPrrQ)rGr)rmrHrIrJrrrrKrrprosequence_outputrmrr total_loss ignored_indexrr start_lossend_losss r:rTz!MBartForQuestionAnswering.forwardsP  &=+DI&0djj  ' )/#9+'"7 '  ' "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J2%!#33")"?"?&99$55&-&G&G")"?"?&99  r<rT)rWrXrYrFrrrOr\r)rr*rrrrrrTr]r^s@r:rrs< *..259:>:>371526:>!%W <<$&W  t+W !++d2 W !& 0 04 7 W e//047 W ))D0W ''$.W ((4/W  %0047W $;W +,W  4 4W W r<rc(eZdZdZfdZdZxZS)MBartDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cdt||t||_|j yrd)rErFr,rBrrs r:rFzMBartDecoderWrapper.__init__ s&  #F+  r<c&|j|i|Srd)rB)rGargsrps r:rTzMBartDecoderWrapper.forwardst||T,V,,r<)rWrXrYrZrFrTr]r^s@r:rrs  -r<rc\eZdZddiZfdZdZdZee dde jdzde jdzd e jdzd e jdzd e dzd e jdzd e jdzdedzdee jzdeedeezfdZxZS)MBartForCausalLMrXz!model.decoder.embed_tokens.weightcd|_d|_t| |t ||_t j|j|jd|_ |jy)NTFr) rr4rErFrrrrrrr\rrs r:rFzMBartForCausalLM.__init__sX $)!  (0 yy!3!3V5F5FUS  r<cB|jjjSrdrrBrrs r:rDz%MBartForCausalLM.get_input_embeddings$szz!!...r<c:||jj_yrdrrFs r:rGz%MBartForCausalLM.set_input_embeddings's*/ 'r<Nr)rmrrrrrirlogits_to_keeprprc |jjd|||||||d| } | d} t| tr t | dn| } |j | dd| ddf}d}|a|j |j}t}||jd|jj|jd}t||| j| j| j| j S)aP labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25") >>> model = MBartForCausalLM.from_pretrained("facebook/mbart-large-cc25") >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```)r)rmrrrrrrNr.)rlrmrrrr-r)rrBrr[slicer\r rMrrrrrrrrr-)rGr)rmrrrrrirrrpror slice_indicesrmrlrrs r:rTzMBartForCausalLM.forward*sL>PTZZ=O=O > )"7#9+' >  >   8B>SV8W~ot4]k mA}a,?@A  YYv}}-F')HFKKDKK,B,BCV[[QS_UD0#33!//))$55   r<) NNNNNNNNr)rWrXrYrUrFrDrGrrrOr)r\r*r rr[rrrrrTr]r^s@r:rrs/= /0.2.2:>;?(,26*.!%-.A ##d*A  t+A  %0047 A !& 1 1D 8 A  A ((4/A   4'A $;A ell*A +,A  2 2A A r<r)rrrryr>r)Nr)RrZr collections.abcrrOrtorch.nnrrrr r activationsr cache_utilsr r r generationr masking_utilsrrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr r!r"r# utils.genericr$utils.output_capturingr%r&configuration_mbartr( get_loggerrWrr\r[r;rwr>r`Modulerhr~rrrrrrr,r>rryrrr__all__rr<r:rsy $ AA&!CC)J:G&8E, !   H %%,,c*;bll;8 =r|| =(!% II% <<% % << % LL4' % T\ %%'(%:r)RYYr)j525pZ2Z|bii0?4r@'r@jh 'h Vp %p p f \D$8/\D  \D~ } %9}  } @g 4g g V -. - Y +_Y x r<