^j dZddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl mZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZmZdd l m!Z!ddl"m#Z#m$Z$m%Z%ddl&m'Z'e$jPe)Z*dejVde,de,fdZ-Gddej\Z/Gddej`Z1GddeZ2GddeZ3Gddej`Z4Gdd ej`Z5e#Gd!d"e!Z6Gd#d$e6Z7Gd%d&e6Z8e#Gd'd(e6Z9e#d)*Gd+d,e6eZ:e#d-*Gd.d/e6Z;e#Gd0d1e6Z<Gd2d3e6Z=Gd4d5e6eZ>gd6Z?y)7zPyTorch MVP model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)PreTrainedModel)auto_docstringloggingtorch_compilable_check) MvpConfig input_ids pad_token_iddecoder_start_token_idc|j|j}|ddddfj|ddddf<||dddf<| td|j |dk(||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrr shifted_input_idss o/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/mvp/modeling_mvp.pyshift_tokens_rightr*.s}"++IOO<(CRC0668ae4adLMM""#4#(HPUPZPZcgcncncucufS"o (11!4Lw|dkk9::r+)rN) __name__ __module__ __qualname____doc__intr5r=TensorrC __classcell__r7s@r)r-r-?sWFsF3Fmq ; ;?B ;V[VbVbeiVi ; ;r+r-cPeZdZdZ ddedededzdedzdedzdedzf fd Z dd ejd ejdzd e dzd ejdzdejdzdede ejejdze ejdzffdZ xZ S) MvpAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsdropout is_decoderbias layer_idxct|||_||_||_||z|_|j |z|jk7rt d|jd|d|j dz|_||_||_ tj||||_ tj||||_ tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rT)r4r5rPrQrRhead_dimr&scalingrSrUrLineark_projv_projq_projout_proj)r6rPrQrRrSrTrUr7s r)r5zMvpAttention.__init__]s "" !Y. MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r+ hidden_stateskey_value_statespast_key_valuesattention_mask attn_promptoutput_attentionsreturnc  |du}|j\} } } |j||jz} d} |St|trA|j j |j} |r |j}n|j}n|}|r|n|}|rK|I| rGj|jj}|j|jj}n|j|}|j|}|j| d|j |j"j%dd}|j| d|j |j"j%dd}|Kj'|||j\}}|r)t|trd|j |j<|t)j*|dj-| ddd|gd}t)j*|dj-| ddd|gd}|\t)j.| d| |djdj1|j2}t)j*||gd}| |j zd|j"f}| j| | |j |j"j%dd} | j4|} |j4|}|j4|}|jd}t)j6| |j%dd}|j| |j z| |fk7r/t9d | |j z| |fd |j|{|j| d| |fk7r#t9d | d| |fd |j|j| |j | ||z}|j| |j z| |}t:j<j?|d}|r?|j| |j | |}|j| |j z| |}nd}t:j<jA||j@|jB }t)j6||}|j| |j z| |j"fk7r7t9d | |j | |j"fd |j|j| |j | |j"}|j%dd}|j5| | |jD}|jG|}||fS)z#Input shape: Batch x Time x ChannelNFr"rr2Trdimz$Attention weights should be of size z , but is z!Attention mask should be of size ptrainingz `attn_output` should be of size )$sizer]rY isinstancer is_updatedgetrUcross_attention_cacheself_attention_cachelayerskeysvaluesr[r\viewrQrX transposeupdater=catrAzerostor<reshapebmmr&r functionalsoftmaxrRrkrPr^)r6r_r`rarbrcrdkwargsis_cross_attentionrDtgt_len_ query_statesrncurr_past_key_valuescurrent_states key_states value_states prompt_mask proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r)rCzMvpAttention.forwardzs;.T9',,.Wa{{=1DLL@   &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML^4J;;~6L#b$..$--PZZ[\^_`J',,S"dnndmmT^^_`bcdL*+?+F+FzS_aeaoao+p( L%*_FY*ZAEO..t~~>  "KN$9$9#r2r$JJ#W]^_J 99k!n&;&;CR&Ll%[abcL)#kk#q';q>;N;Nq;QRUUVdVkVkl !&K+Hr!SDNN*B > #((gt~~t}}U__`acde +|++Z8 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r+r_rbself_attn_promptrdNrecJ|}|j||||\}}tjj||j|j}||z}|j |}|}|j |j|}tjj||j|j}|j|}tjj||j|j}||z}|j|}|jtjk(rhtj|jsEtj |jj"dz }tj$|| |}||fS)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, encoder_attention_heads, pro_len, head_dim)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r_rbrcrdrii)minmax)rrr}rRrkrrrrrrr;r=float16isfiniteallfinforclamp)r6r_rbrrdresidualr clamp_values r)rCzMvpEncoderLayer.forwardsg$!&*nn')(/ '5' # |  --mt||VZVcVc-d  =0 11-@  **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0 --m<   %-- /}8U8Y8Y8[++m&9&9:>>EK!KK K<[YMl**r+F) rFrGrHrr5r= FloatTensorrrrCrLrMs@r)rrsz=y=**/ )+(()+)))+ ++ )+  $; )+ u  %"3"3d":: ; )+r+rcfeZdZddeffd Z ddej dej dzdej dzdej dzdej dzd ej dzd edzd edzd edzd e eje ejejfdzffdZ xZ S)MvpDecoderLayerNrct||j|_t |j|j |j d||_|j|_t|j|_ |j|_ tj|j|_t |j|j |j d||_tj|j|_tj$|j|j&|_tj$|j&|j|_tj|j|_y)NT)rPrQrRrSrU)rRrSrU)r4r5rrPrOdecoder_attention_headsrrrRr rrrrrr encoder_attnencoder_attn_layer_normrZdecoder_ffn_dimrrr)r6rrUr7s r)r5zMvpDecoderLayer.__init__+s %nn44,,  ~~ #F$>$>?"(";";$&LL$@!( NN  * *,,  (*||DNN'C$99T^^V-C-CD99V33T^^D " T^^ <r+r_rbencoder_hidden_statesencoder_attention_maskrcross_attn_promptrard use_cacherec  .|} |j|||||\}} tjj||j|j}| |z}|j |}d} |h|} |j ||||||\}} tjj||j|j}| |z}|j|}|} |j|j|}tjj||j|j}|j|}tjj||j|j}| |z}|j|}|f}|r|| | fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. cross_attn_prompt (`torch.FloatTensor`): prompt of cross attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r_rarbrcrdriN)r_r`rbrcrard) rrr}rRrkrrrrrrrr)r6r_rbrrrrrardrrrself_attn_weightscross_attn_weightsoutputss r)rCzMvpDecoderLayer.forwardGs>!,0>>'+)(/ ,:, ( ( --mt||VZVcVc-d  =0 11-@ " ,$H040A0A+!65- /"3 1B1 -M-MM11-4< >Gr+N)NNNNNNFT) rFrGrHrr5r=rKr rrrrCrLrMs@r)rr*s=y=>/3596:0415(,).!%L||L t+L %||d2 L !& t 3 L  ,,- L!<<$.LL $;L$;L u  %(9(95;L;L(L"MPT"TT ULr+rcleZdZdZdedededeffd Zdejdejfd Z xZ S) MvpClassificationHeadz-Head for sentence-level classification tasks. input_dim inner_dim num_classespooler_dropoutct|tj|||_tj ||_tj|||_y)Nrj)r4r5rrZdenseDropoutrRr^)r6rrrrr7s r)r5zMvpClassificationHead.__init__sD YYy)4 zzN3  )[9 r+r_rec|j|}|j|}tj|}|j|}|j |}|Sr)rRrr=tanhr^)r6r_s r)rCzMvpClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r+) rFrGrHrIrJrr5r=rKrCrLrMs@r)rrsL7 : : : :  :U\\ellr+rc`eZdZdZfdZdej deej fdZxZ S) MvpPromptz)Layer-wise prompt for encoder or decoder.c 8t||j|_||_||_|j |z|_tj|j|_ tj|j|j |_ tjtj|j |jtjtj|j|dz|j z|_y)Nrr2)r4r5 prompt_length num_layersrQrrXrrrR Embeddingprompt_embedding SequentialrZprompt_mid_dimGELU prompt_trans)r6rrrQr7s r)r5zMvpPrompt.__init__s #11$")3 zzFNN3 " V-A-A6>> RMM IIfnnf&;&; < GGI IIf++Z!^fnn-L M r+ prompt_idsrec*|j|j|}|j|j|jdz|j |j }|j|}|jgdjd}|S)Nr2)rr2rr) rrrurrrQrXrRpermutesplit)r6rprompts r)rCzMvpPrompt.forwardsw""4#8#8#DET//11DdnnVZVcVcdf% -33A6 r+) rFrGrHrIr5r=rKrrCrLrMs@r)rrs+3  %,,53Fr+rcBeZdZUeed<dZdZfdZedZ xZ S)MvpPreTrainedModelrmodelTct||t|tr t j |j yyr)r4 _init_weightsrmMvpForConditionalGenerationinitzeros_final_logits_bias)r6moduler7s r)rz MvpPreTrainedModel._init_weightss2 f% f9 : KK00 1 ;r+c|jj}tjgddddd|gg|j}|j ||d}|S)N)r r2r r2r<)rbr)rrr=tensorr<ne)r6 pad_tokenr dummy_inputss r)rzMvpPreTrainedModel.dummy_inputssWKK,, LL"2Q2q)4L!MVZVaVab 'll95" r+) rFrGrHr__annotations__base_model_prefixsupports_gradient_checkpointingrpropertyrrLrMs@r)rrs. &*#2 r+rceZdZdZddedej dzdedzffd Z dde jdzde jdzd e jdzd edzd edzd edzd e ezfdZxZS) MvpEncodera Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`MvpEncoderLayer`]. Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt Nr embed_tokens use_promptct|||j|_|j|_|j }|j |_|j|_ |jrtj|nd|_ tj|j ||j|_t%|j||_tj(t+|j,Dcgc] }t/|c}|_tj2||_||_|r7|j8|_t;||j,|j<|_d|_ |jCycc}w)N?F)"r4r5rRencoder_layerdrop layerdroprr padding_idxmax_position_embeddingsmax_source_positionsscale_embeddingmathsqrt embed_scalerr vocab_sizerr-embed_positions ModuleListrangeencoder_layersrrrrlayernorm_embeddingrrrrrgradient_checkpointing post_init)r6rrrrPrr7s r)r5zMvpEncoder.__init__s4  ~~ 11NN !..$*$B$B!393I3I499Y/sLL):):ItGWGWX<  * *  mmeFLaLaFb$c_V%<$cd #%<< #: $ !'!5!5D $-%%..%D ! ',# %ds:F rrb inputs_embedsrdoutput_hidden_states return_dictrec ||n|jj}||n|jj}||n|jj}| | t d|$|}|j } |j d| d}n-| |jdd} |dddddf}n t d||j||jz}|j|} || z} |j| } tjj| |j|j} |j rIt#j$|j&j)|j*} |j-| } |t/|j| |}|rdnd}|rdnd}t1|j2D]t\}}|r|| fz}d}|jr&t#j4g}||j6krd }|rd }n"|| ||j r |nd| }|d } |sl||d fz}v|r|| fz}|st9d| ||fDSt;| ||S)a8 Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timer"z5You have to specify either input_ids or inputs_embedsri)rr rbFT)NN)rrdrrc3&K|] }|| ywrr.0vs r) z%MvpEncoder.forward..{seqWXWdeslast_hidden_stater_ attentions)rrdrrr&r$rurlrrrr rr}rRrkrr=r>rrzr<rr enumeraterrrandrrr)r6rrbr rdrrrinput input_shape embed_posr_rrencoder_statesall_attentionsidx encoder_layerto_dropdropout_probability layer_outputss r)rCzMvpEncoder.forward sP2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "E++K!r;r?;I  &',,.s3K!!Q(+ETU U   --i84;K;KKM((/ % 1 00?  --mt||VZVcVc-d  ??d&8&89<>Ne]NN$See e+>Vd  r+NF)NNNNNN)rFrGrHrIrrrrr5r= LongTensorrKrrrrCrLrMs@r)rrsy t8K`dgk`kF.2.226)-,0#'t ##d*t  t+t ((4/ t  $; t #Tk t D[t   t r+rc$eZdZdZddededzffd Z ddejdzdejdzdejdzd ejdzd e dzd ejdzd edzd edzdedzdedzde e zfdZxZS) MvpDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MvpDecoderLayer`] Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt rrNc t|||j|_|j|_|j |_|j|_|jrtj|jnd|_ tj|j |j|j |_t%|j|j|_tj(t+|j,Dcgc]}t/||c}|_tj2|j|_||_|r]|j8|_t;||j,|j<|_t;||j,|j<|_ d|_!|jEycc}w)Nr)rUF)#r4r5rRdecoder_layerdroprrrrmax_target_positionsrrrrrrrrrr-rrrdecoder_layersrrrrr rrrrrrr r )r6rrir7s r)r5zMvpDecoder.__init__sd  ~~ 11!..$*$B$B!8>8N8N499V^^4TWLL):):FNNDL\L\]<  * * NN mmSXY_YnYnSo$pa_Vq%I$pq #%<<#? $ !'!5!5D $-%%..%D ! &/%%..&D " ',# '%qs Grrbrrrar rrdrrrec ||n|jj}| | n|jj} ||n|jj}| | n|jj} | | t d|$|} |j } |jd| d}n-| |jdd} |dddddf} n t d||j||jz}|jr%|jr|rtjdd}|rd|b||jjr4t!t#|jt#|jnt#|j}||j%nd}t'|j||| }||t)|j||| }|j+| |}||z}|j-|}t.j0j3||j2|j }|j4rZt7j8|j:j=|j>}|jA|}|jC|}| rd nd}|rd nd}|r|d nd}tE|jFD]\}}| r||fz }|jr%t7jHg}||jJkr?||||||j4r|nd|j4r|nd||| }|d}|sz||dfz }|||dfz }| r||fz }| stMd|||||fDStO|||||S)aU Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer"zEYou have to specify either decoder_input_ids or decoder_inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rr)rr rbra)rr rbrrir)rrrrardrrr2c3$K|]}|| ywrrrs r)rz%MvpDecoder.forward..Ws=s)rrar_rcross_attentions)(rrdrrrr&r$rurlrrr rklogger warning_onceis_encoder_decoderr r get_seq_lengthrrrr rr}rRrr=r>rrzr<rrrrrrrrr)r6rrbrrrar rrdrrrrrr8 positionsr_rrrall_hidden_statesall_self_attnsall_cross_attentionsr! decoder_layerr$r%s r)rCzMvpDecoder.forwards@2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKst t  "E#//K!r;r?;I  &',,.s3K!!Q(+Ede e   --i84;K;KKM  & &4==##p" 0)4 8V8V$L $DlZ^ZeZeFfg!5  FUE`!?!?!Afg+;;')+   ! ,1G1S%>{{+5&; & "((0FG % 1 00?  --mt||VZVcVc-d  ??d&8&89<:>(,26!%)-,0#'u ##d*u  t+u  %0047 u !& 0 04 7 u  u ((4/u $;u  $;u #Tku D[u  : :u r+r)ceZdZdgZdddZdeffd ZdZdZdZ e dd e jd zd e jd zd e jd zd e jd zdee jd zded zde jd zde jd zded zded zded zded zdeezfdZxZS)MvpModelrz shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightrcNt|||j|j}}|j|_t j ||j||_t||j|_ t||j|_ |jyr)r4r5rrrrrrsharedrencoderr)decoderr )r6rrrr7s r)r5zMvpModel.__init__ms~  "("5"5v7H7HZ  ++ll:v~~{K !&&*;*;< !&&*;*;<  r+c|jSr)r>r6s r)get_input_embeddingszMvpModel.get_input_embeddingszs {{r+c~||_|j|j_|j|j_yr)r>r?rr@r6values r)set_input_embeddingszMvpModel.set_input_embeddings}s) $(KK !$(KK !r+c*|jsJd|jd|jjjd|jjjd|jj jdy)NzHIf you want to use lightweight tuning, make sure that `use_prompt=True`.FT)rrequires_grad_r?rr@rrBs r)set_lightweight_tuningzMvpModel.set_lightweight_tuningsjj jj E" %%44T: %%44T: &&55d;r+Nrrbdecoder_input_idsdecoder_attention_maskencoder_outputsrar decoder_inputs_embedsrrdrrrec 4|D|B| tdt||jj|jj}| | n|jj } | | n|jj } | | n|jj} | | n|jj} ||j|||| | | }nI| rGt|ts7t|dt|dkDr|dndt|dkDr|dnd}|j|||d|||| | | |  }| s||zSt|j|j |j"|j$|j&|j|j"|j$ S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. NzIf no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.)rrbr rdrrrrr2r rrbrrrar rrdrr)rradecoder_hidden_statesdecoder_attentionsr1encoder_last_hidden_staterencoder_attentions)r&r*rrr rdrrrr?rmrlenr@rrrar_rr1)r6rrbrKrLrMrar rNrrdrrrdecoder_outputss r)rCzMvpModel.forwardsT  $)>)F  U !34;;33T[[5W5W! 2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK&'6'6 y 0 <.2.259:>:>(,26:>!%)-,0#'g ##d*g  t+g !++d2 g !& 0 04 7 g e//047 g g ((4/g  %0047g $;g  $;g #Tkg D[g  # #g g r+r<ze The MVP Model with a language modeling head. Can be used for various text generation tasks. ) custom_introceZdZddiZdeffd Z ddededzded ejffd Z ded dfd Z d Z e dd ejdzdej dzdejdzdejdzdeej$dzdedzdej$dzdej$dzdejdzdedzdedzdedzdedzd eezfdZdej fdZxZS)rlm_head.weightzmodel.shared.weightrcxt||t||_|j dt j d|jjjftj|j|jjjd|_ |jy)NrrFrW)r4r5r<rregister_bufferr=ryr>r.rrZrlm_headr rs r)r5z$MvpForConditionalGeneration.__init__s  f%  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  r+Nnew_num_tokenspad_to_multiple_of mean_resizingrecLt||||}|j||Sr)r4resize_token_embeddings_resize_final_logits_bias)r6rarbrcnew_embeddingsr7s r)rez3MvpForConditionalGeneration.resize_token_embeddingss.8I[]jk &&~6r+c6|jjd}||kr|jddd|f}nStjd||z f|jj}tj |j|gd}|j d|y)Nr"rrrgr)rr$r=ryr<rxr_)r6raold_num_tokensnew_bias extra_biass r)rfz5MvpForConditionalGeneration._resize_final_logits_biass//55b9 ^ +--a..@AHa.)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;r+cn|jj|jjdyr&rrJr`rIrBs r)rJz2MvpForConditionalGeneration.set_lightweight_tuning$ ))+ ##E*r+rrbrKrLrMrar rNlabelsrrdrrc | | n|jj} | R| rtjdd} |7|5t | |jj |jj }|j||||||||| | | |  }|j|d|jz}d}| Ft}||jd|jj| jd}| s|f|ddz}||f|zS|St|||j|j|j |j"|j$|j&|j( S) a\ decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example of summarization: Fine-tuning a model ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.", ... return_tensors="pt", ... ) >>> labels = tokenizer("Bad Reasons To Quit Your Job", return_tensors="pt")["input_ids"] >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... generated_ids = model.generate(**inputs) >>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True) ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F) rbrKrMrLrar rNrrdrrrr"r losslogitsrarQrRr1rSrrT)rrr2warningr*rr rr`rrrurrrarQrRr1rSrrT)r6rrbrKrLrMrar rNrorrdrrrr lm_logitsmasked_lm_lossloss_fctoutputs r)rCz#MvpForConditionalGeneration.forwardsR&1%:>(,26:>*.!%)-,0#'v ##d*v  t+v !++d2 v !& 0 04 7 v e//047 v v ((4/v  %0047v   4'v $;v  $;v #Tkv D[v  !v v phELLhr+rz Mvp model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c~eZdZdeffd ZdZe ddejdzdejdzdejdzdejdzd e ejdzd ejdzd ejdzd ejdzd e dzde dzde dzde dzde ezfdZxZS)MvpForSequenceClassificationrc t||fi|t||_t |j |j |j |j|_|jyr) r4r5r<rrr num_labelsclassifier_dropoutclassification_headr )r6rrr7s r)r5z%MvpForSequenceClassification.__init__sZ *6*f% #8 NN NN     % % $   r+cn|jj|jjdyr&)rrJrrIrBs r)rJz3MvpForSequenceClassification.set_lightweight_tunings& ))+   //6r+NrrbrKrLrMr rNrorrdrrrec | | n|jj} |d} |$|"td|jj|j |||||||| | | |  }|d}|j |jjj|j}ttj|jdjdk(d||ddf}t|jd|jdzdk\d|j!|j#dd |j#d ddd ddf}|j%|}d}||jj&|jj(dk(rd |j_nv|jj(dkDrL|j*tj,k(s|j*tj.k(rd |j_nd |j_|jj&d k(rSt1}|jj(dk(r&||j3|j3}n|||}n|jj&d k(rGt5}||j!d |jj(|j!d }n,|jj&d k(rt7}|||}| s|f|ddz}||f|zS|St9|||j:|j<|j>|j@|jB|jD|jF S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: Fine-tuning a model on `num_labels` classes ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForSequenceClassification >>> num_labels = 2 # for example, this is a binary classification task >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForSequenceClassification.from_pretrained("RUCAIBox/mvp", num_labels=num_labels) >>> inputs = tokenizer("Classify: Hello, my dog is cute", return_tensors="pt") >>> labels = torch.tensor(1) # the real label for inputs >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax() ``` NFz8Passing input embeddings is currently not supported for rbrKrLrMr rNrrdrrrrz7All examples must have the same number of tokens.z3Each example must contain at least one token.r" regressionsingle_label_classificationmulti_label_classificationrq)$rrNotImplementedErrorr7rFreq eos_token_idrzr<rr=unique_consecutivesumnumelr$rurlr problem_typerr;r?rJrsqueezerrrrarQrRr1rSrrT)r6rrbrKrLrMr rNrorrdrrrrr_eos_maskselectedsentence_representationrsrrrwrxs r)rCz$MvpForSequenceClassification.forwards6J&1%>KbKbJcd ** )/#9+'"7/!5#    << 8 89<<]=Q=QR  $ $X\\!_ 5 ; ; = B E !1- NN1 !4!4Q!7 71 < A #+-- 0B0B10Er=K]K]^`Ka"bcdfhjkck"l))*AB  {{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r+rW)rFrGrHrr5rJrr=r'rKrZrrrrrCrLrMs@r)r}r}sN y 7.2.259:>:>26:>*.!%)-,0#'N ##d*N  t+N !++d2 N !& 0 04 7 N e//047 N ((4/N  %0047N   4'N $;N  $;N #TkN D[N  0 0N N r+r}ceZdZfdZdZe ddejdzdejdzdejdzdejdzde ejdzd ejdzd ejdzd ejdzd ejdzd e dzde dzde dzde dzde e zfdZxZS)MvpForQuestionAnsweringct||d|_|j|_t||_t j |j|j|_|jyr1) r4r5rr<rrrZ hidden_size qa_outputsr rs r)r5z MvpForQuestionAnswering.__init__Es[   ++f% ))F$6$68I8IJ r+cn|jj|jjdyr&)rrJrrIrBs r)rJz.MvpForQuestionAnswering.set_lightweight_tuningQs$ ))+ &&u-r+NrrbrKrLrMstart_positions end_positionsr rNrrdrrrec | | n|jj} ||d} |j||||||| | | | |  }|d}|j|}|j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|}|jd|}t|}|||}|||}||zd z }| s||f|ddz}||f|zS|St||||j|j|j|j|j |j"|j$ S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. Example: Fine-tuning a model for extrative question answering, and our model also supports generative question answering using `BartForConditionalGeneration` ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForQuestionAnswering >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForQuestionAnswering.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Answer the following question: Who was Jim Henson? [SEP] Jim Henson was a nice puppet", ... return_tensors="pt", ... ) >>> target_start_index = torch.tensor([18]) >>> target_end_index = torch.tensor([19]) >>> loss = model(**inputs, start_positions=target_start_index, end_positions=target_end_index).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1] >>> predict_answer = tokenizer.decode(predict_answer_tokens) ``` NFrrrr"rg) ignore_indexr2) rr start_logits end_logitsrarQrRr1rSrrT)rrrrrr contiguousrUrlrrrrarQrRr1rSrrT)r6rrbrKrLrMrrr rNrrdrrrrsequence_outputrsrr total_loss ignored_indexrw start_lossend_lossrxs r)rCzMvpForQuestionAnswering.forwardUsV&1%:>371526:>!%)-,0#'F <<$&F  t+F !++d2 F !& 0 04 7 F e//047 F ))D0F ''$.F ((4/F  %0047F $;F  $;F #TkF D[F 4 4!F F r+rc(eZdZdZfdZdZxZS)MvpDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cdt||t||_|j yr)r4r5r)r@r rs r)r5zMvpDecoderWrapper.__init__s&  !&)  r+c&|j|i|Sr)r@)r6argsrs r)rCzMvpDecoderWrapper.forwardst||T,V,,r+)rFrGrHrIr5rCrLrMs@r)rrs  -r+rcreZdZddiZfdZdZdZdZe dde jdzd e jdzd e jdzd e jdzd e dzd e jdzde jdzdedzdedzdedzdedzdee jzdeezfdZxZS)MvpForCausalLMr]z!model.decoder.embed_tokens.weightcd|_d|_t| |t ||_t j|j|jd|_ |jy)NTFrW) rSr4r4r5rrrrZrrr`r rs r)r5zMvpForCausalLM.__init__sX $)!  &v. yy!3!3V5F5FUS  r+cB|jjjSrrr@rrBs r)rCz#MvpForCausalLM.get_input_embeddingsszz!!...r+c:||jj_yrrrEs r)rGz#MvpForCausalLM.set_input_embeddingss*/ 'r+cn|jj|jjdyr&rmrBs r)rJz%MvpForCausalLM.set_lightweight_tuningrnr+Nrrbrrrar rorrdrrlogits_to_keeprec | | n|jj} | | n|jj} | | n|jj} |jj |||||||| | |  }|d}t | tr t| dn| }|j|dd|ddf}d}|Ft}||jd|jj|jd}| s|f|ddz}||f|zS|St|||j|j|j |j"S)ap labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MvpForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForCausalLM.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 8, 50267] ```NrPrr"r)rrrsrar_rr1)rrdrrrr@rmrJslicer`rrurrrar_rr1)r6rrbrrrar rorrdrrrrrr_ slice_indicesrsrrrwrxs r)rCzMvpForCausalLM.forwardslN2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  ')HFKKDKK,B,BCV[[QS_UDY,F'+'7D7V# CV C0#33!//))$55   r+) NNNNNNNNNNNr)rFrGrHrYr5rCrGrJrr=r'rKrr rrJrrrCrLrMs@r)rrsO*,OP /0+.2.2:>;?(,26*.!%)-,0#'-.O ##d*O  t+O  %0047 O !& 1 1D 8 O  O ((4/O   4'O $;O  $;O #TkO D[O ell*O  2 2O O r+r)rrrr}r<r)@rIrr=rtorch.nnrrrrr activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrconfiguration_mvpr get_loggerrFr2rKrJr*rr-ModulerOrrrrrrr)r<rr}rrr__all__rr+r)rs AA&!CC)J9.DD(   H % %,, c [^ ";BLL;6P2299P2f:+0:+zi0iZBII0 2*` #` Fa #a HL !L L ^ \h"4o\h  \h~ a #5a  a HX 0X X x -* -h 'h V r+