^j|<dZddlZddlZddlmZddlmZmZmZmZddlm Z ddl m Z m Z ddlmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZdd lmZmZddlm Z ejBe"Z#d'dZ$GddejJZ&GddejJZ'GddeZ(eGddeZ)eGdde)Z*edGdde)eZ+edGd d!e)Z,eGd"d#e)Z-eGd$d%e)Z.gd&Z/y)(zPyTorch MPT model.N)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional)Cache DynamicCache)GenerationMixin)create_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) MptConfigcRtjd|z dtj|jddd|}dt j t j |z}tjd|dztj|j}|||z z}dtjd|z }|jd|dd}||k7r9tj|ddddddf|ddddddfgdddd|df}||z}|jdS) a Link to paper: https://huggingface.co/papers/2108.12409 - Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation. This implementation has been copied from the alibi implementation of MPT source code that led to slightly different results than the Bloom alibi: https://huggingface.co/mosaicml/mpt-7b/blob/main/attention.py#L292 r)dtypedeviceg?N.dimr) torcharangeint32viewmathceillog2int64floatpowconcatsqueeze) num_headssequence_lengthalibi_bias_maxralibinum_heads_power_of_2baseslopess o/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/mpt/modeling_mpt.pybuild_mpt_alibi_tensorr3*s LL_,au{{6 R W WXY[\^_ap qE $))I*> ?? <</!35;;v V \ \ ^D >$88 9D 599Q% %F [[0!Q 7Fy(vaAsl3VAssCK5HIqQRSU_V_U_adRde FNE == c eZdZdZd dededzffd Z d dejdejde dzd ejdzfd Z xZ S) MptAttentionzzMulti-head self attention. Using torch or triton attention implementation enables user to also use additive bias. Nconfig layer_idxct||j|_|j|_|j|_|j|jz|_|jj|_|j4dtj|j|jz z |_|jj|_ |jj|_ tj|jd|jzd|_tj|j|jd|_||_y)Nrr Fbias)super__init__ hidden_sizen_heads max_seq_lenmax_seq_lengthhead_dim attn_config softmax_scaler#sqrt attn_pdropattn_dropout_pclip_qkvrLinearWqkvout_projr8)selfr7r8 __class__s r2r=zMptAttention.__init__Fs !--~~ $00((DLL8 #//==    %!"TYYt/?/?$,,/N%O!OD $00;;**33 IId..D4D4D0D5Q  $"2"2D4D4D5Q "r4 hidden_states position_biaspast_key_valuesattention_maskc f|jdd\}}|j|}|jr(|j|j |j}|j dd\} } } | j |||j |jjdd} | j |||j |jjdd} | j |||j |jjdd} | |j| | |j\} } tj| | jdd|jz} ||n||jz} |t|jdk7r!t!dt|j| jd}t#d |j%d| z }t#d |j%d|z }|dd|d|df}| |z} |9| j'|tj(| j*j,} t.j0j3| j5dj7| j*}t.j0j9||j:|j< }tj|| }|j?d dddjAjC||d}|jE|}||fS) Nr)minmaxr rrz6Expecting position_bias shape to be 3 dimensions, got rptraining)#shaperJrHclampchunkreshaper?rB transposeupdater8rmatmulrDget_seq_lengthlen ValueErrorrTsize masked_fillfinforrSrrsoftmaxr'todropoutrGrYpermute contiguousr"rK)rLrNrOrPrQkwargs batch_size seq_length mixed_qkv query_states key_states value_statesattention_scores query_length key_lengthposition_bias_query_indexposition_bias_key_index attn_weightscontext_states attn_outputs r2forwardzMptAttention.forwardVs"/!4!4Ra!8 JIIm, ==!T]]N NI1:1J. j,#++J DLLRVR_R_`jjklnop '' J dmm\ffghjkl #++J DLLRVR_R_`jjklnop  &'6'='=j,X\XfXf'g $J  << j6J6J2r6RSVZVhVhh%4%rrIup_projGELUact down_projrCrFhidden_dropout)rLr7r>rMs r2r=zMptMLP.__init__sm (( yya+oEJ 77v.1{?KeL$00;;r4rNresidualreturnc|j|j|}|j|}tj||j |j }||z}|S)NrW)rrrFrirrY)rLrNrintermediate_outputoutputs r2r{zMptMLP.forwardsWm!<= "nn];.$2E2EPTP]P]^(" r4) r}r~rrr=rrr{rrs@r2rrs5rlayer_norm_epsilonnorm_1r;r?r+r6attnnorm_2rffnrCrF dropout_raterDropoutresid_attn_dropout)rLr7r8r>rMs r2r=zMptBlock.__init__s ((  1J1JK   3  1J1JK  &>"..99"$**T->->"?r4rNrOrQ layer_past use_cacheoutput_attentionsc |j|}|} |j||||\} } |j| | z}|j|}|} |j || } | | fS)N)rOrQrP)rrrrr) rLrNrOrQrrrrllayernorm_outputr attn_outputsrxrs r2r{zMptBlock.forwards ;;}5 &*YY ')& &/& " l// =H ;;}5!*H5|##r4r|)NFF) r}r~rrrr=rrr boolr{rrs@r2rrsw@y@S4Z@2$("'!$||!$||!$ !$ DL !$  !$ !$r4rc&eZdZUeed<dZdZdgZy)MptPreTrainedModelr7 transformerTrN)r}r~rr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesr4r2rrs %&*## r4rc*eZdZdeffd ZdZddZdejfdZ e ddejdzd e dzd ejdzd ejdzd e dzd e dzde dzde dzdeejdfezfdZxZS)MptModelr7c t|||j|_|j|_t j |j|j|_t jt|jDcgc]}t||c}|_ t|j|j|_d|j _d|_|j'ycc}w)N)r8rF)r<r=r>r?r+r Embedding vocab_sizewte ModuleListrangen_layersrblocksrrnorm_fr;gradient_checkpointing post_init)rLr7irMs r2r=zMptModel.__init__s  !--<< 1 143C3CDmmERXRaRaLb$cqXf%B$cd   0 0f6O6OP  &+# %dsC7c|jSr|r)rLs r2get_input_embeddingszMptModel.get_input_embeddingss xxr4Nct||||Sr|)r3)rLr+r,r-rs r2r3zMptModel.build_mpt_alibi_tensors%i.RXYYr4new_embeddingsc||_yr|rrLrs r2set_input_embeddingszMptModel.set_input_embeddingss !r4 input_idsrPrQ inputs_embedsrroutput_hidden_states return_dictr.c ||n|jj}||n|jj}||n|jj}||n|jj}| | t d||j \} } n||j \} } } n t d|jr%|jr|rtjdd}||j|}|r|t|j}|} |rdnd}|rdnd}|j|j|jj| j }t#|j||| j%t&j(}|j*D])}|r|| fz}|| ||||| }|d } |s!||d fz}+|j-| } |r|| fz}|st/d | |||fDSt1| |||S) input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r7rr)r7rrQrP)rrQrrrOrrc3&K|] }|| ywr|r).0vs r2 z#MptModel.forward..^sghgts)last_hidden_staterPrN attentions)r7rrrrrcrZrrYlogger warning_oncerr r3r+r@rr rhrrrrtupler)rLrrPrQrrrrrrlrmrn_rNall_self_attentionsall_hidden_statesr. causal_maskblockoutputss r2r{zMptModel.forwardsE42C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "%.__ "J  &(5(;(; %J ATU U  & &4==##p"   HHY/M 0*$++>O% $5b4"6BD++DNNDKKrlm_headrrLr7rMs r2r=zMptForCausalLM.__init__ssI  #F+yy!3!3V5F5FUS  r4rc||_yr|)rrs r2set_output_embeddingsz$MptForCausalLM.set_output_embeddings{s % r4NrrPrQrlabelsrrrrlogits_to_keeprc | | n|jj} |j|||||||| } | d} t| tr t | dn| }|j | dd|ddf}d}|*|jd|||jjd| }| s|f| ddz}||f|zS|St||| j| j| jS)a\ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` NrPrQrrrrrr)logitsrrrlossrrPrNrr) r7rr isinstancerslicer loss_functionrrrPrNr)rLrrPrQrrrrrrrrltransformer_outputsrN slice_indicesrrrs r2r{zMptForCausalLM.forward~s@&1%SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopDY!4QR!88F)-)9TGf$ Ev E0/??-;;*55   r4) NNNNNNNNNr)r}r~r_tied_weights_keysrr=rrrrrr rrrrr{rrs@r2rrjs+,DEy&ELL&.2(,.2-1&*!%)-,0#'-.? ##d*? ?  t+ ? ||d* ?  t# ? $;?  $;? #Tk? D[? ell*?  u|| @ @? ? r4ra The MPT Model transformer with a sequence classification head on top (linear layer). [`MptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). c8eZdZdeffd Zdej fdZe ddejdzde dzdej dzd ej dzd ej dzd e dzd e dzd e dzde dzde ej e zfdZxZS)MptForSequenceClassificationr7ct|||j|_t||_t j |j|jd|_|jyr) r<r= num_labelsrrrrIr>scorerrs r2r=z%MptForSequenceClassification.__init__sV   ++#F+YYv1163D3D5Q  r4rc||_yr|)rrs r2rz2MptForSequenceClassification.set_output_embeddingss # r4NrrPrQrrrrrrrc r| | n|jj} |j|||||||| } | d} |j| } ||jd}n|jd}|jj |dk7r t d|jj d}n|||jj k7j| jtj}tj|jd| jtj}||zjd}n.d}tj|jj d| tj|| j |f}d}|^|jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}nc|||}nY|jj"d k(rt1}|||}n,|jj"d k(rt3}|||}| s|f| ddz}||f|zS|St5||| j6| j8| j: S)6 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrrz=Cannot handle batch sizes > 1 if no padding token is defined.rU)rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)r7rrrrZ pad_token_idrcrhrrr!r argmaxrrrMr} problem_typerrlongrrr*rrrrPrNr)rLrrPrQrrrrrrrlrrNrrmlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r2r{z$MptForSequenceClassification.forwards>&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- v6))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r4 NNNNNNNNN)r}r~rrr=rrrrrr rrrr{rrs@r2rrsy$ELL$.2(,.2-1&*!%)-,0#'e ##d*e e  t+ e ||d* e  t# e $;e  $;e #Tke D[e  u|| ? ?e e r4rceZdZdeffd Ze ddejdzdedzdejdzdejdzdejdzd e dzd e dzd e dzd e dzd e eje zfdZ xZS)MptForTokenClassificationr7ct|||j|_t||_t |dr|j |j }n't |dr|j |j}nd}tj||_ tj|j|j|_ |jy)Nclassifier_dropoutrg?)r<r=rrrhasattrrrrrrirIr> classifierr)rLr7rrMs r2r=z"MptForTokenClassification.__init__Gs   ++#F+ 6/ 0V5N5N5Z!'!:!:  V- .63H3H3T!'!6!6 !$ zz"45 ))F$6$68I8IJ r4NrrPrQrrrrrrrc | | n|jj} |j|||||||| } | d} |j| } |j | } d}|l|j | j }|j\}}t}|| j||z|j|j||z}| s| f| ddz}||f|zS|St|| | j| jS)rNrrr)rrrNr)r7rrrirrhrrZrr"rrrNr)rLrrPrQrrrrrrdeprecated_argumentsrrNrrrmrnr rs r2r{z!MptForTokenClassification.forwardXs+>&1% qa_outputsrrs r2r=z MptForQuestionAnswering.__init__sA  #F+))F$6$6: r4NrrQrstart_positions end_positionsrrrrc  "||n|jj}|j||||||} | d} |j| } | j dd\} }| j dj } |j dj }d}||t|jdkDr|j d}t|jdkDr|j d}| jd}|jd|}|jd|}t|}|| |}|||}||zdz }|s| |f| ddz}||f|zS|St|| || j| j S) rN)rQrrrrrrrUr) ignore_indexr)r start_logits end_logitsrNr)r7rrrsplitr*rkrbrdr[rrrNr)rLrrQrrrrrrrlrsequence_outputrrr total_loss ignored_indexr  start_lossend_lossrs r2r{zMptForQuestionAnswering.forwards4&1%r2s_ LL$.)/9.,(   H %.E)299E)PRYY*6$)6$r%%%G !G G T N 'N  N b  s #5s  s lU 2U U pP 0P P f r4