^j_G dZddlZddlmcmZddlmZddlmZddl m Z ddl m Z m Z ddlmZdd lmZmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZddlm Z ddl!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,ejZe.Z/ d4dej`e1ej`zdzde2dzdej`dzdej`e2zfdZ3eGddejhZ5GddejhZ6GddejhZ7Gdd e)Z8Gd!d"e*Z9Gd#d$e"Z:Gd%d&eZ;Gd'd(e(Z<Gd)d*e'Z=Gd+d,e#Z>Gd-d.e%Z?Gd/d0e&Z@Gd1d2e$ZAgd3ZBy)5zPyTorch Mixtral model.N)nn)initialization)ACT2FN)Cache DynamicCache)use_experts_implementation)create_causal_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)PreTrainedModel)Unpack)TransformersKwargslogging)OutputRecorder) MistralAttentionMistralForCausalLMMistralForQuestionAnswering MistralForSequenceClassificationMistralForTokenClassification MistralModelMistralPreTrainedModelMistralRMSNormMistralRotaryEmbedding) MixtralConfig gate_logits num_expertsattention_maskreturnc|t|tsyt|trC|dj}tj|Dcgc]}|j |c}d}tj jjd}tj||d\}} tj jj| |} |>tj| jd} tj|d} n|j\} }|jd| |zz}|dddddddfj|| |||fjd||j }tj | j|zdtj |dz } |ddddddfj|| ||fjd|j |}tj ||zdtj |dz } tj | | j#dz}||zScc}w)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrdim) isinstancetupledevicetorchcattor functionalsoftmaxtopkone_hotmeanfloatshapeexpandreshapesum unsqueeze)r r!top_kr"compute_device layer_gateconcatenated_gate_logitsrouting_weights_selected_experts expert_masktokens_per_expertrouter_prob_per_expert batch_sizesequence_lengthnum_hidden_layersexpert_attention_mask router_per_expert_attention_mask overall_losss v/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/mixtral/modular_mixtral.pyload_balancing_loss_funcrJ5s9:*[%"@+u%$Q..#(99^i-jPZjmmN.K-jpq#r hh))112JPR1SO**_eDA((%%--.> LK!JJ{'8'8':B"'O!C&4&:&:# O4::1=*B^_ 4AtT1 2 V& OUKX Y WR , R  "IIk&7&7&9rbras rIryzMixtralSparseMoeBlock.forwards2?2E2E/ OZ ==T..2 U--m<EEcDL]L]F]_beievev_vw wM%**2}/B/B2/FG (, -(@%=+ ]KO %--j/:V r_) rzr{r|rPr+r~r)ryrrs@rIrrs1.U\\eELL%,,s rIryzMixtralDecoderLayer.forwards!,,]; )4>> ' 3)%+     q!=0  55mD /  =0 r_)NNNN)rzr{r|rintrPr+r~r) LongTensorrrrryrrs@rIrrsd}ddIM.204(, ||#5<<#=>E t+  &&-   +, r_rcVeZdZeedeedZejdZ y)MixtralPreTrainedModelr)index)rr` attentionscltj|||jj}t |t rEt j|jd|t j|jd|yt |tr#t j|jd|yy)Ng)r2std) r _init_weightsrMinitializer_ranger(rLinitnormal_rXrYrr)r]modulers rIrz$MixtralPreTrainedModel._init_weights s{%%dF3kk++ fn - LL,,3C @ LL)) =  1 2 LLSc :3r_N) rzr{r|rrrr_can_record_outputsr+rgrrr_rIrrs7'(9C,& U]]_;;r_rceZdZ d dejdzdej dzdejdzdedzdejdzdedzde e d e fd Z y) MixtralModelN input_idsr"rr inputs_embeds use_cacherr#c z|du|duz r td|r|t|j}||j|}|V||j nd}t j |jd|j|z}|jd}|jjtnt} | |j||||} |} |j| |} |jd|jjD]} | | f| |||| d|} |j!| } t#| | S) Nz:You must specify exactly one of input_ids or inputs_embeds)rMrr)r*)rMrr"rr)r)r"rrrr)last_hidden_stater) ValueErrorrrM embed_tokensget_seq_lengthr+aranger4r*r8sliding_windowr r rotary_emblayersrEnormr)r]rr"rrrrrpast_seen_tokens mask_function causal_maskr`r decoder_layers rIryzMixtralModel.forwardsd  -t"; <YZ Z 0*$++>O   --i8M  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L.2kk.H.H.P*Vw #;;')+%  & "oom,oW![[)H4;;+H+HI M)*) /#$7 M  -0 %++  r_)NNNNNN) rzr{r|r+rr~r FloatTensorboolrrrryrr_rIrrs.2.204(,26!%4 ##d*4  t+4 &&- 4  4 ((4/ 4 $;4 +,4  4 r_rc"eZdZddiZfdZ ddej dzdejdzdej dzdedzd ejdzd ej dzd e dzd e dzd e ejzde e defdZxZS)MixtralForCausalLMzlm_head.weightzmodel.embed_tokens.weightct||t||_|j|_|j |_|j|_yr)rOrPrmodelrouter_aux_loss_coefrQr!rr\s rIrPzMixtralForCausalLM.__init__SsF  !&) $*$?$?!!33#)#=#= r_Nrr"rrrlabelsroutput_router_logitslogits_to_keeprr#c j||n|jj}|jd|||||||d| } | j} t | t r t | dn| } |j| dd| ddf}d}||j|||jfi| }d}|rYt| j|j|j|}|+||j|j|j zz }t#|||| j$| j&| j(| jS)a~ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MixtralForCausalLM >>> model = MixtralForCausalLM.from_pretrained("mistralai/Mixtral-8x7B-v0.1") >>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-v0.1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N)rr"rrrrr)lossaux_losslogitsrr`rrr)rMrrrr(rslicelm_head loss_function vocab_sizerJrr!rrr-r*r rr`r)r]rr"rrrrrrrroutputsr` slice_indicesrrrs rIryzMixtralForCausalLM.forwardZsUJ%9$D $++JjJj  +5$** + )%+'!5 +  +  11 8B>SV8W~ot4]k mA}a,?@A  %4%%ffdooPPD /%%  (( H !11HKK 4LLL(#33!//))!//  r_) NNNNNNNNr)rzr{r|_tied_weights_keysrPr+rr~rrrrrrr ryrrs@rIrrPs*,GH>.2.204(,26*.!%,0-.P ##d*P  t+P &&- P  P ((4/ P   4'P $;P #TkP ell*P +,P  #P r_rc eZdZy) MixtralForSequenceClassificationNrrr_rIrrrr_rc eZdZy)MixtralForTokenClassificationNrrr_rIrrrr_rc eZdZy)MixtralForQuestionAnsweringNrrr_rIrrrr_r)rrrrrr)NrN)Cr}r+torch.nn.functionalrr.rrr activationsr cache_utilsrr integrationsr masking_utilsr r modeling_layersr modeling_outputsr rmodeling_utilsrprocessing_utilsrutilsrrutils.output_capturingrmistral.modeling_mistralrrrrrrrrrconfiguration_mixtralr get_loggerrzloggerr~r)rrJModulerLrrrrrrrrrrrr__all__rr_rIrs& &!.6R9Q-&04   1   H % # *. O&ell 33d:O&tO&LL4' O&  \\C O&d$#RYY$#$#N< <$BII& ^  3  ' #4#L;3;$5 <5 pZ +Z z 'G  $A  "=  r_