^jCddZddlmZddlZddlmZddlmZddlm Z m Z ddl m Z dd l mZmZdd lmZdd lmZmZdd lmZmZdd lmZddlmZmZmZmZddlm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0ejbe2Z3Gdde)Z4Gdde%Z5Gdde.Z6GddejnZ8Gdd e-Z9Gd!d"ejnZ:Gd#d$e'Z;Gd%d&eZ<Gd'd(eZ=eGd)d*e=Z>Gd+d,e(e=e Z?gd-Z@y).zPyTorch AFMoE model.)CallableN)nn)initialization)Cache DynamicCache)GenerationMixin)create_causal_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)OutputRecordercapture_outputs) GptOssRMSNorm)LlamaAttentionLlamaForCausalLMLlamaRotaryEmbeddingapply_rotary_pos_embeager_attention_forward)Qwen2MoeExperts Qwen2MoeMLP) AfmoeConfigc eZdZy)AfmoeRotaryEmbeddingN__name__ __module__ __qualname__r/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/afmoe/modular_afmoe.pyr%r%/r+r%c eZdZy) AfmoeRMSNormNr&r*r+r,r/r/3r-r+r/c eZdZy)AfmoeMLPNr&r*r+r,r1r17r-r+r1cZeZdZdZfdZdej dej fdZxZS)AfmoeTokenChoiceRouterz Token-choice top-K router for MoE routing. This router assigns each token to the top-K experts based on sigmoid scores, matching the released checkpoints. ct|||_|j|_|j |_|j |_tj|j|j d|_ yNFbias) super__init__confignum_experts_per_toktop_k num_experts route_scalerLinear hidden_sizegateselfr: __class__s r,r9zAfmoeTokenChoiceRouter.__init__Bs^  // !--!--IIf00&2D2D5Q r+ hidden_states expert_biasc|j\}}}|jd|}|j|jtj }t j |}t j||z|jd\}}|jd|}|jdddz} || z }||jz}|||fS)Nr")kdim)rJindexT)rJkeepdimg#B ;) shapeviewrAtotorchfloat32sigmoidtopkr<gathersumr>) rCrErF_ hidden_dim router_logitsscoresselected_experts top_scores denominators r,forwardzAfmoeTokenChoiceRouter.forwardJs(..1j%**2z:  -033EMMB }-#jj+)=QRS ]]q0@]A  nnTn:UB +- $"2"22 j*:::r+) r'r(r)__doc__r9rPTensorr] __classcell__rDs@r,r3r3;s) R ;U\\ ; ;r+r3c eZdZy) AfmoeExpertsNr&r*r+r,rcrcYr-r+rcc(eZdZdZfdZdZxZS)AfmoeSparseMoeBlockz Mixture of Experts (MoE) module for AFMoE. This module implements a sparse MoE layer with both shared experts (always active) and routed experts (activated based on token-choice routing). c2t|||_t||_t ||j |jz|_t||_ tjtj|jd|_y)NF) requires_grad)r8r9r:r3routerr1moe_intermediate_sizenum_shared_expertsshared_expertsrcexpertsr ParameterrPzerosr=rFrBs r,r9zAfmoeSparseMoeBlock.__init__esp  ,V4 &vv/K/KfNgNg/gh#F+ << F4F4F(GW\]r+c|j\}}}|jd|}|j||j\}}}|j |j|||} |j |||j|||} | | zS)NrH)rMrNrhrFrkrl) rCrE batch_sizeseq_lenrWhidden_states_flatrVr[rZ shared_output routed_outputs r,r]zAfmoeSparseMoeBlock.forwardms*7*=*=' GZ*//J?+/++mTEUEU*V':'++,>?DDZQXZde  %79I:V[[  },,r+)r'r(r)r^r9r]r`ras@r,rere]s^ -r+receZdZdZdedeffd Z d dejde ejejfdejdzd e dzd e e d e ejejff d Z xZS)AfmoeAttentionaJ Multi-headed attention module with optional sliding window and gating. This attention mechanism supports both full attention and sliding window attention, and includes Q/K normalization and gating of the output. It inherits from [`LlamaAttention`] to minimize the amount of custom logic we need to maintain. r: layer_idxct||||j|dk(|_|jr |jnd|_t |j |j|_t |j |j|_ tj|j|j|j zd|_y)Nsliding_attentionepsFr6)r8r9 layer_typesis_local_attentionsliding_windowr/head_dim rms_norm_epsq_normk_normrr?r@num_attention_heads gate_projrCr:rwrDs r,r9zAfmoeAttention.__init__s +#)"4"4Y"?CV"V7;7N7Nf33TX"4==f6I6IJ "4==f6I6IJ 6#5#5v7Q7QTXTaTa7ahmnr+NrEposition_embeddingsattention_maskpast_key_valuekwargsreturnc |jdd}g|d|j}|j|j|}|j |j|} |j |j|} |j |} |j|jdd}|j| jdd} | jdd} |jr|\} } t|| | | \}} | |j| | |j\} } tj|j j"t$}|||| | f||j&sdn |j(|j*|j,d|\}}|jg|dj/}|t1j2| z}|j5|}||fS)NrHr"r)rdropoutscalingr~)rMrq_projrNk_projv_projrr transposerr}rupdaterwr get_interfacer:_attn_implementationrtrainingattention_dropoutrr~ contiguousrPrRo_proj)rCrErrrr input_shape hidden_shape query_states key_states value_states gate_statescossinattention_interfaceoutput attn_weights attn_outputs r,r]zAfmoeAttention.forwards$))#2. 88b8$--8 {{=166|D [[/44\B {{=166|D nn]3 {{<0::1a@ [[,66q!< #--a3  " "*HC';L*VY[^'_ $L*  %'5'<'.) t+ .)  .) +, .) u||U\\) *.)r+rvceZdZdZdedeffd Z ddejdejdzdejdzd e dzd e dzd e ejejfdzd e ed ejfdZxZS)AfmoeDecoderLayerz AFMoE decoder layer with dual normalization. This layer applies self-attention followed by either a dense MLP or MoE block, with dual normalization (pre and post) around each component. r:rwc(t||j|_||_t |||_t |j|j|_t |j|j|_ t |j|j|_ t |j|j|_ ||jk\|_ |jrt||_yt!||_y)N)r:rwrz)r8r9r@rwrv self_attnr/rinput_layernormpost_attention_layernormpre_mlp_layernormpost_mlp_layernormnum_dense_layers moe_enabledremlpr1rs r,r9zAfmoeDecoderLayer.__init__s !--"'vK ,F,>,>FDWDWX(4V5G5GVM`M`(a%".f.@.@fFYFY!Z".v/A/AvGZGZ"[%(?(??   *62DH'DHr+NrEr position_idsr use_cacherrrc |}|j|}|jd||||||d|\}} |j|}||z}|}|j|}|j |}|j |}||z}|S)N)rErrrrrr*)rrrrrr) rCrErrrrrrresidualrVs r,r]zAfmoeDecoderLayer.forwards!,,]; )4>> ')%) 3    q55mD  =0 !..}= / // >  =0 r+)NNNNN)r'r(r)r^r#rr9rPr_ LongTensorrboolrrr FloatTensorr]r`ras@r,rrs({(s(2/304'+!%HL!||! t+!&&- !  ! $; !#5<<#=>E!+,!   !r+rczeZdZUdZeed<dZdgZdgZe e de e dZ gd Zd Zd Zd Zd Zd Zd Zfd ZxZS) AfmoePreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r:modelrpast_key_valuesr)rK)rXrE attentions)rrrrrrnormrFTct|||jj}t |t rEt j|jd|t j|jd|yt |tr*t j|jjyt |tr t j|jyy)zInitialize the weightsr)meanstdN)r8 _init_weightsr:initializer_range isinstancercinitnormal_ gate_up_proj down_projr3zeros_rAweightrerF)rCmodulerrDs r,rz"AfmoePreTrainedModel._init_weights#s f%kk++ fl + LL,,3C @ LL)) =  6 7 KK ** +  3 4 KK** +5r+)r'r(r)r^r#__annotations__base_model_prefix_no_split_modules_skip_keys_device_placementrr3rrv_can_record_outputs_keep_in_fp32_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_can_compile_fullgraph_supports_attention_backendsupports_gradient_checkpointingrr`ras@r,rrsx ,-#4"5'(>aH*$ N!"&&*# , ,r+rceZdZdZdeffd Zeee dde jdzde jdzde jdzde jdzd e dzd edzd eed eezfd ZxZS) AfmoeModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`AfmoeDecoderLayer`] Args: config: AfmoeConfig r:c  t|||j|_|j|_t j |j|j|j|_t jt|jDcgc]}t||c}|_ t|j|j|_t#||_d|_|j)ycc}w)Nrzr:F)r8r9 pad_token_id padding_idx vocab_sizer Embeddingr@ embed_tokens ModuleListrangenum_hidden_layersrlayersr/rrr% rotary_embgradient_checkpointing post_initrs r,r9zAfmoeModel.__init__9s  !.. ++LL):):FO   --i8M  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L?-F++!."0#2 K#5"C{"C%F%U%U#  &  ;; " ")T[[-D-Dc-IJM"oom\J )$++ 6  A})24;;3J3J13MN).#$7 M  -0 %+/8O  >B  r+)NNNNNN)r'r(r)r^r#r9rrrrPrr_rrrrrrrr]r`ras@r,rr0s { .2.22604(,!%< ##d*<  t+< ((4/ < &&- <  < $;< +,<  ' '<  < r+rcBeZdZddiZddiZddgdgfiZdZee dd e jdzd e jdzd e jdzd e dzd e jdzde jdzdedzdedzdee jzdeedefdZy)AfmoeForCausalLMzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputrElogitsctj||t||_|j|_t j |j|jd|_|jyr5) rr9rrrrr?r@rr)rCr:s r,r9zAfmoeForCausalLM.__init__sU%%dF3'  ++yy!3!3V5F5FUS  r+Nrrrrrlabelsroutput_router_logitslogits_to_keeprrc ||n|jj}|jd|||||||d| } | j} t | t r t | dn| } |j| dd| ddf}d}||j|||jfi| }t||| j| j| j| jS)N)rrrrrrr)lossrrrErrXr*)r:rrrrrslicer loss_functionrr rrErrX)rCrrrrrrrrrroutputsrE slice_indicesrrs r,r]zAfmoeForCausalLM.forwards %9$D $++JjJj +5$** + )%+'!5 +  +  11 8B>SV8W~ot4]k mA}a,?@A  %4%%ffdooPPD(#33!//))!//   r+) NNNNNNNNr)r'r(r)_tied_weights_keys_tp_plan_pp_planr9rrrPrr_rrrrrrr r]r*r+r,rrs'*,GH23H_-z:;H.2.204(,26*.!%,0-.+ ##d*+  t++ &&- +  + ((4/ +   4'+ $;+ #Tk+ ell*+ +,+  #+ + r+r)rrr)Ar^collections.abcrrPrrr cache_utilsrr generationr masking_utilsr r modeling_layersr modeling_outputsr rmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrrgpt_oss.modeling_gpt_ossrllama.modeling_llamarrrrrqwen2_moe.modeling_qwen2_moer r!configuration_afmoer# get_loggerr'loggerr%r/r1Moduler3rcrervrrrr__all__r*r+r,r!s$ &.)R9QF&RR7E4H,   H % /  =  { ;RYY;< ? -"))->B)^B)J?2?D*,?*,ZV %V V r9 ')=9 x r+