^j|ddlmZddlmZddlmZddlZddlmZddlm Z ddl m Z dd l mZmZdd lmZdd lmZmZmZdd lmZdd lmZddlmZddlmZmZmZm Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6m7Z7edGddejpZ9GddejpZ:Gdd ejpZ;Gd!d"ejpZ<Gd#d$ejpZ=d%Z>Gd&d'ejpZ?Gd(d)ejpZ@Gd*d+ejpZAd,ZBed-dXd.ZCd/ejd0eEd1ejfd2ZF dYd3ejpd4ejd5ejd6ejd7ejdzd8eGd9eGd:e(e*fd;ZHeeCGd<d=ejpZIGd>d?eZJe+Gd@dAe&ZKe+GdBdCe&ZLGdDdEejpZMe+GdFdGeKZNe+GdHdIeKeZOe+dJKeGdLdMe ZPe+dNKeGdOdPeZQe+dQKGdRdSeLZRe+dTKGdUdVeLeZSgdWZTy)Z)Callable) dataclass)OptionalN)nn)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)maybe_autocastmerge_with_config_defaults)capture_outputs) AutoModel) AriaConfigAriaTextConfigRMSNormcheZdZddeddffd Zdej dej fdZdZxZ S) AriaTextRMSNormepsreturnNct|tjt j ||_||_y)z> AriaTextRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer+ __class__s q/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/aria/modeling_aria.pyr/zAriaTextRMSNorm.__init__6s1 ll5::k#:; # hidden_statesc"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)Nr#T)keepdim) dtypetor1float32powmeanrsqrtr4r3)r5r: input_dtypevariances r8forwardzAriaTextRMSNorm.forward>sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r9c^t|jjd|jS)Nz, eps=)tupler3shaper4r5s r8 extra_reprzAriaTextRMSNorm.extra_reprEs* ))*+6$2G2G1HIIr9)gư>) __name__ __module__ __qualname__floatr/r1TensorrFrK __classcell__r7s@r8r*r*4s7$$$$;U\\;ell;Jr9r*c(eZdZdZfdZdZxZS)AriaProjectorMLPa! Feed-Forward Network module for the Aria Projector. Args: in_features (`int`): Input embedding dimension. hidden_features (`int`): Hidden dimension of the feed-forward network. output_dim (`int`): Output dimension. ct|tj||d|_tj||d|_t d|_y)NFbiasgelu_new)r.r/rLinear linear_in linear_outr act)r5 in_featureshidden_features output_dimr7s r8r/zAriaProjectorMLP.__init__VsB ;eL))OZeL*%r9ch|j|j|}|j|}|SN)r\rZr[)r5r:s r8rFzAriaProjectorMLP.forward\s- !>?  6 r9rLrMrN__doc__r/rFrQrRs@r8rTrTIs & r9rTc6eZdZdZddedeffd ZddZxZS)AriaCrossAttentionzv Aria Cross-Attention module. Args: config (`AriaConfig`): The configuration to use. config dropout_ratecBt||jj}|jj}||_t j||d|_t j||d|_ t j||d|_ t j||d|_ t j|||_ t j||_t j ||_t j ||_y)NFrVT) batch_first)r.r/ vision_configr6num_attention_heads num_headsrrYq_projk_projv_projMultiheadAttentionmultihead_attnlinearDropoutdropout LayerNorm layer_norm layer_norm_kv)r5rfrgr6rlr7s r8r/zAriaCrossAttention.__init__ks **66 ((<< "ii [uE ii [uE ii [uE !33KX\]ii [9 zz,/ ,,{3\\+6r9c|j|j|}|j|}|j|}|j |}|j ||||\}}|j |j|}|S)a Forward pass of the AriaCrossAttention module. Args: key_value_states (`torch.Tensor`): Input tensor for key and value. hidden_states (`torch.Tensor`): Input tensor for query. attn_mask (`torch.Tensor`, *optional*, defaults to None): Attention mask. Returns: torch.Tensor: Output tensor after cross-attention.  attn_mask)rmrvrwrnrorqrtrr) r5key_value_statesr:rzquerykeyvalue attn_output_s r8rFzAriaCrossAttention.forward|s  DOOM:;--.>?kk*+ ,-,,UC),T Qll4;;{#;< r9)rra) rLrMrNrcr&rOr/rFrQrRs@r8rerebs 7z77"r9recheZdZdZdeffd ZddejdejdzfdZxZ S) AriaProjectora  Aria Projector module. This module projects vision features into the language model's embedding space, enabling interaction between vision and language components. Args: config (`AriaConfig`): Configuration object for the model. rfct||j|_|jj |_|jj|_|jj |_ |jj |_ |jj |_ tjtj |j"|j |_t'||_tj*|j |_t/|j |j|j|_yra)r.r/projector_patch_to_query_dictpatch_to_query_dictrjr6r]rkrlkv_dim text_configr^r_rr0r1zeros'max_value_projector_patch_to_query_dictr|re cross_attnrurvrT feed_forwardr5rfr7s r8r/zAriaProjector.__init__s #)#G#G !//;;--AA**66 %11== ,,88\\%++f.\.\^b^n^n"op ,V4,,t'7'78,T-=-=t?S?SUYUdUder9Nr{rzc4|jd|jd}}||jvr*td|d|jjd|j|}|jd|j dj |dd}|M|j|jd}|j djd|jdd}|j|||}|j|j|}|S) a Forward pass of the Projector module. Args: key_value_states (`torch.Tensor`): Input tensor of shape (batch_size, num_patches, kv_dim). attn_mask (`torch.Tensor`, *optional*, default is None): Attention mask. Returns: `torch.Tensor`: Output tensor of shape (batch_size, query_number, output_dim). rr%zNumber of patches z: not found in patch_to_query_dict amongst possible values .Nr<ry)rIrKeyErrorkeysr| unsqueezerepeatrepeat_interleaverlexpandsizerrrv) r5r{rz batch_size num_patches query_numqueries attention_outouts r8rFzAriaProjector.forwards6#3"8"8";=M=S=STU=VK d66 6$[M1klpmEmEmJmJmLlMMNO ,,[9 **Zi(2215<? r9ra) rLrMrNrcr&r/r1rPrFrQrRs@r8rrs8ff( PTATr9rc.eZdZdZdeffd ZdZxZS)AriaSharedExpertsMLPa/ Shared Expert MLP for shared experts. Unlike routed experts, shared experts process all tokens without routing. This class reconfigures the intermediate size in comparison to the LlamaMLP. Args: config (`AriaTextConfig`): Configuration object for the Aria language model. rfc t|||_|j|_|j|j z|_t j|j|j|j|_ t j|j|j|j|_ t j|j|j|j|_ t|j|_y)NrV)r.r/rfr6intermediate_sizemoe_num_shared_expertsrrYmlp_bias gate_projup_proj down_projr hidden_actact_fnrs r8r/zAriaSharedExpertsMLP.__init__s  !--!'!9!9Fdevicedimr%) rIr1rr>rcumsumlongcatrangematmul) token_statesexpert_weightstokens_per_expert num_tokens out_featuresoutputcumsum_num_tokens zero_tensor expert_numstartendtokensrs r8sequential_experts_gemmrs##A&J!''+L [[\9K9KT`TgTg hF %6A>++auzz:K:R:RSK ;0A"BCN0034 !*- Q/eC(ll6>*#=>uS   Mr9c(eZdZdZfdZdZxZS)AriaGroupedExpertsGemmaP Grouped GEMM (General Matrix Multiplication) module for efficient expert computation. This module utilizes the grouped_gemm library (https://github.com/fanshiqing/grouped_gemm) for optimized performance. If the grouped_gemm library is not installed, it gracefully falls back to a sequential GEMM implementation, which may be slower but ensures functionality. Args: in_features (`int`): Number of input features. out_features (`int`): Number of output features. groups (`int`): Number of expert groups. ct|||_||_||_t j tj||||_ yra) r.r/r]rgroupsrr0r1emptyr3)r5r]rrr7s r8r/zAriaGroupedExpertsGemm.__init__$sB &( ll5;;v{L#QR r9cLt||j|jS)au Perform grouped matrix multiplication. Args: input (`torch.Tensor`): Input tensor of shape (num_tokens, in_features). tokens_per_expert (`torch.Tensor`): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor of shape (num_tokens, out_features). )rr3cpu)r5inputrs r8rFzAriaGroupedExpertsGemm.forward+s''  KK  ! ! #  r9rbrRs@r8rrs S r9rcNeZdZdeddffd ZdZdejfdZxZ S) AriaExpertsrfr,Nct|||_t|j|j dz|j |_t|j |j|j |_y)Nr#) r.r/rfrr6rmoe_num_expertsfc1fc2rs r8r/zAriaExperts.__init__@sa  )&*<*V>VYZ>Z\b\r\rs)&*B*BFDVDVX^XnXnor9ctj||jjd\}}tj j |d}||fS)Nr%)krr<r)r1topkrfmoe_topkr functionalsoftmax)r5 router_logits top_logits top_indicesscoress r8route_tokens_to_expertsz#AriaExperts.route_tokens_to_expertsFsH"'**]dkk>R>RXY"Z K&&zr&:F""r9cB|j|\}}|j}tj|j j tj |jjd|jjdz j |}|}|jd}tj|} |jd| |jjz} |j| |} tj| dd\} } tj j#| | z} |j%| |}tj&|j(d|jjz|j+df|j|j,}|j/d| ||jd|jj|j+d}||j1dzj3d}|S)Nrr%)binsminmaxr<r#rr)rr>r1histcflattenr?r@rfrviewargsort index_selectrrchunkrrsilurrrIrr index_copy_rsum)r5r:r top_k_index top_k_weightsoriginal_dtyperindicesflatten_indicessorted_indicespermuted_tokens fc1_output projectiongate expert_outputunpermuted_tokensrs r8rFzAriaExperts.forwardKs%)%A%A-%P" ]$**!KK    ! $ $U]] 3,, ++a/  "^   !,,r*7'44Q$++J^J^8^_XXo/@A  ;;z1"= D]]'' 3d: ->? !KK   #dkk&:&: :M