^jP#FUdZddlZddlmZddlmZddlmZddlZddl m Z edZ edZ iZ eejj efed <ehd Zd ed ed ee e fdeee e fgee e fffdZ d9dddddej.dej.dej.dej.dzdedededej.fdZedde d9dddddej.dej.dej.dej.dzdedededej.fdZdedzdedefdZd ej.d!ededej.fd"Zd#ej.d$ej.d%ed&ededzd'edej.fd(Zd%ed&eddfd)Zd#ej.d$ej.d%ed&ededzdej.f d*Z d:dddddd+dd,d#ej.d$ej.d-ej.d.ej.dzd/ej.dzd0ej.dzd1ed2ed3ed'ededzd4ed5edzde!ej.ej.ej.ej.ffd6Z"ed7de" d:dddddd+dd,d#ej.d$ej.d-ej.d.ej.dzd/ej.dzd0ej.dzd1ed2ed3ed'ededzd4ed5edzde!ej.ej.ej.ej.ffd8Z#y);zImplementations of ONNX operators as native Torch ops. NOTE: Fake implementations: Refer to https://docs.pytorch.org/docs/stable/library.html#torch.library.register_fake for more details on how to create fake kernels. N)Callable)TypeVar) ParamSpec)_dtype_mappings_P_RONNX_ATEN_DECOMP_TABLE> op_type opset_version fake_implreturnchdtttfdtttfffd }|S)zDDecorator to register an ONNX operator with a custom implementation.funcrcd}tjjdd|d|}|tt t tj j |<|j|S)Nopsetzonnx::.) mutates_args)torchlibrary custom_opr getattropsonnx register_fake)roverloadtorch_oprrrs _/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/torch/onnx/ops/_impl.py decoratorz_onnx_op..decorator&s|=/*==**WIQxj )+   wwuyy~~w'GRS y))rrr)rrrr#s``` r"_onnx_opr%!s0 R( Xb"f-=  r$F) interleaved num_headsrotary_embedding_dimx cos_cache sin_cache position_idsr&r'r(c|jdk(r7tj|jd|j|j Stj |j|j|j S)zFFake implementation for RotaryEmbedding-23 for torch.compile purposes.rr dtypedevice)dimrempty_permutedshaper3r4empty)r)r*r+r,r&r'r(s r"_rotary_embedding_23_fake_implr94s\ uuw!|## GG\  ;;qwwaggahh ??r$RotaryEmbeddingcV|jt}ddtjj dk(fdtjjdk(fdtjjdk(fdtjj dk(xrj dk(fd n@tjj d k(xrj d k(fd |d k(rtj |d }nG|d k(rBtj|dk7fdd}||z} || g} tj || }tjt|jd k(d|jd } |dk(r| }|ddddddd|f} |dddddd|df} |dz ntjjdk(xrjdk(fdtjjdk(xrjdk(fdtjjdk(fdtjjdk(fdtjdtjd|r%| dddddddddf} | dddddddddf}ntj| dd\} }| z|zz }| z|zz}|rftj|d}tj|d}tj||fd}tj || j} ntj||fd} tj| | fd}|d k(rtj |Stj |d S)z_RotaryEmbedding-23 https://onnx.ai/onnx/operators/onnx__RotaryEmbedding.html#rotaryembedding-23rNr0c"djS)Nz6position_ids must be 2D when provided. Received shape r7)r,sr"z%rotary_embedding_23..csL\M_M_L`ar$c.ddjdS)Nz6position_ids first dim (batch) must match x.shape[0] ( ). Received rr?) batch_sizer,sr"r@z%rotary_embedding_23..gs$LZLXdeqewewxyezd{|r$r c.ddjdS)Nz;position_ids second dim (sequence) must match x.shape[-2] (rBr r?)r,sequence_lengthsr"r@z%rotary_embedding_23..ks=QRaQbbno{pBpBCDpEoFGr$c<djdjS)NzWcos_cache/sin_cache must be 2D when position_ids is provided. Received cos_cache shape , sin_cache shape r?r*r+sr"r@z%rotary_embedding_23..o&((1'88J9??J[]r$r1c<djdjS)Nz[cos_cache/sin_cache must be 3D when position_ids is not provided. Received cos_cache shape rGr?rHsr"r@z%rotary_embedding_23..urIr$r.r/cdS)NzKnum_heads must be provided for 3D inputs. Received input tensor with shape r) input_shapesr"r@z%rotary_embedding_23..sabmanor$cy)Nzx should be a 4D tensor by nowrrr$r"r@z%rotary_embedding_23..r$c0djdddS)Nzcos has shape  but expected (batch=, seq=, ...)r?)rCcosrEsr"r@z%rotary_embedding_23..$. +@ FSbRccijr$c0djdddS)Nzsin has shape rPrQrRr?)rCrEsinsr"r@z%rotary_embedding_23..rTr$c0djdddS)NzLast dimension of cos cache (rW') should match rotary_embedding_dim/2 ().r?)rSrotary_embedding_dim_halfsr"r@z%rotary_embedding_23.../ " >effAACDr$c0djdddS)NzLast dimension of sin cache (rWrYrZr?)r[rVsr"r@z%rotary_embedding_23..r\r$r5) r7lenr_checkr5permutereshape unsqueezechunkcat)r)r*r+r,r&r'r( input_rank hidden_size head_size new_shapex_rotate x_not_rotatex1x2realimagx_rotate_concatoutputrCrSrLr[rErVs ``` @@@@@@r"rotary_embedding_23rrLs''K[!JQJ!"oO    ! # a     q !Z / |     q !_ 4 G   MMOq 9Y]]_%9 ]  MMOq 9Y]]_%9 ] Q MM!\ * q N o "!n 9, )YG MM!Y ' LLQWW"$LM Iq (Aq////0HQ12334L 4 9     LL ! "Fsyy|'Fj LL ! "Fsyy|'Fj LL " 22 D LL " 22 D // Q C // Q C  aAqt!tm $ aAqt!tm $Xqb1B 8cBh D 8cBh DtR(tR())T4Lb9==(..A99dD\r2 YY,/R 8FQ}}V[11 == ..r$scalerhc:||Sdtj|z S)z/Get the scale factor for attention computation.g?)mathsqrt)rsrhs r"_get_scale_factorrws %5GC$))I2F,FGr$tensorrCc|jd|jd}}||z}|j||||jddjS)z1Reshape 3D tensor to 4D for multi-head attention.r r0)r7view transpose contiguous)rxrCr'rErgrhs r"_reshape_3d_to_4dr}sP$*<<?FLLO[Oy(I JIF 1a r$QKcurrent_q_num_headscurrent_kv_num_headsqk_matmul_output_modec |dk(rt|||||Stjtj||j ddS)z1Get QK output tensor based on the specified mode.rr=rW)_compute_qk_output_for_mode_0r zeros_likematmulr{)r~rrrrsrs r"_get_qk_output_for_aten_spdarsO!, q%';U   Q B0C DEEr$cHtjzdk(fdy)z-Validate Group Query Attention configuration.rcdddS)Nz q_num_heads (z%) must be divisible by kv_num_heads (z ) for GQAr)rrsr"r@z-_validate_gqa_configuration..s- 344YZnYooxyr$N)rr`)rrs``r"_validate_gqa_configurationrs" LL22a7yr$c|}||k7r||z}|j|d}t||jd}tj|}||z} ||z} t j | | jddS)zDHelper function to compute QK output for qk_matmul_output_mode == 0.r r^r1r=rW)repeat_interleaverwr7rurvrrr{) r~rrrrsK_for_qk repeat_factor scale_factor sqrt_scaleQ_scaledK_scaleds r"rrsH22+/CC &&}!&<$UAGGAJ7L<(J:~H*$H <<("4"4R"< ==r$) is_causal kv_num_heads q_num_headsrrssoftcapsoftmax_precisionV attn_maskpast_key past_valuerrrrrc|jd} t|jdk(r|jd}|j}|4| ||jd|jdz|jd|zf}n#| ||jd|jd|zf}|}| |||df}n|jd}|j}|K|jd|jd|jd|jdz|jdf}n |j}|}|jd|jd|jd|df}tj||j|j }tj||j|j }tj||j|j }tj||j|j }||||fS)z@Fake implementation for Attention-23 for torch.compile purposes.rr1r r0r2)r7r_rr8r3r4)r~rrrrrrrrrrsrrrCq_sequence_length output_shapepresent_key_shapepresent_value_shapeqk_output_shaperq present_key present_value qk_outputs r"_attention_23_fake_implrs"J 177|qGGAJww   q!AGGAJ. l* !   l* !  0    a  GGAJww     q!AGGAJ. ! !" / GGAJ GGAJ GGAJ a  [[QWWQXX FF++/qwwqxxPKKK 3177188TM O177188LI ; y 88r$ Attentionc  d\} }}t|j}|jd}t|jdk(rWtj|dk7xr|dk7d|jd}t |||}t |||}t |||}tjt|jdk(xr2t|jdk(xrt|jdk(d|j|}t | |} |tj ||g| n|j}|tj ||g| n|j}||}}|j| }|j| }|j|}|j|}| d k(xr0| dk(xr)| duxr#|duxs|jtjk(}t|||rOtjjj||||d || t||k7 }t||||| | }nY||k7r+||z}|j|| }|j|| }tj |||j|j" }|rytj|dud tj$tj&||tj|j" }|j)|t+d}|?|jtjk(r|j)|t+d}n||z}t | |jd}t-j.|} || z}!|| z}"tj0|!|"j3dd}#|#}|#|z}$| dk(r|$}| d kDr| tj4|$| z z}$| dk(r|$}| w| t6vrW|$j}%|$j9t:j<| }$tj>|$d }&|&j9|%}&n/tj>|$d }&ntj>|$d }&| dk(r|&}tj0|&|}|dk(r#|j3ddjA||d}||||fS)zMAttention-23 https://onnx.ai/onnx/operators/onnx__Attention.html#attention-23)r r0r1rr1cy)Nz;q_num_heads and kv_num_heads must be provided for 3D inputsrrr$r"r@zattention_23..rNr$r r.cy)Nz'Q, K, and V should be 4D tensors by nowrrr$r"r@zattention_23..rNr$Nr^r)r dropout_prrs enable_gqar2cy)Nz'Cannot use both is_causal and attn_maskrrr$r"r@zattention_23..rNr$z-infr=rWr0)!r_r7rr`r}rwrecloner3boolrnn functionalscaled_dot_product_attentionrrzerosr4trilones masked_fillfloatrurvrr{tanh-_ATTENTION_23_ALLOWED_INTERMEDIATE_PRECISIONStorONNX_DTYPE_TO_TORCH_DTYPEsoftmaxrb)'r~rrrrrrrrrrsrr num_head_dim sequence_dimhead_diminput_shape_lenrCr q_head_sizerrrrkv_sequence_length can_use_sdparqrr attn_bias causal_maskrrrrqk_matmul_output qk_with_biasoriginal_dtype qk_softmaxs' r" attention_23rls&,3(L,!''lOJ 177|q 1  2!2 Q GGAJ a[ 9 a\ : a\ : LL AGG Ec!''la/ECLA4E9 ''(#K e[ 1E    8Q-\2 WWY  !  :q/|4 WWY  qA'',/77<0 -. 3 A !Q & A  % A$  ?)//UZZ"?  35IJ$$AA #';;B  1   !   "6 6/3GGM##M|#DA##M|#DAKK 1  LLT!#T  ** %&**88 K"--{lE&MJI  %**,%119*eFmL & 1 ) ; YY|, z>z>!<<(2D2DR2LM% ()3 A %$I S="UZZ w0F%GGL A %$I  ( $QQ!-!3!3+#==>OP  #]]: "]]rs6 $' *t_ T]ACUZZ22H<=B091- !$19"b&1A xB (2r6"223.)- @  !@ ||@||@||@,,% @  @@@ \\@0 R!?@ )- D/  !D/ ||D/||D/||D/,,% D/  D/D/D/ \\D/AD/NHUT\HcHeH  LL &) 69  \\ F ||F ||FF F 4< F  F \\F$47 > ||> ||>> > 4< >  \\ >4&*$(&* Q9!"$(Q9 ||Q9 ||Q9 ||Q9||d" Q9 llT! Q9  t# Q9Q9Q9Q9Q9 4<Q9Q9TzQ9 5<<u||U\\ ABQ9h +r23 &*$(&* }9!"$(}9 ||}9 ||}9 ||}9||d" }9 llT! }9  t# }9}9}9}9}9 4<}9}9Tz}9 5<<u||U\\ AB}94}9r$