^j ddlZddlmZmZmZddlmZejeZ eddZ eddZ eZ eZ d ejd ed ejfd Zd ejdzdejdejd efdZdejd ejdzdedejdejd ejf dZ ddej(j*dejdejdejd ejdzdededzdedzdejdzd eejdffdZy)N)is_torch_npu_availableis_torch_xpu_availablelogging)is_torch_greater_or_equalz2.5T) accept_devz2.8 hidden_statesn_repreturnc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) N)shapeexpandreshape)r r batchnum_key_value_headsslenhead_dims s/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/integrations/sdpa_attention.py repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTattention_maskkeyvaluectrtStxr0|duxr*|jd|jdcxk(xrdkScS)N)_is_torch_xpu_available#_is_torch_greater_or_equal_than_2_8#_is_torch_greater_or_equal_than_2_5r)rrrs ruse_gqa_in_sdpar!sI22 . u>T3I uciiXZm_d_j_jkm_nNuruNuuNuur position_bias is_causalqueryctj|jj}||r|j}|j d|j d}}tj ||dddftj ||dddfk\} | jdd||} tj| ||} | S|} | Stj|||} | S)a  Create a floating-point dtype mask to use with sdpa. The mask contains the values of `position_bias` to positions where we should attend to tokens, and -inf where we should not. It will be added to the QK^T result in the attention, before the softmax. Note that using such a mask will usually prevent sdpa from dispatching to the most efficient kernel implementations. Note that we cannot create this in advance when we create the mask in the model, as the position_bias is usually learned differently in every layer. Nr)devicer ) torchfinfodtypeminr&rarangeviewwhere) r"rr#r$r min_dtyper&q_length kv_length causal_maskposition_bias_masks rcreate_position_bias_maskr3)s CII&**I ZZF"'++a.#))A,iH Xf5ag>%,,yagBhimopipBqq &**1a9EK!&[-!S   "/  #[[ R rmoduledropoutscalingc  *| jddrtjdi} t|drM|jdkDr>t |||s-t ||j}t ||j}nddi} |jd} |jd} ||n t|d d}| dkDxr|duxr|}tjjr*t|tjr|j}tr[|Y|j tj"k7r*>%***L"..~/B/B/DEHHVN^+1 XAU!Q  1$%aIXIq()  $)!Q9H9*<=M 2=.R[]bdgh ((%%BB    !   K''1-88:K  r)gNNN)r'utilsrrrutils.import_utilsr get_logger__name__r?r rrrHrFintrrIr!r3rLModulefloattuplerTrrr^s KK:   H %'@RV&W#&?RV&W#0202 UU\\ U# U%,, U vELL4$7 vell vSXS_S_ vdh v#<<#LL4'## << #  #  \\ #X !)-W HHOOW <<W W << W LL4' W  WT\Wd{W<<$&W 5<< Wr