^j ddlZddlmZmZddlmZej eZeZ dejdejjdejfdZ ddejjdejd ejd ejd ejdzd ed edzdedzdedzdedzdejdzdeejdffdZy)N)_flash_attention_forward!flash_attn_supports_top_left_mask)loggingquerymodulereturnc~|jtjk(r|jj}tj |rtj |St|jdr|jjStd|jDjjSy)ziIf the query is in float32, return a target dtype compatible with flash attention. Return None otherwise. _is_quantizedc3jK|]+}t|tjjs(|-yw)N) isinstancetorchnnLinear).0layers t/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/integrations/flash_attention.py z#get_target_dtype..s&b%z%QVQYQYQ`Q`?abs)33N) dtyperfloat32devicetypeis_autocast_enabledget_autocast_dtypehasattrconfignextmodulesweight)rr device_types rget_target_dtyper! s {{emm#ll''  $ $[ 1++K8 8 V]]O 4==&& &b6>>+;bbiioo o keyvalueattention_maskdropoutscalingsliding_windowsoftcap is_causals_auxc  8| jddrtjd|jd} t d|jDr t d|j dd}|j dd}|j dd}t||} | | n |j} t||||f| | ||||t| |jjt|dr |jnd| | j|j ndd | }|dfS) Noutput_attentionsFzFlash Attention does not support `output_attentions=True`. Please set your attention to `eager` if you want any of these features.rc3&K|] }|dk( yw)rN)rdims rrz*flash_attention_forward..1s +3!8 +szTensor query has shape with a zero dimension. FlashAttention does not support inputs with dim=0. Please check your input shapes or use SDPA instead. layer_idx) query_lengthr*r& softmax_scaler(r)use_top_left_mask target_dtypeattn_implementationr2r+)getlogger warning_onceshapeany ValueError transposer!r*r_use_top_left_maskr_attn_implementationrr2tor)rrr#r$r%r&r'r(r)r*r+kwargsseq_lenr6 attn_outputs rflash_attention_forwardrEs9zz%u- W kk!nG +u{{ ++ B  OOAq !E --1 C OOAq !E$E62L'2 8H8HI*     %,!"MM>>&-fk&B&""  HHU[[ !%( )K.  r")gNNNNN)rmodeling_flash_attention_utilsrrutilsr get_logger__name__r9r?TensorrModulerr!floatintbooltuplerEr/r"rrPs3 h   H %68 ELL %((// ekk ( !% !!%C HHOOC <<C C << C LL4' C  CT\C$JCT\Cd{C <<$ C 5<< Cr"