^j 8UddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZd dlmZe rddl m!Z!ddl"m#Z#ddl$m%Z%er ddl&Z&ddl'm(Z(ejRe*Z+dedededZ,ejZj]ddj_Z0e0evZ1erUddl2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;ddl2mZ?ddl2m@ZAddl2mBZCdeDfdZ@deDfd ZBdaEeFdzeGd!<d"eFfd#ZHd"eFfd$ZIdSd%ZJn/d&Z1d'Z@d(ZBGd)d*Z6Gd+d,Z7Gd-d.Z5d/Z:d0Z9d1ZJd2d d3d4d d3d4d d3d5d6d3d7dd3d8d9d:d;ZKeFeDeFeDeDffeGd<<iZLeFeDe dzfeGd=<d>eDdzd"eMfd?ZN dTd>eDd@edzdAeMd"e dzfdBZOeLfdCeDdDeFeDe dzffdEZPdSdUdFZ> dVdCeDdGeDdzdHeQeDzdzdAeMd"e f dIZz-use_kernel_forward_from_hub..[ss)_kernels_enabled$_kernels_use_kernel_forward_from_hublogger warning_once_TRANSFORMERS_USE_HUB_KERNELS)r)s r1r'r'Ts2 7 C C   XYvXwx # "r3 func_namecbtr t|StjdtdS)Nr+c|Sr-r.funcs r1r2z*use_kernel_func_from_hub..dsr3)r4!_kernels_use_kernel_func_from_hubr6r7r8)r9s r1r(r(]s2 4Y? ?   XYvXwx % $r3_KERNEL_MAPPING_CACHEreturnc dtddditdtddtd d d d idtjtdddtj tdddiidtjtdddtj tdddiidtj tj ztdddtjtj ztdddiidtj tj ztdddtjtj ztdddiidtjtj ztdddiitjtdddtj tddditjtdddtj tddditj tddditj tddditjtdddtj tdddidtjtdddtj tddditjtdddtj tddditj tddditj tdddid dtj tj ztd!d"diidtj tj ztd!d#diidtj tj ztd!d$diidtj tj ztd!d%diidtj tj ztd!d&diidtj tj ztd!d'diid(}tj td)d*d+itd)d*d+tj td,d*d+id-dtjtj ztdd.d+iid/}||z}|S)0Ncudaz!kernels-community/deformable-detrMultiScaleDeformableAttentionr)repo_idr)versiony)min_capabilitymax_capability)type propertieszAtlas-Inference/gdnQwen3_5GatedDeltaNet(ef12347fc77d6ddf1cb72c0bd0af1c7d6cc69172T)rDr)revisiontrust_remote_codekernels-community/mamba-ssmcausal_conv1d_fncausal_conv1d_updatezkernels-community/liger-kernelsLigerSwiGLUMLPrLigerTiledSwiGLUMLP LigerGEGLUMLPLigerTiledGEGLUMLP LigerLinear LigerRMSNormzkernels-community/rmsnormRMSNormzkernels-community/mlx_rmsnorm)rBrocmxpumpsnpuzkernels-community/megablocksMegaBlocksMoeMLPCPUMegaBlocksMoeMLP)rBrYrZcpuzkernels-community/activationFastGELU QuickGELUNewGELUSiluGeluGeluTanh)rCrKrPrQ SwiGLUMLPGeGLUMLPLinearrXr]r`rarbSiLUGeLUrezkernels-community/rotaryapply_rotary_transformers)rDr9rEzkernels-community/aiter-rope)rZrBrYLigerForCausalLMLoss)rotary_pos_embForCausalLMLoss)rrrr!TRAINING INFERENCE TORCH_COMPILEr)_KERNEL_MAPPING_FUNCTION_KERNEL_MAPPINGs r1_build_kernel_mappingrtls?>."-SQTU#15G&*  %MM? =#5 !$ NNO =#5 !%  !MM? =#9 !$ NNO =#9 !%  %NNT%7%77 A#3 !: MMD$6$66 A#8 !9  NNT%7%77 A#2 !: MMD$6$66 A#7 !9  MMD$6$66 A#0 !9MM? A#1 !$ NNO A#1 !% MM? A#1 !$ NNO A#1 !% NNO ;#, !%NNO ?#, !%MM? A#1 !$ NNO A#1 !% Q4nMM? >#5 !$ NNO >#5 !% MM? >#5 !$ NNO >#5 !% NNO >#5 !%NNO >#8 !%A'!RNNT%7%77 >#- !:NNT%7%77 >#. !:NNT%7%77 >#, !:NNT%7%77 >6[\:NNT%7%77 >6[\:NNT%7%77 >:_`:Iki bNNN :Fakl% '6B]ghNNN >Jeop% MMD$6$66 AMcmn9 !$  0*,DDr3c.t tatS)zPReturn the default transformers kernel mapping, building it lazily on first use.)r?rtr.r3r1get_kernel_mapping_transformersrvws ! ($9$; !$$r3c2| t}t|yr-)rvr")mappings r1$register_kernel_mapping_transformersry~s ?57G(r3Fc d}|S)Nc|Sr-r.r/s r1 decoratorz.use_kernel_forward_from_hub..decoratorsJr3r.argskwargsr|s r1r'r's r3c d}|S)Nc|Sr-r.r<s r1r|z+use_kernel_func_from_hub..decoratorsKr3r.r}s r1r(r(s r3ceZdZdZdZy)rctd)NzNLayerRepository requires `kernels` to be installed. Run `pip install kernels`. RuntimeErrorselfr~rs r1__init__zLayerRepository.__init__sop pr3ctd)NzMLayerRepository requires `kernels` to be installed. Run `pip install kernels.NotImplementedErrorrs r1loadzLayerRepository.loads%&uv vr3N__name__ __module__ __qualname__rrr.r3r1rrs q wr3rceZdZdZdZy)r ctd)NzSLocalLayerRepository requires `kernels` to be installed. Run `pip install kernels`.rrs r1rzLocalLayerRepository.__init__stu ur3ctd)NzRLocalLayerRepository requires `kernels` to be installed. Run `pip install kernels.rrs r1rzLocalLayerRepository.loads%d r3Nrr.r3r1r r s  v r3r ceZdZdZy)rctd)NzMFuncRepository requires `kernels` to be installed. Run `pip install kernels`.rrs r1rzFuncRepository.__init__sno or3N)rrrrr.r3r1rrs pr3rctd)Nz^replace_kernel_forward_from_hub requires `kernels` to be installed. Run `pip install kernels`.rr~rs r1r#r#s l  r3ctd)NzVregister_kernel_mapping requires `kernels` to be installed. Run `pip install kernels`.rrs r1r"r"ssttr3ctd)Nzcregister_kernel_mapping_transformers requires `kernels` to be installed. Run `pip install kernels`.rrs r1ryrys q  r3zkernels-community/causal-conv1d)rDrErOz!kernels-community/finegrained-fp8zkernels-community/deep-gemmzkernels-community/sonic-moez ep-support)rDrM)z causal-conv1dz mamba-ssmzfalcon_mamba-ssmzfinegrained-fp8z deep-gemmz sonic-moe_HUB_KERNEL_MAPPING_KERNEL_MODULE_MAPPINGattn_implementationc>|duxrtjd|duS)zJCheck whether `attn_implementation` matches a kernel pattern from the hub.Nz'^[^/:]+/[^/:]+(?:@[^/:]+)?(?::[^/:]+)?$)research)rs r1 is_kernelrs- 4' c II@BU V^b br3attention_wrapperallow_all_kernelscddlm}ddlm}d|vr|j ddn|}t |syt sttd|vr%|j d\}}|j}n|}d}|j}|jd\}}} |j}| r| jnd} d} | t|d } | rd nd} t|| | | } d}t| dr |t }|}n)t| drddlm}||n|}d}n| t'| |}|j)||j)|||| S#t$rt$r} td |d| dd} ~ wwxYw)a Load and register the kernel associated to `attn_implementation`. Args: attn_implementation: A string, usually a kernel repo like "kernels-community/flash-mla". attn_wrapper: a callable for the wrapper around the attention implementation. In `transformers` we have a wrapper around the `flash_attn_var_len` call, and the same goes for `sdpa` and `eager`. They just prepare the arguments properly. This is mostly used for continuous batching, where we want the `paged` wrapper, which calls the paged cache. allow_all_kernels (`bool`, optional): Whether to load kernels from unverified hub repos, if it is a custom kernel outside of the `kernels-community` hub repository. r)ALL_MASK_ATTENTION_FUNCTIONS)ALL_ATTENTION_FUNCTIONS|rN:@r)"requested_attention_implementationrErrMrErz-An error occurred while trying to load from 'z': .flash_attention_2flash_attn_varlen_funcsparse_atten_func)msa_attention_forwardsdpa) masking_utilsrmodeling_utilsrsplitrr ImportError_MISSING_KERNELS_MESSAGEstrip partitionrr% ValueError Exceptionhasattrr msa_attentionrgetattrregister)rrrrractual_attn_namerD kernel_name_revrEis_fa4kernelemask_implementationkernel_functionrs r1load_and_register_attn_kernelrs =8.attach_hidden_kernelsQs(92>DDF 1HD"4 5 5 788!"bii0$=dVDBB &&tR0 1r3cZt|diD]}t||st||y)Nr)rrdelattr)rrs r1detach_hidden_kernelsz(kernelize..detach_hidden_kernels\s1F$5r: &Dvt$%  &r3NrBrY)rI)inherit_mapping)devicemodeT)rrrapplytrainingr!rprorrIrr kernel_configuse_local_kernelr$kernel_mapping_kernels_kernelize _use_kernels)modelrrr device_typerrs r1r&r&Ls   !233 1&+ )*%*^^t~~$,\`ll'' & %5%7 K[)    *"'"5"5"F"FFO#E$7$7$F$FXgh D"5dC D D uV$ ?! )* D D  )*s$B8D$D(D$D!D$$D7rMrEclddlm}tsttd||d}t |||||S)Nr) __version__ transformers) frameworkrErD)rMrE user_agentrN)rrrrget_kernel_hub)rrMrErrrs r1r%r%ws?   !233-+R]^J hJbs r3 module_namesc8ttrgfd}|S)aO This decorator attaches the target function within the module as a plain attribute (not as a submodule). Keep in mind that this registration is only meant for `kernelize` to recognize its target modules (i.e. function exchanged for a weightless `nn.Module` with the same forward) to then exchange to the kernel variation (in-place) if the conditions are met. We cache each of these function-based registrations: After proper registration and exchange it is removed from the module's `_modules` dict as it does not really act as `nn.Module` but a base function. c<|jfd}||_|S)Nc|g|i||jjdi}D]B}t|ddxst|ddxs t|dd}|td||||<Dy)Nrrkernel_layer_namer9z)Could not infer kernel function name for )__dict__ setdefaultrr)rr~rhidden_kernelsrrr orig_inits r1new_initz8use_kernelized_func..decorator..new_inits d ,T ,V ,"]]556GLN" *B D16r#6=6r;5 <$'PQSPV%WXX(*t$ *r3)r)r0rrrs @r1r|z&use_kernelized_func..decoratorsLL  *"   r3)rr)rr|s` r1use_kernelized_funcr s#,)$~ . r3c#,K dadday#dawxYww)z Context manager used to adjust the value of the global `ALLOW_HUB_KERNELS`. This is needed, as this argument cannot be forwarded directly to the `__init__` of the models, where we set the attention implementation. TNF)rr.r3r1allow_all_hub_kernelsr s""  "Es  parent_cls child_names kernel_clsc|jfd}td|j|fd|i}d|j|_|S)a6 Create a new class that inherits from `parent_cls` and fuses the child modules specified in `child_names with the provided `kernel_cls`. The first child in `child_names` will be replaced with the `kernel_cls`, and the rest will be replaced with `nn.Identity()` to keep the same interface. c|g|i|Dcgc]}t||}}|}t|d|ddD]!}t||tj#ycc}w)Nrr)rsetattrrIdentity) rr~rrchildrenkernel_instancer r original_inits r1 patched_initz9make_parent_class_for_kernel_fusion..patched_initsvd,T,V,4?@DGD$'@@$h/k!no6O /D D$ . /AsA&Fusedr)rrIrr)r r rr patched_clsrs `` @r1#make_parent_class_for_kernel_fusionrsW''M/z2234zmjR^E_`K!&z'>'>&?@K r3c  "t|drt|jdstd|jd|jj}i}i}d}|j j D]J\}}t|ttfrd|i}t|tr*t|jdk7rtd|td|d|tt|j}t|trY|\} } | jd d} | jd d} | jd d xst } | d | ind | i} | d | iz} | | f}n|} dt d } | | f}| j#d\}}}|r|std| d||j$rt't)||}nt+d)||d| }|j-}|td|t.j0j|j2}|rt5||jddnd}|:d|j6vr,t9j:|j<d}||_t|tr0||||<|j|_|||<|||j<nt|tr|&td|jd|jd|j|_|Dcgc]}|d }}|Dcgc]}|jAddd}}ttC|dk7rtd||djEdd}|Dcgc]}|jAddd}}|&tGjHd5||}dddd }|jKD]u\}"tMjN||stQ"fd|Dstd |d!|d"|jd#}tS"}tU|||||j<w|std$|d%|jd&tW|d#'t|d(r5tY|} t[|j\}!| | |!z}!t_||!d#'|||j<M||_ycc}wcc}wcc}w#1swY)xYw)*N config_class model_typezModel z# has no config_class or model_type.rzLExpected exactly one kernel repo regardless of device/mode specificity, got zInvalid hub repo z for layer rMrErNF)rErNrzInvalid kernel repo string ) repo_pathr))rDr)z*Could not load kernel class from hub_repo=Layoutforwardcyr-r.rs r1 _noop_forwardz?register_kernel_replacements_and_fusions.._noop_forward&sr3z Fused kernel z* requires a companion layout class named 'zLayout' in the same module.rrzGAll patterns for a fused kernel must share the same parent module, got *z\w+metac36K|]}t|ywr-)r).0childrs r1 z;register_kernel_replacements_and_fusions..WsKe7651KszModule z* does not have the expected child modules z required for the fused kernel TzNo module matched pattern z for fused kernel z3. Provide the full dotted path from the model root.) overwriteconversion_mappingr.)0rrrrrrrrstrtuplerlenvaluesnextiterrrrrr rrrsysmodulesrrr functoolswrapsrrrsplitsetrtorchr named_modulesr fullmatchallrIrr r listr)r )#r0configrr patch_mapping new_mapping meta_modelr)hub_reporepo_strmetadatarMrErN final_reporDrlayer_name_in_reporepor kernel_mod layout_clsr!item glob_patternspparent_patternsparent_patternr  matched_anyr module_clsexisting transformsrs# @r1(register_kernel_replacements_and_fusionsrPsq 3 'ws7G7G/V6#,,/RSTT!!,,J%'MKJ - < < B B DC6 H he -h'H h %8??$%* bckblm0 K ~VW WX__./0 h &!) Hh||J5Hll9d3G ( -@% H ]L] /6/B 7+U]H^H ,.?@ @H"H-JH#$;LMH"H-J)1););C)@&&0:8,kR\Q_`a a  ) )'w--D #-D YY[  I(VW W[[__Z%:%:; R\WZJ,?,?+@)GNbf  !iz7J7J&J __Z// 0 1 "/J  j# &!*4 J',6+>+>J ((2M* %0:K ++ , E *! #J$7$7#:;(1122MO ,6+>+>J (1;<T!Wr*r?r__annotations__rtrvryrrboolrrrintr:r rr rIrrP__all__r.r3r1rgsN $% j4185602   H %N+>*??]^q]rrsu !# /@% H N N P04HH   V##%C%*.4$;-I4IV%T%)   wwpp u "CqQ:qI$AaP#FSTU:qI: U 2T#tCH~-.8:S*t"33493:$diKK19DK\`K$K\Pf1 #1 S*t:K5K0L1 h(+Z $# Dj3Y    "$d8nx&?$P " " c 4U/ U/ U/"U/ U/p r3