^j53<ddlmZdZ ddZdZGddeZy) )PushToHubMixincd} t|j}|jj }|dk(r(ddlm}|rddl}t|djy |S#t$rtd|wxYw) z Infers the device type from the model parameters. Args: model: The model instance. Returns: The device type. z { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } zPCannot determine model device, please provide a device to the mapping. Example: cudar)is_torch_availableNversionrocm) next parameters StopIteration ValueErrordevicetypeutilsrtorchgetattrhip)modelEXAMPLE_MAPPINGparamdev_typerrs k/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/utils/kernel_config.py infer_devicers O U%%'( ||  H6.   ui(,,8 O!  ^_n^o p   s A""A:Nc ddlm}|dvrtd||jdd} |jdd} |||| | |||ii||<y)NrLayerRepositoryrr xpunpuneurontpuBOnly cuda, rocm, xpu, npu, neuron and tpu devices supported, got: :)repo_id layer_namerrevisiontrust_remote_code)kernelsrr split) r&r repo_namemodecompatible_mappingrr'r(rrepo_layer_namer%s radd_to_mappingr/:sy( DD]^d]efggooc*1-Oooc"1%G /*!"3   &z"cddlm}ddlm}|dvrt d||j dd}|j dd}||||||ii||<y) Nr)Path)LocalLayerRepositoryrr"r#r$) repo_pathr&)pathlibr2r)r3r r*) r&rr+r,r-r2r3r.r4s radd_to_mapping_localr6Psv, DD]^d]efggooc*1-O$Q'I &y/* &z"r0c6eZdZdZddZ d dZdZdZd dZy) KernelConfigze Kernel configuration class. This class is used to configure the kernel mapping for a model. Nc6||ni|_i|_||_y)N)kernel_mappingregistered_layer_namesuse_local_kernel)selfr:r<s r__init__zKernelConfig.__init__hs"0>0JnPR&(# 0r0c Lddlm} ||| |||||ii|j|<y)Nrr)r%r&r'rr()r)rr:) r=r%registered_namer&rr,r'rr(rs r update_kernelzKernelConfig.update_kernelms< , o#)%#&7  0 O,r0c|jD]+\}}t|ds|j|j|<-y)Nkernel_layer_name) named_moduleshasattrrCr;)r=rnamemodules rstore_registered_layer_namesz)KernelConfig.store_registered_layer_names~sA!//1 MLD&v234:4L4L++D1 Mr0c d}|j|t|jts$t d|dt |j|jj D]t\}}||jjvrt d|dd}t|ttfrd|i}d}t|tr|j D]\}}|}|s|d vrt d |t|tr|\}} t| tst d t | d | d | jddx} t| tst d| | jddx} &| jddx} t d| d| t|tr d|vsd|vst d|d|ft d|d|y)ar Validates the kernel_mapping to ensure that: 1. Each layer_name in the mapping is registered in the model (i.e., the model contains a module with a matching kernel_layer_name). 2. Each kernel value is - either a string of the form 'org/repo:layer_name' or a tuple with the same as string and a dict of {"revision"/"version/trust_remote_code": ...}, - or a dict mapping device types ("cuda", "rocm", "xpu", "npu") to such values as above. 3. Each device key in a dict is one of "cuda", "rocm", "xpu", or "npu". 5. Each trust remote code key must be a bool. 6. Each revision or version key must exist mutually exclusive if it has been passed explicitly. 7. Each repo_name is a valid repository and layer name in the format 'org/repo:layer_name' (i.e., a string containing both a slash and a colon). 8. If a local path is detected, it should be in the format '/abs/path:layer_name', where the absolute path points to the kernel repository, like "/home/user/layer_norm". Args: model: The model instance whose modules are checked for registered kernel_layer_name attributes. Raises: ValueError: If a layer_name is not registered in the model, if a device is not supported, or if a repo_name is not a valid 'org/repo:layer_name' string. a} For single device form remote { "RMSNorm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, For multiple devices form remote { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", "rocm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } You can also pass metadata along to inform about specific kernel information { "RMSNorm": ("kernels-community/layer_norm:LlamaRMSNorm", {"version": 1, "trust_remote_code": True}), ... }, For single device form local { "RMSNorm": "/abs/path:LlamaRMSNorm", ... }, For multiple devices form local { "RMSNorm": { "cuda": "/abs/path:LlamaRMSNorm", "rocm": "/abs/path:LlamaRMSNorm", ... }, ... } z7Kernel mapping must be a dict of the following format: z, got: zLayer z[ is not registered in the model, please register it first using use_kernel_forward_from_hubFNTrr"z^The passed metadata as second entry in a tuple needs to be a dict but found: type(metadata) = z for .r(z6Expected a bool value for `trust_remote_code` but got r'rzExpected valid combination for version/revision (mutually exclusive but one of them) to be passed when passed as tuple, but got revision= z and version= /r#zKernel mapping for 'zl' must be a valid repo name with a layer name (e.g., 'org/repo:layer_name' or '/abs/path:layer_name'), got: z'Kernel mapping must follow the format: ) rH isinstancer:dictr ritemsr;valuesstrtuplegetbool) r=rMAPPING_FORMATr&kernelskip_device_checkrrepor+metadatar(r'rs rsanitize_kernel_mappingz$KernelConfig.sanitize_kernel_mappingss() T ))%0$--t4I.IYY`aefjfyfyaz`{| #'"5"5";";"=. l J!"5U8*A!G# 2:>QSW1XX-e#-.?#F&0&\]n\o$p'"!")1 Z(FFHO'/||It'DDG!U"#-!YNV\Yh_f^i!k# &i5I9MQT\eQe(2:,?klukvw= D!#J>JZZabhai!jkk]. lr0c \ddlm}i}t|}|jj D]\}}|j r |j n |j}|r||jz}t|ttfr||i}|j D]\} } | |k7r |jrt|| | ||(t| tr<| \} } | jdd} | jdd}| jdd}n| } d} d}d}t|| | |||| | ||_y) a Transforms a simple kernel_mapping of the form: { "RMSNorm": ("kernels-community/layer_norm:LlamaRMSNorm", {"version": 1, "trust_remote_code": True}), ... }, or for local path: { "RMSNorm": "/home/user/liger_kernels:LigerRMSNorm", ... }, into a nested mapping: { "RMSNorm": { "cuda": { Mode.INFERENCE: LayerRepository( repo_id="kernels-community/layer_norm", layer_name="LlamaRMSNorm", version=1, trust_remote_code=True, ) } } } or for local path: { "RMSNorm": { "cuda": { Mode.INFERENCE: LocalLayerRepository( repo_path=Path("/home/user/liger_kernels"), layer_name="LigerRMSNorm", ) } } } that's compatible with the kernels library. The device is inferred from the model's parameters if not provided. The Mode is inferred from the model's training state. r)Moder'Nrr(Fr$)rr+r,r-rr'r()r)r[rr:rNtrainingTRAINING INFERENCE TORCH_COMPILErLrPrQr<r6rRr/)r=rcompiler[r-current_devicer&rUr,rrWr+rXr'rr(s rcreate_compatible_mappingz&KernelConfig.create_compatible_mappingsDd !%e,"&"5"5";";"=&  J$)NN4==Dd000&3,/(&1 &   ^+(((VT4I[\dE**.'Ix'||J=H&ll9d;G(0 5H%(P% $I#HG(-%!''9#%&7 ) & P1r0)NF)Nr$F)F) __name__ __module__ __qualname____doc__r>rArHrYrbr0rr8r8cs,1 ot "M slj^1r0r8)r$NF)rrrr/r6r8rgr0rrhs/#%Rjo,&s1>s1r0