^j:dZddlmZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl Z ddl mZddlmZdd lmZmZmZmZmZd d lmZd d lmZej6eZe d GddZej>d(dZ ej>d)dZ!ej>d*d+dZ"e jFjHd*d,dZ%d*d-dZ&ej>d.dZ'd/dZ(d0dZ)d1d2dZ* d3dZ+ d4dZ,d5dZ-d6dZ. d7dZ/ d8dZ0ed d!" d9 d:d#Z1 d;d$Z2 d;d%Z3duSDeepGEMM integration: fused grouped GEMM kernels from `kernels-community/deep-gemm`. Provides: - `deepgemm_bf16_experts_forward`: BF16 M-grouped experts forward. - `deepgemm_fp8_fp4_linear`: end-to-end FP8/FP4 linear (output dtype follows the input). - `deepgemm_fp8_fp4_experts_forward`: FP8 (or FP4 on SM100+) M-grouped experts forward. - `deepgemm_fp8_fp4_megamoe_experts_forward`: FP8xFP4 Mega MoE forward (SM100+). Requirements: CUDA, Hopper (SM90+), CUDA runtime ≥ 12.3, kernels-community/deep-gemm ≥ 2.5 (Mega MoE symbols required). Mega MoE additionally needs SM100+ at call time. ) annotationsN)Callable) dataclass)logging)deprecate_kwarg)KERNELS_MAX_VERSIONKERNELS_MIN_VERSIONis_kernels_availableis_torchdynamo_compilingresolve_internal_import)lazy_load_kernel)to_localT)frozenceZdZUdZded<ded<ded<ded<ded<ded<ded <ded <ded <ded <d ed<y)DeepGEMMz>Curated entry points exposed by `kernels-community/deep-gemm`.rfp8_fp4_matmulgrouped_fp8_fp4_matmul_ntgrouped_fp8_fp4_matmul_nngrouped_bf16_matmul_ntgrouped_bf16_matmul_nnper_token_cast_to_fp8!transform_sf_into_required_layouttransform_weights_for_mega_moeget_symm_buffer_for_mega_moefp8_fp4_mega_moeint m_alignmentN)__name__ __module__ __qualname____doc____annotations__m/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/integrations/deepgemm.pyrr9sIH''''$$$$##'//$,,"**r&rcrtjjdxstjjd}|r|Stjd}|r I77<< >:L ww~~l# l# *q!YYq\  * nn[*..2LMQQR[]_`G"==-bq1LE5u:s5z) )'',,y-8K ww~~k" k" Ja ">I J5;;q>*C A,??? WW\\)Y 9F ww~~f f K "?J K"5;;q>2 #t+lT.Ab-HHH = * * ^4     J J$    K K $    s H7H*/A&H7 I!%I=I J%I460J*H4/H77IIIII10I14I=9JJJctsCtsdtdtdtdStj j sytj j\}}|rdnd}||vr|rdnd }d |d ||d S|d k(rdnd}t}|d|dd |ddStjjtjj|ddsd|d|dd |ddSt}|d|d|dd |ddS||kr/d||d|dd |dd|dd |dd |d!|dd |ddStd"}|y#t|d$d} t|d%d} t|d&d} t|d'd} t|d(d} t!|d)*}t|d+d}t|d,d}t|d-d}t|d.d}t|d/d}d$| fd%| fd&| fd'| fd(| fd)|fd+|fd,|fd-|fd.|fd/|ff Dcgc] \}}|| }}}|r*d0d1j|d2tdtdtd St#| | | | | ||||||3 Scc}}w)4aLoad DeepGEMM once or returns an error message if env or any required symbol is missing. This is wrapped in a function that will raise an `ImportError` with the error message. The reason we raise in the wrapper rather than here is that @functools.cache will only cache a return value, not an exception. `requires_sm100` raises a Blackwell-specific error for callers (FP4 / Mega MoE) that won't work on Hopper, instead of the generic SM90+ message. zUDeepGEMM kernel requires the `kernels` package. Please install a compatible version (z <= version < z), e.g. `pip install kernels==`z9DeepGEMM kernel requires CUDA, but CUDA is not available.)r=) r=zBlackwell (SM100)z"Hopper (SM90) or Blackwell (SM100)zDeepGEMM requires z; current device is SMr;r=) rW)rXNu(DeepGEMM's JIT needs a CUDA toolkit ≥ rrz8, but none was found. Set `CUDA_HOME` to a CUDA toolkit.binr+z:DeepGEMM's JIT compiles with nvcc, but none was found in `u,/bin`. Point `CUDA_HOME` at a full CUDA ≥ z& toolkit (not a runtime-only install).zDeepGEMM found nvcc in `u/bin` but could not read its CUDA version (no parseable `version.json`, `version.txt`, or `include/cuda.h`). Point `CUDA_HOME` at a complete CUDA ≥ z toolkit.zDeepGEMM on SMu needs a CUDA ≥ z toolkit, but nvcc z in `u.` is too old. Point `CUDA_HOME` at a CUDA ≥ z deep-gemmucFailed to load `kernels-community/deep-gemm` — check that a build matches the current torch/CUDA.fp8_fp4_gemm_nt$m_grouped_fp8_fp4_gemm_nt_contiguous$m_grouped_fp8_fp4_gemm_nn_contiguous!m_grouped_bf16_gemm_nt_contiguous!m_grouped_bf16_gemm_nn_contiguouszutils.per_token_cast_to_fp8) chained_pathrrr&get_mk_alignment_for_contiguous_layoutrz-DeepGEMM kernel is missing required symbols: z, z'. Please install a compatible version () rrrrrrrrrrr)r r r r torchr8 is_availableget_device_capabilityr5r,r1r?r>rTrgetattrr r)requires_sm100rNrOallowedarchmin_cudar4 nvcc_versionkernelrrrrrrrrrget_mk_alignmentrnameattrmissings r'_load_deepgemm_kernelrps $ %#%g&'~6I5JJh&'q*  zz&&(Nzz779 u*%w  *8&>bD'v-CE75'QRS S$rk7w"$  :8A;-qRS V55 ww~~bggll9eVDELYKX22:1+a }Lrt )*  *9+6%%-a[M8A;-yJ  ( " w.@! QxXY{m[n?#1\!_$5U9+F$QK=(1+iA  k *F ~tV%6=N '0VX\ ] '0VX\ ]$V-PRVW$V-PRVW3FIfg(/8[]a(b%%,V5UW[%\"#*63QSW#X v'OQUVv'94@  / 35N O 35N O 02H I 02H I *,A B 02S T -/M N +-I J 57G H !1 2   D$ < G";DIIga A %";";553*K'E%A)$&  1sI*ct|y)uWarm the `_load_deepgemm_kernel` cache from an opaque graph node, so Dynamo never traces the loader. Under `torch.compile`, Dynamo ignores `@functools.cache` and traces into `_load_deepgemm_kernel`, whose cold path (hub download + dynamic import via `lazy_load_kernel`) is untraceable and errors under `fullgraph`. `@allow_in_graph` turns the call into an opaque fx node instead — but an fx node's return must be proxyable, and the `DeepGEMM` bundle of Python callables isn't (`Unsupported: torch.* op returned non-Tensor`), so we can't just decorate the real loader. Hence two loaders: this one is opaque, returns `None`, and only warms the cache; the real `_load_deepgemm_kernel` right after is then a plain cache lookup. rfN)rprrs r'_populate_deepgemm_kernelrss 8r&clt|t|}t|tr t ||S)Nrr)rsrp isinstancestr ImportError)rfdeepgemm_or_errors r'load_deepgemm_kernelrys2^<-^L#S)+,, r&cLtjj|ddk\S)z``True`` for Blackwell (SM100+). Cached: device capability is fixed for the process lifetime and this gets hit on every linear/expert forward. rr=)rbr8rddevices r' _is_sm100r} s# :: + +F 3A 6" <> 15`, `>> 7`, `<< 1`) otherwise leak mantissa bits into adjacent UE8M0 byte slots and silently corrupt the SF. SM90 consumes raw fp32 SFs without going through this path. ii)viewrbint32 bitwise_and_float)sfint_views r'_ceil_to_ue8m0r>s<wwu{{#H  & 4 45E F K KEKK XXr&ct|j}|jtjk(r~|;|j d|kr'||j dz}|j |d}|r.|jjtj}n;|j}n*|jtjk(r |r t|}|jdvrtd|jd|s|jS|j d}|j d}d|jz}| |z |z}|jdk(rd |fn||zd |f}t!|j#|k(r|Stj$|j&||j|j } | j)|| S) uLay out `sf` as DeepGEMM's dispatch expects, per arch. On SM100 the int-SF path only *checks* the SF (`tma_stride_check`) and never transforms it, so we hand it a TMA-aligned MN-major layout (`stride(-2) == 1`, `stride(-1) == align(mn, 16/esize)`). On SM90 DeepGEMM transforms SFA itself (`get_mn_major_tma_aligned_tensor`) and only *checks* SFB against `sm90_sfb_check`, which rejects TMA padding (`stride(-1)` must equal `size(-2)`, not `align(mn, …)`); a padded weight SF trips `layout.hpp` whenever `mn` isn't a multiple of `16/esize` (e.g. N=576 → mn=5). So on SM90 we return the raw row-major SF and let DeepGEMM lay it out. Inputs come in three flavors: - `float8_e8m0fnu` on SM100: raw UE8M0 bytes — pack 4 K-bytes → int32 (last dim /4) for the kernel's `(INT, 1, gran_k)` path. - `float8_e8m0fnu` on SM90: SM90 dispatch only accepts FP32 SFs, so cast UE8M0 → FP32 (exact upcast — UE8M0 is the biased-exponent half of a pow-of-2 FP32, so `.float()` rebuilds the original FP32 scale exactly). - `float32`: per-token / per-block SFs from `per_token_cast_to_fp8` or on-disk weights — round to UE8M0 on SM100 (see `_ceil_to_ue8m0`). - `int32`: already-packed UE8M0 — pass through. When `expected_mn` is set and the SF's M-dim is smaller (block-quantized UE8M0, e.g. DSv4-Flash compressor weights with `(N/128, K/128)` SFs), we repeat the SF on the M-axis to per-row before packing — the `(INT, 1, gran_k)` DeepGEMM kernel branch is the only UE8M0 path on SM100; for `gran_mn > 1` the kernel only handles FP32 SFs and would otherwise reject our INT SF here. dim)rrYz"DeepGEMM SF must be 2D or 3D, got Drrrr|)r}r|rrbfloat8_e8m0fnusizerepeat_interleave contiguousrrrrrrrE element_sizetuplestride empty_stridedshapecopy_) r expected_mnis_sm100gran_mnmnkfalign_to aligned_mntarget_stridesouts r'_coerce_sf_for_kernelrKs8#H xx5'''  "rwwr{['@!RWWR[0G%%g2%6B %%ekk2BB U]] "x B  vvxv=bffhZqIJJ }} B BR__&&H3(?#h.J(*A a_BOQPZ;[N RYY[^+   bhhbhhryy YCIIbM Jr&c|jtjk(rddddS| tdt |}|dvrtd|d|jtj k(r|rddddSd dd S) uCPick the `per_token_cast_to_fp8` kwargs from weight dtype + SF dtype + arch. Cases mirror the kernel's recipes: - FP4 weights (`int8`): gran_k=32 packed-UE8M0 SF. SM100+ only. - FP8 weights + UE8M0 SF on SM100: gran_k=128 packed-UE8M0 SF (DSv4). - FP8 weights + UE8M0 SF on SM90: gran_k=128 FP32 SF — the SM90 dispatch in `layout.hpp` only matches FP32 SFs, so we keep act SFs as FP32 (and float the weight SF in `_coerce_sf_for_kernel`; UE8M0 → FP32 is an exact upcast). - FP8 weights + float SF: gran_k=128 float SF (DSv3). T  use_ue8m0gran_kuse_packed_ue8m0z]DeepGEMM requires block-wise quantized FP8 weights, but the experts have no `block_size` set.))r)rru?DeepGEMM requires `block_size` ∈ {(128, 128), (1, 128)}, got r;rF)rr)rrbint8rErr)weightweight_scale_inv block_sizers r'_select_fp8_cast_kwargsrs||uzz!!RTJJ k  z"J//\]g\hhijkk!5!55(!SdKK# ..r&c|j}|jd}tj|j |d|dz j }||zdz |z|z}|t |||dz zz}||z } tjjj| jdd} tj||| |z} |r |jdj } nStj|fd|tj} tj||k|j d| | <| | |fS)aBuild the TMA-aligned grouped layout DeepGEMM expects. Returns `(sorted_to_padded, grouped_layout, total_padded_rows)`: - `grouped_layout` is per-row expert id (Hopper, with `-1` for padding / sentinels) or a cumsum of aligned per-expert counts (Blackwell). - EP sentinels (values == `num_experts`) are routed past the last expert block so DeepGEMM skips them. rr)binsminmax)rrr{rr|r)r|rrbhistcrlongrnn functionalpadcumsumarangefullrwhere) expert_ids_sorted num_experts alignmentuse_psum_layoutr| num_tokenstokens_per_expertaligned_tokens_per_experttotal_padded_rowspadding_per_expertcumulative_paddingsorted_to_paddedgrouped_layouts r'!_build_deepgemm_contiguous_layoutrsL % %F"''*J $5$9$9$;+STZehiZijooq"3i"?!"C !QU^ ^"S[%AYQR]%SS35FF,,001C1J1J11MvV||Jv>ASTeAff299!<@@B%6$8"VSXS^S^_+0;;7H;7VXiXmXmXoqs+t'( ^-> >>r&ctj|g|jdd|j|jd}|||<|S)z;Pad a sorted tensor into the TMA-aligned contiguous layout.rNr)rbemptyrr|r)xrrpaddeds r'_pad_for_deepgemmrsB [[* YQWWQR[ YQRQXQX YF F  Mr&c ||SNr%)x_paddedrs r'&_unpad_from_deepgemm_contiguous_layoutrs $ %%r&c<|jd}|jd}|jd}tj|\} } || |z} || } t | |||\} }}| |k\j d}| j |dz | | | || | ||fS)zSort tokens by expert id and build the M-grouped padded layout. Returns `(sorted_hidden_states_g, sample_weights_g, expert_ids_g, sentinel_mask, perm, sorted_to_padded, grouped_layout, total_padded_rows)`. rr)r)rreshaperbsortr unsqueezeclamp_) hidden_states top_k_index top_k_weightsrrr num_top_k expert_idssample_weights expert_ids_gpermsorted_hidden_states_gsample_weights_grrr sentinel_masks r'_dispatch_routed_inputrs  $I$$R(J"**2.NJ/L$*49+<=%d+ ;\k;;7n&7"[0;;B?MK!O,   r&c t||} | |j| jjdz} | j |dt j |} t j|jd| j| |<| | j|||jdj|S)uRUnpad → weighted multiply → mask sentinels → restore order → top-k reduce.rgrr{rr) rtorr masked_fill_rb empty_likerrr|rsum) out_paddedsorted_weightsrrrrr hidden_dim out_dtyperweightedinv_perms r'_combine_routed_outputr s 1=M NC^&&syy1;;B??H --%H\\$))A,szzBHTN H  " ":y* E I Ia I P S ST] ^^r& output_dtypezv5.16)r9c | td|jtjtjfvrt d|jt |jtjk(}t|||t|j}|jd|jd} |j| fi|\} } tj| jd|jd|j|j} |jdrd d |d fnd} |j!| t#| | j%d f|t#||j%d f| | | j|jdd|jdfz} || j'|| S) uEnd-to-end DeepGEMM linear: per-token activation quant + FP8/FP4 matmul. Static (per-tensor) activation quantization is rejected — DeepGEMM needs per-row SFs. Callers should route static activations through the Triton fallback. Nz@DeepGEMM linear does not support static activation quantization.z7DeepGEMM linear requires FP16 or BF16 activations, got rrrrrrrrr)recipe)NotImplementedErrorrrbbfloat16float16rEryrrr}r|rrrrr.rrradd_)inputrrbiasrractivation_scaledeepgemm cast_kwargsinput_2d qinput_2dscale_2doutput sf_recipes r'deepgemm_fp8_fp4_linearr#s#!"dee {{5>>5==99RSXS^S^R_`aa#6<<5::3MNH)&2BJPYZ_ZfZfPghKzz"ekk"o.H8(88Q[QIx [[+V\\!_U\\Y^YdYd eF2=AS1TA{8,-Z^I  )( q@QRS &'7V[[QR^TU  [[Sb)V\\!_,>> ?F  D Mr&c b|jtjk7rtd|jt }|j r |j n |j}|j}|jd}|jd}|jd} t||||j|jt|\} } } } }}}}t|jr |j n |j"}t|j$}|j&r-t|jr |j(n |j*nd}|j&rt|j,nd}|j r|j.dn|j.d}t1| ||}tj2||||j}|||||t||j&r|j5d||| |jr|j7|n|j9|}tj2|| ||j}|||||t||j&r|j5d||| t;|| | ||||| |j S)N;DeepGEMM experts path requires bfloat16 hidden states, got rrrr)r)rrbrrEry is_transposedrrr|rrrrr}rhas_gate gate_up_projup_proj down_projhas_biasgate_up_proj_bias up_proj_biasdown_proj_biasrrr index_add_ _apply_gateact_fnr)selfrrrrgrouped_bf16_matmulr|rrr sorted_hiddenrrrrrrr weight_up weight_downup_bias down_bias up_out_dimactproj_outrs r'deepgemm_bf16_experts_forwardrMsj enn,VWdWjWjVklmm#%H=A=O=O(99U]UtUt  ! !F  $I##A&J##B'J {M43C3CXEYEY[dek[l    dmm**NI4>>*KZ^ZgZght--DDUDUVmqG15,,-DI)-(:(:$ PQ@RJ M+;=N OC{{,j}ObObcHY.R[\bRcd }}A/1FG-1]]t) H@UH ++'F-J]J] ^C+sNT]^dTef }} q*Il,CD !    r&c |jr tdt|j|jdk(r t d|j tjk7rtd|j t|jj tjk(}|jr |jn |j}|j }|j#d}|j#d}|j#d} t%|j&r |j(n |j*} t%|j&r |j,n |j.} t%|j} t%|j} t1| | |j2t5|}t7||||j8|j:t5|\}}}}}}}}|j=drd d |d fnd}|j>|fi|\}}tA|||}tA|||}tjB|| jDd |tj }||tG|| f| tG| | j#d  f|||t5||j&r|jI|n|jK|}|j>|fi|\}}tjB|| |tj }||tG|| f| tG| | j#d  f|||t5|tM|||||||| |j S)NzDeepGEMM experts selected on a model spanning multiple CUDA devices in one process; its kernels are bound to a single CUDA context and corrupt across devices. Use `experts_implementation='grouped_mm'`, or run one device per process (TP/EP).staticzJDeepGEMM experts dispatch does not support static activation quantization.rrrrrrrrrrr)rr)'_deepgemm_disabled RuntimeErrorrdown_proj_scale_invactivation_schemerrrbrrEryr rr rrr|rrr r r gate_up_proj_scale_invup_proj_scale_invrrr}rrrr.rrrrrrrr)rrrrrgrouped_fp8_fp4_matmulr|rrrrweight_scale_uprweight_scale_downrrr _expert_ids_grrrrrract_fp8 act_scalesrproj_fp8 proj_scalesrs r' deepgemm_fp8_fp4_experts_forwardr0sG   \  #4#;#;< )!"nooenn,VWdWjWjVklmm#4>>3G3G5::3UVH.2.@.@**hFhFh ! !F  $I##A&J##B'Jdmm**NIdmmt::QUQgQghO4>>*K !9!9:))_dooW`agWhiK {M43C3CXEYEY[dek[l   2=AS1TA{8,-Z^I9(88V+VGZ)9;LMG":/?ARSJ{{,iooa.@W\WeWefH ' @QRS )/y~~VXGYZ[!&) .2]]t) H@UH;H::8S{SHk ++'F%.. YC (BSTU +,=;K[K[\^K_`a !&)  "    r&cPtd}t|jj}t|jj}t|j jj tjj}t|jjj tjj}|j}|j}|j}|dzdk7s|dzdk7rtd|d|d|j|j!d|z|d | } |j|j!||d | } |j#|| f|| f\\} } \} } tj$j'| d |_tj$j'| d |_tj$j'| d |_ tj$j'| d |_y )u!One-shot pack + permute of an FP8Experts module's L1/L2 weights into the Mega MoE UTCCP layout. Called lazily on the first megamoe forward; idempotent via the caller's ``_megamoe_transformed`` flag. Steps: 1. Cast UE8M0 SF → FP32 and call ``transform_sf_into_required_layout`` → packed int32 in MN-major TMA-aligned layout. 2. Run ``transform_weights_for_mega_moe``: interleaves gate/up on L1 and transposes both SFs for UTCCP. 3. Overwrite the loader-side parameters in place; the interleave preserves the ``[E_local, 2*I, *]`` leading dims so downstream ``.size(...)`` reads stay valid. Unwraps any ``DTensor`` wrappers FSDP2/EP may have placed around the loader- side Parameters — the kernel takes raw pointers. TrrrrzwDeepGEMM Mega MoE requires `hidden_dim` and `intermediate_hidden` divisible by 32 (FP8 SF granularity); got hidden_dim=z, intermediate_hidden=r;r)rr)r num_groupsF) requires_gradN)ryrr&datar$r rrbrrr intermediate_dimrrrErrrr Parameter) modulergate_up_sf_raw down_sf_raw gate_up_wdown_wintermediate_hiddennum_local_expertsr gate_up_sfdown_sfgate_updowns r'setup_megamoe_weightsrBs"$48Hf;;@@AN655::;K,,11277 CNNPI f&&++ , 1 1%** = H H JF 11**""JB!2R71<44>PTU  44 ## $$r&)returnz str | None)rTtuple[int, int] | None)F)rfboolrTzDeepGEMM | str)rfrVrTNone)rfrVrTr)r|z torch.devicerTrV)r torch.TensorrTrW)rrXrTrXr)rrXrz int | NonerTrX) rrXrrXrz tuple | NonerrVrTdict) rrXrrrrrrVrTz&tuple[torch.Tensor, torch.Tensor, int])rrXrrXrrrTrX)rrXrrXrTrX)rrXrrXrrXrrrrrrVrTr)rrXrrXrrXrrXrrXrrrrrrrz torch.dtyperTrX)NNNN)rrXrrXrrXrtorch.Tensor | NonerrUrztorch.dtype | NonerrZrTrX) rtorch.nn.ModulerrXrrXrrXrTrX)r7r[rTrW) rr[rrXrrXrrXrOz%torch.distributed.ProcessGroup | NonerTrX)6r# __future__r functoolsrAr,rGr/collections.abcr dataclassesrrbutilsrutils.deprecationrutils.import_utilsr r r r r hub_kernelsrtensor_parallelr get_loggerr loggerrcacher5rTrp_dynamoallow_in_graphrsryr}rrrrrrrrrrrr0rBrSr%r&r'rjs[ # $! /*%   H %  $, $ ++\ ppf 9 9 ==, Y:z/ /,8/FR/^b/ />?#?25?BE?X\?+?D&/// / /  /  / /d__ _ _  _ # _  _____41 !%)-'+,0& & &#&  & ' & % &*&&2&R> >>> >  >BY YYY Y  Yx7R~<@ S% S%S%S% S% 9 S%  S%r&