^j+ddlZddlmZddlmZej eZdZdZ dZ da de fd Z ejjd d d ej dej dej dej dej dej dededededededede dej fdZej(dZdej deddfd Zd!Z d(d"ej0j2dej d#ej d$ej d%ej dzdededzd&ej dzdeej dffd'Zy))N)logging)sdpa_attention_forward) attn_implementationcttSddlm}|jdd}|j d\}}}|||xsd|rdndd }d D]*}t t ||drtd |d |d |atS)awLoad the MSA hub kernel once and verify the expected callables are present. The ``attn_implementation`` string may carry a ``paged|`` prefix and/or an ``@`` pin (e.g. ``kernels-staging/msa@v0``); the build currently lives on the repo's ``v0`` branch. The loaded module is cached in a module-level global so registration happens once, not per call. Nr) get_kernel|@rT)revisionversionallow_all_kernels)sparse_atten_func build_k2q_csrzThe MSA kernel loaded from `z` does not expose a callable `zK`. Make sure you request a compatible build, e.g. `kernels-staging/msa@v0`.) _MSA_KERNEL hub_kernelsrsplit partitioncallablegetattr ImportError)r rrepo_id_revkernelfn_names r/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/integrations/msa_attention.pyload_and_register_msa_kernelr$ s'!'',R0G'',OGQ #+stPQei jF967.wi7UV]U^_[[ K ztransformers_msa::sparse_atten) mutates_argsqkvq2k cu_seqlens_q cu_seqlens_ktopk block_sizetotal_k max_seqlen_q max_seqlen_k qheads_per_kvscalingimplreturncFt| }tjj|j5|j |||||| | | \}}|j ||||||||| | |d|  }ddd|j S#1swYj SxYw)a$Opaque wrapper around the CuTe-DSL CSR build + block-sparse kernel. Registered as a ``torch.library`` custom op so ``torch.compile(fullgraph=True)`` treats the whole CSR-build + attention as a single opaque node (no graph break) and ``reduce-overhead`` CUDA graphs can capture it. The internal ``build_k2q_csr`` output is data-dependent in shape, but it never escapes this op (only the fixed-shape ``[total_q, Hq, D]`` attention output does), so the fake/meta impl below is exact. The op is functional (no input mutation). )r0r2r1 qhead_per_kvT)r,r-r1r2blk_kvcausal softmax_scaleN)r$torchcudadevicerr contiguous)r(r)r*r+r,r-r.r/r0r1r2r3r4r5msa k2q_row_ptr k2q_q_indices attn_outputs r#_msa_sparse_atten_oprD<s2 't ,C   188 $ %(%6%6    %%&&7 & " ]++   %%%%!,  4  ! ! ##5 4  ! ! ##s ;BB c,tj|SN)r< empty_like)r(r)r*r+r,r-r.r/r0r1r2r3r4r5s r#_msa_sparse_atten_fakerHus$   A r%querydropoutc|jjdk7s/tjj |jddk7r t d|j dtk7rtdtd|jjtk7rtdtd|d k7r td |jj}|tvrtd td |d y)aValidate kernel capability, dropout and configured topk once per attention module. Mirrors the flash-attention integration, which checks capability/dropout at model init rather than on every forward. The check is cached on the module so the hot path never re-runs it. There is no SDPA fallback: a sparse layer either runs the MSA kernel or this raises. Serves both prefill (q_len > 1) and single-token decode (q_len == 1) -- decode is just a varlen call with one query slot, so there is no context-length threshold. r=r zMSA block-sparse attention requires an SM100 / Blackwell CUDA device. Select a different `attn_implementation` on unsupported hardware.rz2MSA block-sparse attention only supports head_dim .z4MSA block-sparse attention only supports block_size zYMSA block-sparse attention does not support attention dropout; set `attention_dropout=0`.z1MSA block-sparse attention only supports topk in z, got `z0`. Set `index_topk_blocks` to a supported value.N)r>typer<r=get_device_capability RuntimeErrorshapeMSA_SUPPORTED_HEAD_DIM ValueErrorindexerr/MSA_SUPPORTED_BLOCK_SIZE topk_blocksMSA_SUPPORTED_TOPK)modulerIrJr.s r#_validate_msa_initrZs ||F"ejj&F&Fu||&TUV&W[]&] P   {{200MNdMeefghh ~~  $<<OPhOiijkll#~tuu >> % %D %%?@R?SSZ[_Z`a< <  &r%c|j\}} } } |jd|jd} } | | z}|jdtfdtD}|k7rB|jg|jdd|z d}t j ||gd}||j ddj|| z| | j}|j ddj|| z| | j}|j ddj|| z| | j}t jd|dz| z| |jtj}|dk(r||z|ddzjtjjd}t j t jd|jtj|g}n8t jd|dz| z| |jtj}|jtj}|jddddj| || zj}t||||||||| z| | |||j j"}|j|| | | S) Nrrrc3.K|] }|k\s |ywrFr&).0tr.s r# z$_sparse_attention..sBQT qBs )dimr)r>dtype)rRnextrXnew_fullr<cat transposereshaper?aranger>int32tozerospermuterDconfig_attn_implementation)rYrIkeyvaluer4 block_indicesr/cache_positionbsz num_q_headsq_lenhead_dim num_kv_headsk_lenr3 padded_topkpadr(r)r*r,valid_kr-r+rCr.s @r#_sparse_attentionr|s(- %CeX))A, ! %L</M   r "DB"4BBKd$$%T}':':3B'?%TtAS%TVXY =#"6B?  1%%cEk;ITTVA a##C%KxHSSUA 1%%cEk<JUUWA<<C!Gu#4eAHHTYT_T_`L  axN.!"%)--ekk:BB1Eyy%++a "TV]!^_ ||Aa5'8%X]XcXcd   5;; 'C ++aAq ! ) ),e T J U U WC'      e    **K   sE; AAr%rYrorpattention_maskrqc ||jddz}|t|||||f||d|St|ddst|||d|_|j j } |jd} t||||||| | } | dfS) zn TODO: this opens a door to per-layer attn implementation which is something we might want lalter on. Nrg)rJr4_msa_validatedFTrr) rRrrrZrrUr/getr|) rYrIrorpr}rJr4rqkwargsr/rrrCs r#msa_attention_forwardrs++b/T)% E3~ ?FPW [a  6+U 365'2 $**JZZ 01N#FE3w WacqrK  r%)NrNNN)r<utilsrsdpa_attentionr get_logger__name__loggerrXrVrSrstrr$library custom_opTensorintfloatrD register_fakerHrZr|nnModuletuplerr&r%r#rs 2   H %$ c89K5$ ||5$ ||5$ ||5$  5$ ,, 5$ ,, 5$ 5$5$5$5$5$5$5$ 5$ \\5$L5$p##$( ell U t :<BH+/ )-  HHOO  <<    <<  LL4'   T\ <<$&  5<<  r%