^j3/ddlmZmZmZddlZddlmZddlmZddl m Z ddl m Z ddl mZgd Zej j"e dd ej$d eej$fd Zej j"e dd edej$d eej$dedeej$f dZGddej.ZGddej.Zy))FinalOptionalTypeN)nn) functional)register_notrace_function)use_fused_attn)apply_rot_embed_cat) Attention AttentionRopemaybe_add_maskresolve_self_attn_maskscores attn_maskc||S||zSN)rrs `/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/layers/attention.pyrrs&6>FY,>>seq_lenattn is_causalreturnc.|r.|j||ftdjd}|S|d}|S|jtj k(r?t j ||j}|j|td|S|}|S)Nz-infr)dtype)new_fullfloattriu_rtorchbool zeros_like masked_fill_)rrrr attn_biass rrrsMM7G"4eFmDJJ1M      EJJ &$$YdjjA  z5=9  rceZdZUdZeeed< ddededeedeeded ed ed ed ed e de dee e jddffd Z ddejdeejdedejfdZxZS)r aStandard Multi-head Self Attention module with QKV projection. This module implements the standard multi-head attention mechanism used in transformers. It supports both the fused attention implementation (scaled_dot_product_attention) for efficiency when available, and a manual implementation otherwise. The module includes options for QK normalization, attention dropout, and projection dropout. fused_attnNdim num_heads attn_head_dimdim_outqkv_biasqk_norm scale_norm proj_biasgated attn_drop proj_drop norm_layerrc6t|| |d}|xs|}|}|||zdk(sJd||z}|s|r | Jd||_||_||z|_|dz|_t |_tj||jdzfd|i||_ |r | |fi|ntj|_ |r | |fi|ntj|_ tj| |_|r| |jfi|ntj|_| r#tj||jfd|i|nd|_tj|j|fd|i||_tj| |_y) aInitialize the Attention module. Args: dim: Input dimension of the token embeddings. num_heads: Number of attention heads. attn_head_dim: Dimension of each attention head. If None, computed as dim // num_heads. dim_out: Output dimension. If None, same as dim. qkv_bias: Whether to use bias in the query, key, value projections. qk_norm: Whether to apply normalization to query and key vectors. scale_norm: Whether to apply normalization to attention output before projection. proj_bias: Whether to use bias in the output projection. gated: Apply an elementwise sigmoid gate to the attention output (anti attention-sink, GenLIP-style). attn_drop: Dropout rate applied to the attention weights. proj_drop: Dropout rate applied after the output projection. norm_layer: Normalization layer constructor for QK normalization if enabled. devicerNr$dim should be divisible by num_headstj| | | ||jr|jj nd|}nf| |j"z} | | j%dd z} t'|| ||} t)| | } | j+d } |j| } | | z}|j%ddj |||j,}|j/|}|||z}|j1|}|j3|}|S) Nr9rrr dropout_prr')shaperGsigmoidrAreshaper(r=permuteunbindrCrDr&Fscaled_dot_product_attentiontrainingr0pr? transposerrsoftmaxr>rFrHr1)rIrLrrBNCrGrAqkvrr$s rforwardzAttention.forwardos ''1a)-)>tyy|##%Dhhqk!!!Q4>>4==IQQRSUVXY[\^_`**Q-1a{{1~t{{1~1 ??..1a#.2mm$..**# ADJJAq{{2r**D.q$ 9MI!$ 2D<>$'DqA KK1  % %aDMM : IIaL  DA IIaL NN1 r) NNFFFTFrPrPNNNNF)__name__ __module__ __qualname____doc__rr!__annotations__intrrrrModuler<r Tensorrg __classcell__rKs@rr r +sd  +/%)"!$"!!488/8/8/$C= 8/ c] 8/  8/8/8/8/8/8/8/!bii18/ !8/z15# "||"  -" "  "rr c2eZdZUdZej j eed< dde de de e dedede d e d e d e e d e e jd ededededeffd Z dde ej de ej defdZxZS)r z A Self Attention module with ROPE support. Includes options for: * QK normalization option * Attention output (scale) normalization * Fused or unfused QKV projection support r&r'r(r*r+ qkv_fusednum_prefix_tokensr0r1r)r2r,r-r. rotate_halfr/ct|||d}|xs|}| }|||zdk(sJd||z}| s| r | Jd||_||_||z|_|dz|_||_t|_||_ |rAtj||jdzfd|i||_ dx|_ x|_|_nd|_ tj||jfd|i||_ tj||jfd|i||_tj||jfd|i||_| r | |fi|ntj |_| r | |fi|ntj |_tj&||_| r| |jfi|ntj |_|r#tj||jfd|i|nd|_tj|j|fd| i||_tj&||_y) aInitialize the Attention module. Args: dim: Input dimension of the token embeddings num_heads: Number of attention heads dim_out: Output dimension. If None, same as dim. qkv_bias: Whether to add a bias term to the query, key, and value projections qkv_fused: Whether to use fused QKV projection (single linear) or separate projections num_prefix_tokens: Number of reg/cls tokens at the beginning of the sequence that should not have position embeddings applied attn_drop: Dropout rate for attention weights proj_drop: Dropout rate for the output projection attn_head_dim: Dimension of each attention head. If None, computed as dim // num_heads. norm_layer: Normalization layer constructor to use for QK and scale normalization qk_norm: Enable normalization of query (Q) and key (K) vectors with norm_layer scale_norm: Enable normalization (scaling) of attention output with norm_layer proj_bias: Whether to use bias in the output projection rotate_half: Use 'half' ROPE layout instead of default 'interleaved' r4Nrr6r7r8r9r:)r;r<r(r=r>r?rvr r&rwrr@rAq_projk_projv_projrBrCrDrEr0rFrGrHr1)rIr'r(r*r+rurvr0r1r)r2r,r-r.rwr/r5rrJr=rKs rr<zAttentionRope.__init__sN  /.S   ?a' O)O O'i'H ) i+i i)"   9, % !2(*& yydmma&7MhM"MDH6: :DK :$+ DH))CLXLLDK))CLXLLDK))CLXLLDK4;j0R0 4;j0R0 I.7AJt}}33r{{} JOBIIc4==FxF2FUY IIdmmWK9KK I.rroperrc |j\}}}|j|j|jnd}|jc|j|} | j ||d|j |j jddddd} | jd\} } } n|j|j |||j |j jdd} |j|j |||j |j jdd} |j|j |||j |j jdd} |j| |j| } } ||j} t!|dd}t#j$| ddddd| ddft'| dddd| dddf|| gd j)| } t#j$| ddddd| ddft'| dddd| dddf|| gd j)| } |j*r>t-j.| | | ||j0r|j2j4nd | }nf| |j6z} | | jd dz}t9||||}t;||}|j=d }|j3|}|| z}|jddj |||j>}|jA|}|||z}|jC|}|jE|}|S)aForward pass for the attention module. Args: x: Input tensor of shape (batch_size, sequence_length, embedding_dim) rope: Rotary position embeddings tensor for position-aware attention attn_mask: Optional attention mask to apply during attention computation is_causal: If True, use causal (autoregressive) masking Returns: Tensor of shape (batch_size, sequence_length, dim_out) Nr9rNrrrOrwF)halfrUrPrQrSrT)#rVrGrWrArXr(r=rYrZryr_rzr{rCrDrvgetattrr catr type_asr&r[r\r]r0r^r?rrr`r>rFrHr1)rIrLr|rrrarbrcrGrArdrerfnptr~rr$s rrgzAttentionRope.forwards$''1a)-)>tyy|##%D 88 ((1+C++aAt~~t}}EMMaQRTUWXZ[\CjjmGAq! A&&q!T^^T]]KUUVWYZ[A A&&q!T^^T]]KUUVWYZ[A A&&q!T^^T]]KUUVWYZ[A{{1~t{{1~1  ((C46D 1Q4C4]+-@1aq=AQSW^b-cdjklttuvwA 1Q4C4]+-@1aq=AQSW^b-cdjklttuvwA ??..1a#.2mm$..**# ADJJAq{{2r**D.q$ 9MI!$ 2D<>$'DqA KK1  % %aDMM : IIaL  DA IIaL NN1 r)rhNTTrrPrPNNFFTFFNN)NNF)rjrkrlrmr jitrr!rnrorrrrrpr<rqrgrrrss@rr r sI %% %)!"%&!!+/*.!$" %%H/H/H/c] H/  H/  H/ #H/H/H/$C=H/RYYH/H/H/H/H/ !H/Z,004# =5<<(=  - =  =rr rri)typingrrrr rtorch.nnrr[_fxr configr pos_embed_sincosr __all__fxwraprqrror!rrpr r rrrrs(( $*"1 U?5<<?HU\\4J??-1 llELL)  ell (f fRQBIIQr