^j,dZddlmZmZmZmZddlZddlmZddlmcm Z ddl m Z ddl mZddlmZd eefd ZGd d ej&ZGd dej&Zy)a] Halo Self Attention Paper: `Scaling Local Self-Attention for Parameter Efficient Visual Backbones` - https://arxiv.org/abs/2103.12731 @misc{2103.12731, Author = {Ashish Vaswani and Prajit Ramachandran and Aravind Srinivas and Niki Parmar and Blake Hechtman and Jonathon Shlens}, Title = {Scaling Local Self-Attention for Parameter Efficient Visual Backbones}, Year = {2021}, } Status: This impl is a WIP, there is no official ref impl and some details in paper weren't clear to me. The attention mechanism works but it's slow as implemented. Hacked together by / Copyright 2021 Ross Wightman )ListOptionalTupleUnionN)nn)make_divisible) trunc_normal_)_assert permute_maskc|j\}}}}|jd}|dzdz}||jddz} | jd||} tj| ddgj d} tj| d||z g} | jd|dz|} | ddd||dz df} | j||d||j dd|dd} | j|S)a~ Compute relative logits along one dimension As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 Args: q: (batch, height, width, dim) rel_k: (2 * window - 1, dim) permute_mask: permute output dim according to this rrN)shape transposereshapeFpadflattenexpandpermute) qrel_kr BHWdimrel_sizewin_sizexx_pads `/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/layers/halo_attn.py rel_logits_1dr$s77LAq!S{{1~H1 "H U__R $ $A "a"A EE!aV  $ $Q 'E EE%!X\* +E MM"a!eX .E a!X\]"#A !Q1h'..r2xRHA 99\ ""c FeZdZdZ d dedededeffd ZdZdZxZ S) PosEmbedRelz Relative Position Embedding As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 block_sizer dim_headscalecJ||d}t|||_||_||_t j tj|dzdz |fi||_ t j tj|dzdz |fi||_ |jy)z Args: block_size: block size win_size: neighbourhood window size dim_head: attention head dim scale: scale factor (for init) devicedtyperrN) super__init__r(r)r*r Parametertorchempty height_rel width_relreset_parameters) selfr(r r)r*r-r.dd __class__s r#r0zPosEmbedRel.__init__Cs  / $   ,,u{{8a  dnn$**=r%cB|j\}}}}|jd|j|j|j}t ||j d}|j dd}t ||jd}||z}|j|||d}|S)Nr)rrr)r rr)rr@rrAr)rrr(r)r$r5rr4) r7rrBBHW_ rel_logits_w rel_logits_h rel_logitss r#forwardzPosEmbedRel.forwardbsww 2r1 IIb$//4??DMM J$Q_U  KK1 $QoV !L0 ''2r26 r%)NN) __name__ __module__ __qualname____doc__intfloatr0r6rH __classcell__r9s@r#r'r'=sE       6> r%r'ceZdZdZ ddedeedeeeefdededeeded ed ed ed ed effd Z dZ dZ xZ S)HaloAttna Halo Attention Paper: `Scaling Local Self-Attention for Parameter Efficient Visual Backbones` - https://arxiv.org/abs/2103.12731 The internal dimensions of the attention module are controlled by the interaction of several arguments. * the output dimension of the module is specified by dim_out, which falls back to input dim if not set * the value (v) dimension is set to dim_out // num_heads, the v projection determines the output dim * the query and key (qk) dimensions are determined by * num_heads * dim_head if dim_head is not None * num_heads * (dim_out * attn_ratio // num_heads) if dim_head is None * as seen above, attn_ratio determines the ratio of q and k relative to the output if dim_head not used Args: dim (int): input dimension to the module dim_out (int): output dimension of the module, same as dim if not set feat_size (Tuple[int, int]): size of input feature_map (not used, for arg compat with bottle/lambda) stride: output stride of the module, query downscaled if > 1 (default: 1). num_heads: parallel attention heads (default: 8). dim_head: dimension of query and key heads, calculated from dim_out * attn_ratio // num_heads if not set block_size (int): size of blocks. (default: 8) halo_size (int): size of halo overlap. (default: 3) qk_ratio (float): ratio of q and k dimensions to output dimension when dim_head not set. (default: 1.0) qkv_bias (bool) : add bias to q, k, and v projections avg_down (bool): use average pool downsample instead of strided query blocks scale_pos_embed (bool): scale the position embedding as well as Q @ K rdim_out feat_sizestride num_headsr)r( halo_sizeqk_ratioqkv_biasavg_downscale_pos_embedc| |d}t||xs|}||zdk(sJ|dvsJ||_|xst|| zd|z|_||jz|_||jz|_||j z|_|jdz|_| |_ |x|_ |_ ||_ ||dzz|_ d|_d }|dkDr5| xs||zdk7}|rdn||_|j|jz|_ tj ||j df|j| d ||_tj ||j |jzdfd | i||_t'd |j|j|j|jd ||_|rtj*ddntj,|_|j1y)Nr,r)rr)divisorrrF)rUbiasr`)r(r r)r*)r/r0rVr dim_head_qk dim_head_v dim_out_qk dim_out_vr*r[r( block_size_dsrWr block_striderConv2drkvr' pos_embed AvgPool2dIdentitypoolr6)r7rrSrTrUrVr)r(rWrXrYrZr[r-r.r8 use_avg_poolr9s r#r0zHaloAttn.__init__s" / .S"a'''"#a~g6HRS'TXa'a!T^^3#d&6&66"T__4%%- ./99$,""Y]2  A:#?zF':a'?L%1vD !%D4E4E!ED  3b4;L;LS[b_ab))C4>>!A1Z8ZWYZ$ ))]]%%**    +7BLLA&BKKM  r%c|jjjddz}t|jj|t|jj|t|j j |jt|j j|jy)Nrr_r;) rweightrr rirjr4r*r5)r7r<s r#r6zHaloAttn.reset_parametersssffmm!!!$,dffmm-dggnn#.dnn//TZZ@dnn..DJJ?r%c|j\}}}}t||jzdk(dt||jzdk(d||jz}||jz}||z}|j|} | j d|j ||j ||j jdddddd} | j ||jz|j d|jdd} |j|} tj| |j|j|j|jg} | jd|j|jjd|j|jj ||jz|j |j z|djdddd} t#j$| |j |j gd \} } |j&r5| | jdd z|j)| z|j*z} n4| | jdd z|j*z|j)| z} | j-d } | | zjdd}|j d|j |j ||}|jdddddj/j1||j2||j4z||j4z}|j7|}|S) Nrrrr@rrA)rr)rr r(rrrbrfrrVrrirrrWunfoldr rcr2splitr[rjr*softmax contiguousviewrergrm)r7r!rCrr num_h_blocks num_w_blocks num_blocksrrikvattnouts r#rHzHaloAttn.forwardsWW 1aDOO#q("-DOO#q("-DOO+ DOO+ !L0 FF1I II   $,,lDrs`$0/ #& #$s)#>2"))2jEryyEPr%