^jdZddlmZmZmZddlZddlmZddlmcm Z ddl m Z m Z ddlmZddlmZdeefd ZGd d ej(ZGd d ej(Zy)aP Bottleneck Self Attention (Bottleneck Transformers) Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 @misc{2101.11605, Author = {Aravind Srinivas and Tsung-Yi Lin and Niki Parmar and Jonathon Shlens and Pieter Abbeel and Ashish Vaswani}, Title = {Bottleneck Transformers for Visual Recognition}, Year = {2021}, } Based on ref gist at: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 This impl is a WIP but given that it is based on the ref gist likely not too far off. Hacked together by / Copyright 2021 Ross Wightman )ListOptionalTupleN) to_2tuplemake_divisible) trunc_normal_)_assert permute_maskc|j\}}}}||jddz}|jd|d|zdz }tj|ddgj d}tj|d|dz g}|jd|dzd|zdz }|ddd||dz df}|j||d||j dd|dd}|j|S)a Compute relative logits along one dimension As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 Args: q: (batch, heads, height, width, dim) rel_k: (2 * width - 1, dim) permute_mask: permute output dim according to this rrN)shape transposereshapeFpadflattenexpandpermute) qrel_kr BHWdimxx_pads f/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/layers/bottleneck_attn.py rel_logits_1dr!s77LAq!S U__R $ $A "aQ"A EE!aV  $ $Q 'E EE%!QU $E MM"a!eQUQY /E a!QUVmA !Q1a ''B2r:A 99\ ""cLeZdZdZ ddeeefdedeffd ZdZdZ xZ S) PosEmbedRelz Relative Position Embedding As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 feat_sizedim_headscalec||d}t|t|\|_|_||_||_tjtj|jdzdz |fi||_ tjtj|jdzdz |fi||_ |jy)Ndevicedtyperr)super__init__rheightwidthr&r'nn Parametertorchempty height_rel width_relreset_parameters)selfr%r&r'r*r+dd __class__s r r-zPosEmbedRel.__init__=s / "+I"6 TZ   ,,u{{4;;?Q3F'WTV'WXekk$**q.12Dh&URT&UV r"ctjjj|j|j tjjj|j |j y)Nstd)r2r0initnormal_r4r'r5r7s r r6zPosEmbedRel.reset_parametersPsH  doo4::>  dnn$**=r"c*|j\}}}|j||j|jd}t ||j d}|j dd}t ||jd}||z}|j|||}|S)Nr )rrr)r rr)rrArrBr)rrr.r/r!r5rr4)r7rrHW_ rel_logits_w rel_logits_h rel_logitss r forwardzPosEmbedRel.forwardTs772q IIadjj" 5$Q_U  KK1 $QoV !L0 ''2r2 r")NN) __name__ __module__ __qualname____doc__rintfloatr-r6rH __classcell__r9s@r r$r$8sC  S#X    &> r"r$ceZdZdZ ddedeedeeeefdededeeded ed effd Z d Z d Z xZ S)BottleneckAttna Bottleneck Attention Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 The internal dimensions of the attention module are controlled by the interaction of several arguments. * the output dimension of the module is specified by dim_out, which falls back to input dim if not set * the value (v) dimension is set to dim_out // num_heads, the v projection determines the output dim * the query and key (qk) dimensions are determined by * num_heads * dim_head if dim_head is not None * num_heads * (dim_out * attn_ratio // num_heads) if dim_head is None * as seen above, attn_ratio determines the ratio of q and k relative to the output if dim_head not used Args: dim (int): input dimension to the module dim_out (int): output dimension of the module, same as dim if not set stride (int): output stride of the module, avg pool used if stride == 2 (default: 1). num_heads (int): parallel attention heads (default: 4) dim_head (int): dimension of query and key heads, calculated from dim_out * attn_ratio // num_heads if not set qk_ratio (float): ratio of q and k dimensions to output dimension when dim_head not set. (default: 1.0) qkv_bias (bool): add bias to q, k, and v projections scale_pos_embed (bool): scale the position embedding as well as Q @ K rdim_outr%stride num_headsr&qk_ratioqkv_biasscale_pos_embedc | | d} t ||Jd|xs|}||zdk(sJ||_|xst||zd|z|_||jz|_||jz|_||j z|_|jdz|_| |_ tj||j dz|jzdfd |i| |_ t|f|j|jd | |_|dk(rtjddntj |_|j%y) Nr)zBA concrete feature size matching expected input (H, W) is requiredr)divisorrrbias)r&r')r,r-rUr dim_head_qk dim_head_v dim_out_qk dim_out_vr'rXr0Conv2dqkvr$ pos_embed AvgPool2dIdentitypoolr6)r7rrSr%rTrUr&rVrWrXr*r+r8r9s r r-zBottleneckAttn.__init__zsA / $j&jj$.S"a'''"#a~g6HRS'TXa'a!T^^3#d&6&66"T__4%%- .99S$//A"5"F_PX_\^_%Yb9I9IQUQ[Q[b_ab*0A+BLLA&2;;=  r"c:t|jj|jjjddzt|jj |j t|jj|j y)Nrr\r;)r rcweightrrdr4r'r5r?s r r6zBottleneckAttn.reset_parameterss\dhhoo488??+@+@+Ct+KLdnn//TZZ@dnn..DJJ?r"c|j\}}}}t||jjk(dt||jjk(d|j |}t j||j|j|jgd\}}}|j||jz|jdjdd}|j||jz|jd}|j||jz|jdjdd}|jr%||z|j|z|j z} n$||z|j z|j|z} | j#d} | |zjddj||j||} |j%| } | S)Nr)rr r)rr rdr.r/rcr2splitr`rarrUr^rr_rXr'softmaxrg) r7rrCrrrkvattnouts r rHzBottleneckAttn.forwardsWW 1aT^^***B/T^^)))2. HHQK++a$//4??DNN!SYZ[1a IIa$..($*:*:B ? I I"b Q IIa$..($*:*:B ? IIa$..($//2 > H HR P   EDNN1--;DETZZ'$..*;;D|||#ax""2r*221dnnaKiin r") NNrrBNg?FFNN) rIrJrKrLrMrrrNboolr-r6rHrOrPs@r rRrRds0&*37&*!"$)" " c]"  c3h0 "  "  " sm" " " "" H@ r"rR)rLtypingrrrr2torch.nnr0torch.nn.functional functionalrhelpersrr weight_initr trace_utilsr rMr!Moduler$rRr"r r}sZ )( .& #$s)#8)")))XURYYUr"