^j/)dZddlmZmZmZddlZddlmZddlmcm Z ddl m Z GddejZGdd ejZGd d ejZGd d ejZy)aH Non-Local Attention Pooling Layers A collection of global pooling layers that go beyond simple avg/max pooling. LSEPool - LogSumExp pooling, a smooth approximation between avg and max pooling SimPool - Attention-based pooling from 'Keep It SimPool' (ICCV 2023) Based on implementations from: * LSE Pooling: custom implementation by Bill Psomas * SimPool: https://arxiv.org/abs/2309.06891 - 'Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?' by Bill Psomas et al. Hacked together by / Copyright 2024 Ross Wightman, original code by Bill Psomas )OptionalTypeUnionN)use_fused_attncteZdZdZ d dededeffd ZdejdejfdZ xZ S) LsePlus2daLogSumExp (LSE) Pooling for 2D inputs. A smooth approximation to max pooling that provides a learnable interpolation between average and max pooling. When r is large, LSE approaches max pooling; when r is small, it approaches average pooling. Implements: (1/r) * log((1/n) * sum(exp(r * (x - x_max)))) + x_max The x_max subtraction provides numerical stability. r r_learnableflattenct||r8tjt j ||||_||_y|jdt j |||||_y)z Args: r: Initial value of the pooling parameter. Higher = closer to max pooling. r_learnable: If True, r is a learnable parameter. flatten: If True, flatten spatial dims in output. devicedtyper N) super__init__nn Parametertorchtensorr register_bufferr )selfr r r rr __class__s a/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/layers/other_pool.pyrzLsePlus2d.__init__$s_  \\%,,qu"MNDF   ell1V5&Q R xreturnc,tj|d}tj|j||z z}|j dd}|d|jz tj |zz}|jr|jd}|S)Nr)Tdimkeepdim?)Fadaptive_max_pool2drexpr meanlogr rrx_maxexp_xsum_expouts rforwardzLsePlus2d.forward9sz%%a+ $&&AI./***6sTVV|uyy'999 <<++a.C r)$@TTNN __name__ __module__ __qualname____doc__floatboolrrTensorr/ __classcell__rs@rr r sU  $   *%,,rr cneZdZdZ ddedeffd ZdejdejfdZ xZ S) LsePlus1daLogSumExp (LSE) Pooling for sequence (NLC) inputs. A smooth approximation to max pooling that provides a learnable interpolation between average and max pooling. When r is large, LSE approaches max pooling; when r is small, it approaches average pooling. r r ct||r1tjt j ||||_y|jdt j |||y)z Args: r: Initial value of the pooling parameter. Higher = closer to max pooling. r_learnable: If True, r is a learnable parameter. rr N)rrrrrrr r)rr r rrrs rrzLsePlus1d.__init__KsM  \\%,,qu"MNDF  ell1V5&Q Rrrrc|jddj}tj|j||z z}|j dd}|d|jz tj |zz}|jdS)NrTr!r$)maxvaluesrr'r r(r)squeezer*s rr/zLsePlus1d.forward]su!T*11 $&&AI./**D*1sTVV|uyy'999{{1~r)r0TNNr1r:s@rr<r<CsK $ SSS$%,,rr<ceZdZUdZej j eed< d de de dedede e de e e jf fd Zd ej d ej fd ZxZS) SimPool2dabSimPool: Simple Attention-Based Pooling for 2D (NCHW) inputs. From 'Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?' https://arxiv.org/abs/2309.06891 Uses GAP as query initialization and applies cross-attention between the GAP query and spatial features to produce a weighted pooled representation. fused_attnr" num_headsqkv_biasqk_normgamma norm_layerc Ft |||d} ||zdk(sJd||_||z|_|jdz|_||_t |_|xstj}||fi| |_ tj||fd|i| |_ tj||fd|i| |_ |r1||jfi| |_||jfi| |_ytj |_tj |_y)a Args: dim: Input feature dimension (number of channels). num_heads: Number of attention heads. qkv_bias: If True, add bias to query and key projections. qk_norm: If True, apply normalization to queries and keys. gamma: If provided, apply power normalization to values with this exponent. norm_layer: Normalization layer for patches and optionally qk_norm. flatten: If True, flatten output to (B, C). rr"dim must be divisible by num_headsbiasNrrrEhead_dimscalerHrrDr LayerNormnormLinearqkq_normk_normIdentity rr"rErFrGrHrIrrddrs rrzSimPool2d.__init__qs*  /Y!#I%II#"y( ]]d*  (*/2<< s)b) 39(9b939(9b9 $T]]9b9DK$T]]9b9DK++-DK++-DKrrrc4|j\}}}}||z}|jdjdd}|jdd}|j |}|j |j |d|j|jjdd}|j|j |||j|jjdd} |j |||j|jjdd} |j||j| } }|j| jdd} | | z dz} |jr1tj || | j#|j} nR||j$z| jddz}|j'd}|| j#|jz} | j#d |jz }n]|jrtj || | }n9||j$z| jddz}|j'd}|| z}|jddj ||}|S) NrrTr!ư>r"r$)shaper transposer(rRrTreshaperErOrUrVrWrHaminrDr%scaled_dot_product_attentionpowrPsoftmax)rrBCHWNrTx_normrUvv_min v_shiftedattn_outattnr.s rr/zSimPool2d.forwards*WW 1a E IIaL " "1a ( FFq$F '1 FF1I  aDNNDMM B L LQPQ R FF6N " "1a G Q QRSUV W NN1a ? I I!Q O{{1~t{{1~1 :: !FFr4F0EE D(I99!Q djj@YZDJJ!++b"*==|||+)-- ";;,,sTZZ/0C44Q1=DJJ!++b"*==|||+QhmmAq!))!Q/ rrFFNNNNr2r3r4r5rjitFinalr7__annotations__intrr6rrModulerr8r/r9r:s@rrCrCfs %% "!%)48'('('( '(  '( E? '(!bii1'(R))%,,)rrCceZdZUdZej j eed< d de de dedede e de e e jf fd Zd ej d ej fd ZxZS) SimPool1dafSimPool: Simple Attention-Based Pooling for sequence (NLC) inputs. From 'Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?' https://arxiv.org/abs/2309.06891 Uses GAP as query initialization and applies cross-attention between the GAP query and sequence tokens to produce a weighted pooled representation. rDr"rErFrGrHrIc Ft |||d} ||zdk(sJd||_||z|_|jdz|_||_t |_|xstj}||fi| |_ tj||fd|i| |_ tj||fd|i| |_ |r1||jfi| |_||jfi| |_ytj |_tj |_y)a Args: dim: Input feature dimension. num_heads: Number of attention heads. qkv_bias: If True, add bias to query and key projections. qk_norm: If True, apply normalization to queries and keys. gamma: If provided, apply power normalization to values with this exponent. norm_layer: Normalization layer for tokens and optionally qk_norm. rrrKrLrMNrNrYs rrzSimPool1d.__init__s(  /Y!#I%II#"y( ]]d*  (*/2<< s)b) 39(9b939(9b9 $T]]9b9DK$T]]9b9DK++-DK++-DKrrrc|j\}}}|jdd}|j|}|j|j |d|j |j jdd}|j|j |||j |j jdd}|j |||j |j jdd}|j||j|}}|j|jdd} || z dz} |jr1tj||| j!|j} nR||j"z|jddz} | j%d} | | j!|jz} | j!d |jz } n]|jrtj|||} n9||j"z|jddz} | j%d} | |z} | jddj ||} | S) NrTr!rr\r]r^r_r$)r`r(rRrTrbrErOrarUrVrWrHrcrDr%rdrerPrf)rrrgrkrhrTrlrUrmrnrorprqr.s rr/zSimPool1d.forwards''1a FFq$F '1 FF1I  aDNNDMM B L LQPQ R FF6N " "1a G Q QRSUV W NN1a ? I I!Q O{{1~t{{1~1 :: !FFr4F0EE D(I99!Q djj@YZDJJ!++b"*==|||+)-- ";;,,sTZZ/0C44Q1=DJJ!++b"*==|||+QhmmAq!))!Q/ rrrrsr:s@rrzrzs %% "!%)48&(&(&( &(  &( E? &(!bii1&(P%%%,,%rrz)r5typingrrrrtorch.nnrtorch.nn.functional functionalr%configrrxr r<rCrzrrrsh )( "( (V  F] ]@X Xr