^jdZddlZddlmZmZmZmZmZmZm Z m Z ddl Z ddl m Z ddlm cmZddlmZmZddlmZmZmZmZmZmZddlmZddlmZdd lm Z dd l!m"Z"m#Z#d gZ$Gd d e jJZ&Gdde jJZ'Gdde jJZ(Gdde jJZ)Gdde jJZ*Gdde jJZ+de,dede jJfdZ-dYdee.dfde/de.dee.e.e.e.ffd Z0Gd!d"e jJZ1Gd#d$e jJZ2 dZd%ed&e.d'e.d(ee.d)e.d*e.d+e.d,e.d-e/d.e3d/e/d0e/dede jhfd1Z5 d[d%ed&e.d'e.d(ee.d)e.d-e/d.e3d/e/d0e/dede jhfd2Z6Gd3d e jJZ7d\d4e,d5e3dede7fd6Z8d]d7e,dedee,effd8Z9e#e9d9d:d;<e9d9d=d>d?@e9d9dAd;<e9d9dBd>d?@e9d9dCd;<e9d9dDd>dE@e9d9dFd;<e9d9dGdHdE@e9d9dId;<e9d9dJdHdE@e9d9dKdHdL@dM Z:e"d\d5e3dede7fdNZ;e"d\d5e3dede7fdOZe"d\d5e3dede7fdRZ?e"d\d5e3dede7fdSZ@e"d\d5e3dede7fdTZAe"d\d5e3dede7fdUZBe"d\d5e3dede7fdVZCe"d\d5e3dede7fdWZDe"d\d5e3dede7fdXZEy)^a5 Vision OutLOoker (VOLO) implementation Paper: `VOLO: Vision Outlooker for Visual Recognition` - https://arxiv.org/abs/2106.13112 Code adapted from official impl at https://github.com/sail-sg/volo, original copyright in comment below Modifications and additions for timm by / Copyright 2022, Ross Wightman N)AnyCallableDictListOptionalTupleUnionTypeIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)DropPathMlp to_2tuple to_ntuple trunc_normal_use_fused_attn)build_model_with_cfg)feature_take_indices) checkpoint)register_modelgenerate_default_cfgsVOLOceZdZdZ ddededededededed effd Zd ejd ejfd Z xZ S)OutlookAttentionz,Outlook attention mechanism for VOLO models.dim num_heads kernel_sizepaddingstrideqkv_bias attn_drop proj_dropc | | d} t |||z} ||_||_||_||_| dz|_tj||fd|i| |_ tj||dz|zfi| |_ tj||_ tj||fi| |_ tj||_tj||||_tj"||d|_y) aInitialize OutlookAttention. Args: dim: Input feature dimension. num_heads: Number of attention heads. kernel_size: Kernel size for attention computation. padding: Padding for attention computation. stride: Stride for attention computation. qkv_bias: Whether to use bias in linear layers. attn_drop: Attention dropout rate. proj_drop: Projection dropout rate. devicedtypebias)rr r!T)rr! ceil_modeN)super__init__rrr r!scalennLinearvattnDropoutr#projr$Unfoldunfold AvgPool2dpool)selfrrrr r!r"r#r$r'r(ddhead_dim __class__s [/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/models/volo.pyr.zOutlookAttention.__init__*s2 / )#"&  % 39(9b9IIc;!#3i#?F2F I.IIc3-"- I.iiKQWX LLVFdS xreturnc|j\}}}}|j|jdddd}tj||j z tj||j z }}|j |j||j||jz|j|jz||zjddddd}|j|jddddjdddd} |j| j|||z|j|j|jz|j|jzjddddd} | |jz} | jd} |j| } | |zjdddddj|||jz|jz||z}tj |||f|j|j"|j }|j%|jdddd}|j'|}|S) Forward pass. Args: x: Input tensor of shape (B, H, W, C). Returns: Output tensor of shape (B, H, W, C). rrr+r) output_sizerr r!)shaper2permutemathceilr!r7reshaperrr9r3r/softmaxr#Ffoldr r5r$) r:r@BHWCr2hwr3s r>forwardzOutlookAttention.forwardVs!WW 1a FF1I  aAq )yyT[[)499Q_+E1 KKN " " t~~qDNN2   t// /Q 88?1aA8N yy1aA./771aCyy&& q1udnnd&6&69I9I&I   t// /118Aq!Q1G djj |||#~~d# AX  q!Q1 - 5 5aT=M=M9MPTP`P`9`bcfgbg h FF11a&d6F6FPTP\P\eiepep q IIaii1a+ , NN1 r?)rDrrFrXNN) __name__ __module__ __qualname____doc__intboolfloatr.torchTensorrW __classcell__r=s@r>rr's6 !"!!*T*T*T *T  *T  *T*T*T*TX  %,, r?rceZdZdZdddddej ej dddf deded ed ed ed ed edede ejde ejde ffd Z de jde jfdZxZS) Outlookerz9Outlooker block that combines outlook attention with MLP.r@rXFNrrr r!r mlp_ratior# drop_path act_layer norm_layerr"c | | d}t|| |fi||_t||f|||| |d||_|dkDr t |nt j|_| |fi||_ td|t||z| d||_ |dkDrt ||_ yt j|_ y)afInitialize Outlooker block. Args: dim: Input feature dimension. kernel_size: Kernel size for outlook attention. padding: Padding for outlook attention. stride: Stride for outlook attention. num_heads: Number of attention heads. mlp_ratio: Ratio for MLP hidden dimension. attn_drop: Attention dropout rate. drop_path: Stochastic depth drop rate. act_layer: Activation layer type. norm_layer: Normalization layer type. qkv_bias: Whether to use bias in linear layers. r&)rr r!r"r#rX in_featureshidden_featuresriN)r-r.norm1rr3rr0Identity drop_path1norm2rr]mlp drop_path2)r:rrr r!rrgr#rhrirjr"r'r(r;r=s r>r.zOutlooker.__init__|s> / *r* $    $     2;R(9-R[[]*r*  i0    2;R(9-R[[]r?r@rAc||j|j|j|z}||j|j |j |z}|SzoForward pass. Args: x: Input tensor. Returns: Output tensor. rrr3rprurtrsr:r@s r>rWzOutlooker.forwardQ  $**Q- 89 9 A 78 8r?)rYrZr[r\r0GELU LayerNormr]r_r Moduler^r.r`rarWrbrcs@r>rereysC!!!)+*,,,"5S5S5S 5S  5S  5S5S5S5SBII5SRYY5S5Sn  %,, r?rec eZdZUdZej j eed< d de de dede de f fd Z d ejd ejfd Z xZS) Attentionz!Multi-head self-attention module. fused_attnrrr"r#r$cX||d}t |||_||z} | dz|_t |_t j||dzfd|i||_t j||_ t j||fi||_ t j||_ y)a,Initialize Attention module. Args: dim: Input feature dimension. num_heads: Number of attention heads. qkv_bias: Whether to use bias in QKV projection. attn_drop: Attention dropout rate. proj_drop: Projection dropout rate. r&r)rDr*N) r-r.rr/rrr0r1qkvr4r#r5r$) r:rrr"r#r$r'r(r;r<r=s r>r.zAttention.__init__s& / ")#% (*99S#'??B?I.IIc3-"- I.r?r@rAc|j\}}}}|j|j|||zd|j||jzj ddddd}|j d\}}} |j rrWzAttention.forwards0WW 1ahhqk!!!QUAt~~qDNN?RS[[\]_`bcefhij**Q-1a ??..1a.2mm$..**A DJJAq{{2r**D<>$'DqA KK1  % %aAq 1 IIaL NN1 r?)FrXrXNN)rYrZr[r\r`jitFinalr^__annotations__r]r_r.rarWrbrcs@r>rrs}+ %% "!!/// /  /  />%,,r?rceZdZdZddddej ej ddfdededed e d ed ed e ejd e ejffd Z de jde jfdZxZS) Transformerz9Transformer block with multi-head self-attention and MLP.@FrXNrrrgr"r#rhrirjc ~| | d} t |||fi| |_t|f|||d| |_|dkDr t |nt j|_||fi| |_ td|t||z|d| |_ |dkDrt ||_ yt j|_ y)aInitialize Transformer block. Args: dim: Input feature dimension. num_heads: Number of attention heads. mlp_ratio: Ratio for MLP hidden dimension. qkv_bias: Whether to use bias in QKV projection. attn_drop: Attention dropout rate. drop_path: Stochastic depth drop rate. act_layer: Activation layer type. norm_layer: Normalization layer type. r&)rr"r#rXrlNro)r-r.rprr3rr0rqrrrsrr]rtru) r:rrrgr"r#rhrirjr'r(r;r=s r>r.zTransformer.__init__s2 / *r* ceYU^ebde 1:R(9-R[[]*r* h3C)O8LXahegh1:R(9-R[[]r?r@rAc||j|j|j|z}||j|j |j |z}|Srwrxrys r>rWzTransformer.forward+rzr?)rYrZr[r\r0r{r|r]r_r^r r}r.r`rarWrbrcs@r>rrsC ""!!)+*,,,!S!S!S !S  !S  !S!SBII!SRYY!SF  %,, r?rc eZdZdZ d dededeedededef fd Zd e jd e jfd Z xZ S) ClassAttentionz6Class attention mechanism for class token interaction.rrr<r"r#r$c "||d} t |||_|||_n ||z}||_|dz|_t j ||j|jzdzfd|i| |_t j ||j|jzfd|i| |_t j||_ t j |j|jz|fi| |_ t j||_ y)a{Initialize ClassAttention. Args: dim: Input feature dimension. num_heads: Number of attention heads. head_dim: Dimension per head. If None, computed as dim // num_heads. qkv_bias: Whether to use bias in QKV projection. attn_drop: Attention dropout rate. proj_drop: Projection dropout rate. r&Nr)rEr*) r-r.rr<r/r0r1kvrr4r#r5r$) r:rrr<r"r#r$r'r(r;r=s r>r.zClassAttention.__init__<s* / "  $DMi'H$DM% ))C!?!!CY(YVXY3  >TXTQSTI.IIdmmdnnrWzClassAttention.forwardas)''1a WWQZ  1a G O OPQSTVWYZ\] ^yy|1 FF1QAX;  ' '4>>1dmm Ltzz Y1;;r2&&|||#~~d#AX((A.66q!T]]T^^=[\ IIi( NN9- r?)rNFrXrXNN) rYrZr[r\r]rr^r_r.r`rarWrbrcs@r>rr9s|@ &*"!!#/#/#/sm #/  #/  #/#/J%,,r?rceZdZdZddddddej ej ddf dededeed e d e d e d e d e de ejde ejffd Z dejdejfdZxZS) ClassBlockz3Class block that combines class attention with MLP.NrFrXrrr<rgr"dropr#rhrirjc | | d} t|| |fi| |_t|f|||||d| |_|dkDr t |nt j|_| |fi| |_ td|t||z| |d| |_ |dkDrt ||_ yt j|_ y)a1Initialize ClassBlock. Args: dim: Input feature dimension. num_heads: Number of attention heads. head_dim: Dimension per head. If None, computed as dim // num_heads. mlp_ratio: Ratio for MLP hidden dimension. qkv_bias: Whether to use bias in QKV projection. drop: Dropout rate. attn_drop: Attention dropout rate. drop_path: Stochastic depth drop rate. act_layer: Activation layer type. norm_layer: Normalization layer type. r&)rr<r"r#r$rX)rmrnrirNro)r-r.rprr3rr0rqrrrsrr]rtru)r:rrr<rgr"rr#rhrirjr'r(r;r=s r>r.zClassBlock.__init__}s: / *r* "      2;R(9-R[[]*r*  i0    2;R(9-R[[]r?r@rAc$|ddddf}||j|j|j|z}||j|j |j |z}t j||ddddfgdS)zForward pass. Args: x: Input tensor of shape (B, N, C) where first token is class token. Returns: Output tensor with updated class token. NrrG)rrr3rprurtrsr`cat)r:r@rs r>rWzClassBlock.forwards~a!eH  $**Q-0H II I9N0O PP yy)Qq!"uX.A66r?)rYrZr[r\r0r{r|r]rr_r^r r}r.r`rarWrbrcs@r>rrzs= '+!"!!)+*,,,3S3S3Ssm 3S  3S  3S3S3S3SBII3SRYY3Sj 7 7%,, 7r?r block_typekwargsrAc 6|dk(r tdi|SJd|)zGet block based on type. Args: block_type: Type of block ('ca' for ClassBlock). **kwargs: Additional keyword arguments for block. Returns: The requested block module. cazInvalid block type: ro)r)rrs r> get_blockrs,T#F##9,ZL99ur?size.lamr/c|d|z}|d|z}tj|tj}tj|tj}tjd|z }||zj }||zj } tj d|d} tj d|d} tj | |dzz d|} tj | | dzz d|} tj | |dzzd|}tj | | dzzd|}| j| j|j|jfS)a1Get random bounding box for token labeling. Reference: https://github.com/zihangJiang/TokenLabeling Args: size: Input tensor size tuple. lam: Lambda parameter for cutmix. scale: Scaling factor. Returns: Bounding box coordinates (bbx1, bby1, bbx2, bby2). rrE)r(?rr)r`tensorfloat32sqrtr]randintclampitem)rrr/rSrRW_tH_tcut_ratcut_wcut_hcxcybbx1bby1bbx2bby2s r> rand_bboxrs3 Q5A Q5A ,,q .C ,,q .Cjjc"G 7]   !E 7]   !E q!T "B q!T "B ;;rEQJ1 -D ;;rEQJ1 -D ;;rEQJ1 -D ;;rEQJ1 -D 99; TYY[$))+ ==r?ceZdZdZ d dededededededeffd Zd ejd ejfd Z xZ S) PatchEmbedz6Image to patch embedding with multi-layer convolution.img_size stem_conv stem_stride patch_sizein_chans hidden_dim embed_dimc || d} t ||dvsJ|rtjtj||fd|ddd| tj |fi| tj dtj||fdd d dd| tj |fi| tj dtj||fdd d dd| tj |fi| tj d |_nd |_tj||f||z||zd | |_||z||zz|_ y ) aInitialize PatchEmbed. Different from ViT which uses 1 conv layer, VOLO uses multiple conv layers for patch embedding. Args: img_size: Input image size. stem_conv: Whether to use stem convolution layers. stem_stride: Stride for stem convolution. patch_size: Patch size (must be 4, 8, or 16). in_chans: Number of input channels. hidden_dim: Hidden dimension for stem convolution. embed_dim: Output embedding dimension. r&)r+rrDF)rr!r r*T)inplacerNrr!) r-r.r0 SequentialConv2d BatchNorm2dReLUconvr5 num_patches) r:rrrrrrrr'r(r;r=s r>r.zPatchEmbed.__init__sN2 / Z'''   (JoAk[\cholnoz0R0% *jgaST[`gdfgz0R0% *jgaST[`gdfgz0R0% DIDIII   #k1,     % 2x:7MNr?r@rAcb|j|j|}|j|}|S)zForward pass. Args: x: Input tensor of shape (B, C, H, W). Returns: Output tensor of shape (B, embed_dim, H', W'). )rr5rys r>rWzPatchEmbed.forward)s- 99  ! A IIaLr?) FrrrD@NN) rYrZr[r\r]r^r.r`rarWrbrcs@r>rrs@ #   2O2O2O 2O  2O  2O2O2Oh  %,, r?rcpeZdZdZ d dededeffd ZdejdejfdZxZ S) Downsamplez#Downsampling module between stages. in_embed_dim out_embed_dimrcjt|||d}tj||f||d||_y)zInitialize Downsample. Args: in_embed_dim: Input embedding dimension. out_embed_dim: Output embedding dimension. patch_size: Patch size for downsampling. r&rN)r-r.r0rr5)r:rrrr'r(r;r=s r>r.zDownsample.__init__;s:  /IIlMkzZdkhjk r?r@rAcx|jdddd}|j|}|jdddd}|S)zForward pass. Args: x: Input tensor of shape (B, H, W, C). Returns: Output tensor of shape (B, H', W', C'). rrDrrE)rJr5rys r>rWzDownsample.forwardNs> IIaAq ! IIaL IIaAq !r?)rENN) rYrZr[r\r]r.r`rarWrbrcs@r>rr8sQ-   lll l&  %,, r?rblock_fnindexrlayersrrr r!rgr"r#drop_path_ratec g}t||D]H}| |t|d|zzt|dz z }|j||f|||||| | || | d |Jtj|}|S)aGenerate outlooker layers for stage 1. Args: block_fn: Block function to use (typically Outlooker). index: Index of current stage. dim: Feature dimension. layers: List of layer counts for each stage. num_heads: Number of attention heads. kernel_size: Kernel size for outlook attention. padding: Padding for outlook attention. stride: Stride for outlook attention. mlp_ratio: Ratio for MLP hidden dimension. qkv_bias: Whether to use bias in QKV projection. attn_drop: Attention dropout rate. drop_path_rate: Stochastic depth drop rate. **kwargs: Additional keyword arguments. Returns: Sequential module containing outlooker blocks. Nr) rr r!rrgr"r#rhr'r(rangesumappendr0r)rrrrrrr r!rgr"r#rr'r(rblocks block_idx block_dprs r>outlooker_blocksr]sJF6%=) "i#fVen2E&EF#f+XY/Z  h   #     ]]F #F Mr?c g} t||D]C} || t|d|zzt|dz z } | j|||f|||| d| Etj| } | S)aeGenerate transformer layers for stage 2. Args: block_fn: Block function to use (typically Transformer). index: Index of current stage. dim: Feature dimension. layers: List of layer counts for each stage. num_heads: Number of attention heads. mlp_ratio: Ratio for MLP hidden dimension. qkv_bias: Whether to use bias in QKV projection. attn_drop: Attention dropout rate. drop_path_rate: Stochastic depth drop rate. **kwargs: Additional keyword arguments. Returns: Sequential module containing transformer blocks. Nr)rgr"r#rhr) rrrrrrgr"r#rrrrrs r>transformer_blocksrs:F6%=)  "i#fVen2E&EF#f+XY/Z  h          ]]F #F Mr?c- eZdZdZddddddddd d d d d d d d ej d dd dddfdeededededededede eede eede e dfde e dfde de de de d e d!e d"e ejd#e e edfd$e d%e d&ef,fd' Zd(ejd)dfd*Zej$j&d)efd+Zej$j&dCd,e d)eeeffd-Zej$j&dDd.e d)dfd/Zej$j&d)ejfd0ZdEdede ed)dfd1Zd2ej8d)ej8fd3Zd2ej8d)ej8fd4Zd2ej8d)eej8e ej8ej8e eeeeffffd5Z dFd2ej8d6e eeeefd7e d8e d9ed:e d)eeej8e ej8eej8fffd;Z! dGd6eeeefdZ"d2ej8d)ej8fd?Z#dCd2ej8d@e d)ej8fdAZ$d2ej8d)ej8fdBZ%xZ&S)HrzVision Outlooker (VOLO) model.rrDtokenrrN)TFFFrfFrX)rrTrErrr num_classes global_poolrstem_hidden_dim embed_dimsr downsamples.outlook_attentionrgr" drop_rate pos_drop_rateattn_drop_raterrj post_layers use_aux_head use_mix_token pooling_scalect!|||d}t|}t|| } t |}||_||_||_||_||_ |dx|_ |_ |rd|_ |dk(sJdd|_ tddd||||d d ||_|}|d |z|z|d |z|zf}t!j"t%j&d |d |d |dfi||_t!j*| |_g|_g|_g}d }t3t|D]}| |r#t5t6||||| |f| || ||d |} n#t9t:||||| |f| || |||d|} |j=| |j.j=||j0j=t?|||d||d z }| |s|j=tA||||d zdfi||dz}|d z }t!jB||_"d|_#|t!jBt3t|Dcgc]"}tI||f|d| d| d| |d|d|$c}|_#t!j"t%j&d d |dfi||_%tM|jJd|r@|d kDr!t!jN|j|fi|nt!jP|_)nd|_)||jfi||_*t!j*||_+|d kDr!t!jN|j|fi|nt!jP|_,tM|j(d|j[|j\ycc}w)a/Initialize VOLO model. Args: layers: Number of blocks in each stage. img_size: Input image size. in_chans: Number of input channels. num_classes: Number of classes for classification. global_pool: Global pooling type ('token', 'avg', or ''). patch_size: Patch size for patch embedding. stem_hidden_dim: Hidden dimension for stem convolution. embed_dims: List of embedding dimensions for each stage. num_heads: List of number of attention heads for each stage. downsamples: Whether to downsample between stages. outlook_attention: Whether to use outlook attention in each stage. mlp_ratio: Ratio for MLP hidden dimension. qkv_bias: Whether to use bias in QKV projection. drop_rate: Dropout rate. pos_drop_rate: Position embedding dropout rate. attn_drop_rate: Attention dropout rate. drop_path_rate: Stochastic depth drop rate. norm_layer: Normalization layer type. post_layers: Post-processing layer types. use_aux_head: Whether to use auxiliary head. use_mix_token: Whether to use token mixing for training. pooling_scale: Pooling scale factor. r&rFrrz)return all tokens if mix_token is enabledFTrEr)rrrrrrr)r)rgr"r#rj)rgr"rr#rjznetwork.)num_chs reductionmoduleNrX)rrrgr"r#rhrj{Gz?stdro)/r-r.lenrrrrr mix_tokenr num_featureshead_hidden_sizebetagrad_checkpointingr patch_embedr0 Parameterr`zeros pos_embedr4pos_drop stage_ends feature_inforrrerrrdictr ModuleListnetwork post_networkr cls_tokenrr1rqaux_headnorm head_dropheadapply _init_weights)"r:rrrrrrrrrrrrgr"rrrrrjrrrrr'r(r; num_layersr patch_gridrristager=s" r>r.z VOLO.__init__s$j  /[ )Ij))4 X&&  &&*4>rNBD1 DI') V+V V)"'% !& m    qkZ/=@(1+Q[B[_lBlm ekk!Z]JqMS]^`Sa&heg&hi ]3  s6{#& A #(qMaL  (l%,)  +qMaL  (l%#1,)   NN5 ! OO " "9 -    $ $T*Q-1W_`i_jUk%l m NI1~z*Q-AE9JATQSTUQQ M& P}}W- !  " " s;/0 /N "2'm'm%, )   / !D  \\%++aJrN*Qb*QRDN $..c 2 OZ]^BIId&7&7KKdfdododqDM DMt007B7 I.GRUVBIId//CC\^\g\g\i dnn#. 4%%&; /s%'OmrAc t|tjrjt|jdt|tjr8|j +tj j|j dyyyy)z\Initialize weights for modules. Args: m: Module to initialize. rrNr) isinstancer0r1rweightr*init constant_)r:r s r>rzVOLO._init_weightsns\ a # !(( ,!RYY'AFF,>!!!&&!,-?' $r?c ddhS)zwGet set of parameters that should not have weight decay. Returns: Set of parameter names. r rror:s r>no_weight_decayzVOLO.no_weight_decayys[))r?coarsec&tdddggdS)zGet parameter grouping for optimizer. Args: coarse: Whether to use coarse grouping. Returns: Parameter grouping dictionary. z ^cls_token|pos_embed|patch_embed)z^network\.(\d+)\.(\d+)N)z^network\.(\d+)r))z ^cls_tokenr+)z^post_network\.(\d+)N)z^norm)i)stemrblocks2)r)r:r)s r> group_matcherzVOLO.group_matchers$41*  r?enablec||_y)zqSet gradient checkpointing. Args: enable: Whether to enable gradient checkpointing. N)r)r:r/s r>set_grad_checkpointingzVOLO.set_grad_checkpointings #)r?c|jS)zYGet classifier module. Returns: The classifier head module. )rr's r>get_classifierzVOLO.get_classifiersyyr?c"||_|||_t|jdr |jjj nd}t|jdr |jjj nd}|dkDr#tj|j|||ntj|_|jB|dkDr#tj|j|||ntj|_ yy)zReset classifier head. Args: num_classes: Number of classes for new classifier. global_pool: Global pooling type. Nr#rr&) rrhasattrrr#r'r(r0r1rrqr)r:rrr'r(s r>reset_classifierzVOLO.reset_classifiers'  "*D ,3DIIx,H!!((d*1$))X*F   &&DKVYZ?II   {6H`b`k`k`m == $OZ]^II!!;vULdfdododq M %r?r@cPt|jD]i\}}|dk(r ||jz}|j|}|jr+t j js t||}b||}k|j\}}}}|j|d|}|S)zForward pass through token processing stages. Args: x: Input tensor of shape (B, H, W, C). Returns: Token tensor of shape (B, N, C). rErF) enumeraterr r rr`r is_scriptingrrIrM)r:r@idxblockrQrRrSrTs r>forward_tokenszVOLO.forward_tokenss$DLL1 JCax&MM!$&&uyy/E/E/Gua(!H WW 1a IIaQ r?c2|j\}}}|jj|dd}tj||gd}|j D]A}|j r+tjjs t||}:||}C|S)zForward pass through class attention blocks. Args: x: Input token tensor of shape (B, N, C). Returns: Output tensor with class token of shape (B, N+1, C). rFrrG) rIrexpandr`rrrrr9r)r:r@rQrrT cls_tokensr;s r> forward_clszVOLO.forward_clss''1a^^**1b"5 IIz1o1 -&& E&&uyy/E/E/Gua(!H   r?c |j|}|jdddd}|jr0|jr#tj j |j|jj}|jd|jz|jd|jz}}t|j||j\}}}}|j} |j|z|j|z} } |j|z|j|z} } |jddd| | | | ddf| dd| | | | ddf<| }nd\}}}}|j|}|j |j#|}|j%|}|j&dk(r|j)d }n|j&d k(r |dddf}n|}|j*|S|j+|ddddf}|js|d |j-ddzzS|jr|jr|j/|jd|jd }|j} |jddd||||ddf| dd||||ddf<| }|j/|jd||z|jd }||||||ffS) a Forward pass for training with mix token support. Args: x: Input tensor of shape (B, C, H, W). Returns: If training with mix_token: tuple of (class_token, aux_tokens, bbox). Otherwise: class_token tensor. rrErDr)r/N)rrrravgrGr?rF)r rJrrr` distributionsBetarsamplerIrrrcloneflipr<rr@rrmeanrmaxrM)r:r@rpatch_hpatch_wrrrrtemp_xsbbx1sbby1sbbx2sbby2x_clsx_auxs r> forward_trainzVOLO.forward_trains    Q  IIaAq ! >>dmm%%**499dii@GGIC wwqzT-?-??tOaOaAaWG%.qvvxDDVDV%W "D$dWWYF--4d6H6H46O5E--4d6H6H46O5E56VVAYq%+uUZ{\]?]5^F1eEk5;1 2A%/ "D$d    "    (  #A IIaL   u $FFqFME    (adGEE == L a12h'}}31a00 0 >>dmmMM%++a.'7EKKPROTE[[]F16Aq$t)TRVYXY?Y1ZF1d4idA- .EMM%++a.'G2CU[[QS_UEedD$555r?indicesr stop_early output_fmtintermediates_onlyc |dvsJdg}tt|j|\}} |D cgc]} |j| }} |j| } |j\} } } }|j |j dddd}t jjs|s |j}n|jd| dz}t|D]\}}|dk(r ||jz}|j|}|jr+t jjs t||}n||}||vso|r|dk\r|j|}n|}|j!|j dddd|r|S|j\} }}}|j#| d|}|j$|j'|}|j|}||fScc} w) a Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to all intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )NCHWzOutput format must be NCHW.rrErDrNrF)rrrrIr rJr`rr9rr8r r rrrrrMrr@)r:r@rUrrVrWrX intermediates take_indices max_indexrrQ_heightwidthrr:r;x_interrRrSrTs r>forward_intermediateszVOLO.forward_intermediates$s*Y&E(EE& "6s4??7KW"U i4@Aq*A AOOI.  gg1fe   Q  ' '1a 3 99 ! ! #:llGll>IM2G#G, BJCax&MM!$&&uyy/E/E/Gua(!Hl"C1H"iilGG$$W__Q1a%@A B  WW 1a IIaQ     (  #A IIaL-OBsG" prune_norm prune_headc( tt|j|\}}|j|}|jd|dz|_|rt j |_|r+t j|_|jdd|S)aHPrune layers not required for specified intermediates. Args: indices: Indices of intermediate layers to keep. prune_norm: Whether to prune normalization layer. prune_head: Whether to prune classification head. Returns: List of kept intermediate indices. Nrr) rrrrr0rqrrrr6)r:rUrcrdr\r]s r>prune_intermediate_layerszVOLO.prune_intermediate_layerses|  "6s4??7KW"U iOOI. ||NY]3  DI  " D   ! !!R (r?c|j|jdddd}|j|}|j|j |}|j |}|S)zForward pass through feature extraction. Args: x: Input tensor of shape (B, C, H, W). Returns: Feature tensor. rrErDr)r rJr<rr@rrys r>forward_featureszVOLO.forward_featuressg   Q  ' '1a 3    "    (  #A IIaLr? pre_logitscL|jdk(r|jd}n|jdk(r |dddf}n|}|j|}|r|S|j|}|j4|j |ddddf}|d|j ddzz}|S)zForward pass through classification head. Args: x: Input feature tensor. pre_logits: Whether to return pre-logits features. Returns: Classification logits or pre-logits features. rBrrGrNrrC)rrIrrrrJ)r:r@rjoutauxs r> forward_headzVOLO.forward_heads   u $&&Q&-C    (AqD'CC NN1  Jiin == $--!QR%)Ccggajm++C r?cL |j|}|j|}|S)zForward pass (simplified, without mix token training). Args: x: Input tensor of shape (B, C, H, W). Returns: Classification logits. )rirnrys r>rWz VOLO.forwards- @  ! !! $   a r?F)T)N)NFFrZF)rFT)'rYrZr[r\r0r|rr]strrrr^r_r r}r.rr`rignoresetr(rrr.r1r3r6rar<r@r rTrbrgrirnrWrbrcs@r>rrs7(  #&#%.2-1,G2M""!#%$&$&*,,,5A!%"'!"3e'Ie'e' e'  e'  e'e'!e'!c+e' S *e'tSy)e' %T3Y/e'e'e'e' !!e'""#e'$"%e'&RYY'e'("%S/2)e'*+e', -e'./e'N -ryy -T - YY*** YY D T#s(^  , YY)T)T)) YY rCrhsmrW[r$ .U\\ell&;6u||;6ellE%,,X]XdXdfkloqtvy{~l~fJEA7A1B;6@8<$$',? ||? eCcN34?  ?  ?  ? !%?  tELL!5tELL7I)I#JJ K? F./$# 3S >*  c 8%,,5<<(ell4  %,, r?variant pretrainedc d|jdd}tt||fdt|di|S)zCreate VOLO model. Args: variant: Model variant name. pretrained: Whether to load pretrained weights. **kwargs: Additional model arguments. Returns: VOLO model instance. out_indicesrD feature_cfggetter)rw feature_cls)poprrr)rtrurrws r> _create_volor|sE**]A.K   [hG     r?urlc 4|ddddddttddd d |S) zCreate model configuration. Args: url: URL for pretrained weights. **kwargs: Additional configuration options. Returns: Model configuration dictionary. r)rDrrNQ?bicubicTzpatch_embed.conv.0)rrz apache-2.0) r}r input_size pool_sizecrop_pct interpolationfixed_input_sizerIr first_conv classifierlicenser )r}rs r>_cfgrs8=t)%.B*:N    r?ztimm/zLhttps://github.com/sail-sg/volo/releases/download/volo_1/d1_224_84.2.pth.tarr) hf_hub_idr}rzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d1_384_85.2.pth.tarr)rDrr)rr}rrzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d2_224_85.2.pth.tarzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d2_384_86.0.pth.tarzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d3_224_85.4.pth.tarzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d3_448_86.3.pth.tar)rDrzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d4_224_85.7.pth.tarzMhttps://github.com/sail-sg/volo/releases/download/volo_1/d4_448_86.79.pth.targffffff?zMhttps://github.com/sail-sg/volo/releases/download/volo_1/d5_224_86.10.pth.tarzLhttps://github.com/sail-sg/volo/releases/download/volo_1/d5_448_87.0.pth.tarzMhttps://github.com/sail-sg/volo/releases/download/volo_1/d5_512_87.07.pth.tar)rDr) zvolo_d1_224.sail_in1kzvolo_d1_384.sail_in1kzvolo_d2_224.sail_in1kzvolo_d2_384.sail_in1kzvolo_d3_224.sail_in1kzvolo_d3_448.sail_in1kzvolo_d4_224.sail_in1kzvolo_d4_448.sail_in1kzvolo_d5_224.sail_in1kzvolo_d5_448.sail_in1kzvolo_d5_512.sail_in1kc >tddddd|}tdd|i|}|S)VOLO-D1 model, Params: 27M.r+r+rrErrr rrrrrruro) volo_d1_224rr|rur model_argsmodels r>rr4p\6JVepiopJ  L: L LE Lr?c >tddddd|}tdd|i|}|S)rrrrrruro) volo_d1_384rrs r>rr rr?c >tddddd|}tdd|i|}|S)VOLO-D2 model, Params: 59M.rr+ r+rrrrrrrrruro) volo_d2_224rrs r>rr(4q]7KWfqjpqJ  L: L LE Lr?c >tddddd|}tdd|i|}|S)rrrrrruro) volo_d2_384rrs r>rr0rr?c >tddddd|}tdd|i|}|S)VOLO-D3 model, Params: 86M.rrrr+rrrruro) volo_d3_224rrs r>rr8rr?c >tddddd|}tdd|i|}|S)rrrrrruro) volo_d3_448rrs r>rr@rr?c >tddddd|}tdd|i|}|S)VOLO-D4 model, Params: 193M.rrrrrrrrrruro) volo_d4_224rrs r>rrH4r]7KWgrkqrJ  L: L LE Lr?c >tddddd|}tdd|i|}|S)rrrrrruro) volo_d4_448rrs r>rrPrr?c Btddddddd|}td d|i|}|S) jVOLO-D5 model, Params: 296M. stem_hidden_dim=128, the dim in patch embedding is 128 for VOLO-D5. rrr+rrr+rrrrgrruro) volo_d5_224rrs r>rrXB 4+?K[S4,24J  L: L LE Lr?c Btddddddd|}td d|i|}|S) rrrrr+rrruro) volo_d5_448rrs r>rrerr?c Btddddddd|}td d|i|}|S) rrrrr+rrruro) volo_d5_512rrs r>rrrrr?r) rrDrrErfFrrXNN)rfFrrXrp)rf)Fr\rKtypingrrrrrrr r r`torch.nnr0torch.nn.functional functionalrO timm.datar r timm.layersrrrrrr_builderr _featuresr _manipulater _registryrr__all__r}rrerrrrrqrr]r_rrrr^rrrrr|r default_cfgsrrrrrrrrrrrror?r>rsm* JJJ AZZ*+#< (OryyOdE EPA AH1"))1h>RYY>BD7D7N :# : : : >E#s(O>%>>E#sTWY\J\D]>BCCL""T "7777S 7  7  77777777 ]]!7@ "****S *  *  *****]]*Zw299wt#434*c#$sCx.*%! Z" Z0" Z" Z0" Z" Z0" Z" [-1" [" Z-1" [-1S-&- `DCDDCDDCDDCDDCDDCDDCDDCD D C D   D C D   D C D  r?