^jr+dZddlZddlmZmZmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZddlZddlmZddlmZmZmZmZddlmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+dd l,m-Z-dd l.m/Z/dd l0m1Z1m2Z2dd l3m4Z4m5Z5gd Z6eGddZ7eGddZ8dde e9dfde e:dfde9de e7dffdZ; dde e9dfde e:dfde e9dfde9de e e7f dZ< dde e=e=fde9de e9e e9fde>de e7dff dZ?d e e7ee7fde e7fd!Z@d"e e9d#e9de9fd$ZAeGd%d&ZBGd'd(ejZDd)e=d*e9d+e9d,e9d-e e9e9fd.eBde ejfd/ZEGd0d1ejZFGd2d3ejZGGd4d5ejZHGd6d7ejZIGd8d9ejZJGd:d;ejZKGd<d=ejZLeMeFeGeHeIeJeKeL>ZNd?e=d@ejfdAZOdBe e=ejffdCZPGdDdEejZR dd*e9d+e9dGe=dHe=dIe=d.eBf dJZSddLZTdMZUdNe e=efdOe7dPe8fdQZV ddUe:dVe9dWe9de e efdXZWdRdSddeVddfdYe8dZe:d[e9d\e e=efd]e:d^e9d_e e9d.e eBd`e efdaZXddYe8dce>fddZYGdedfejZZddgejdhe=die>ddfdjZ[eMdidke8e7dlddmdKddnoe7dldKdpdKddnoe7dqdrdsdKddtoe7dqdudsdKddvoe7dqdTdsdddvofdwddxydze8e7dlddmdKddnoe7dldKdpdKddnoe7dqdrdsdKddtoe7dqdTdsdKddvoe7dqddsdddvofdwddxyd{e8e7dldd|dKddnoe7dldSd|dKddnoe7dqd}d~dKddtoe7dqdKddKddvoe7dqdddddvofdddyde8e;dddd|de8e;ddddde8e;ddddde8e;dddde8e;dddde8e;ddTddde8e;dddde8e;ddTddde8e;dddde8e;ddTddde8e;dddddeMddde8e7dqdKdddwdtoe7dqdTddKdwdtoe7dqdrddKdwdtoe7dqdTddKddnofdmddddde8e7ddddddneMe7dqdTddKdwdtoe7dqdrddKdwdtoe7dqdTddKddnofdmddddeMdbde8e7dqdKdddwdtoe7dqdKddKdwdtoe7dqdKddKdwdtoe7dqdKddKdwdtofddddde8e7dqdKdddwdtoe7dqdKddKdwdtoe7dqdKddKdwdtoe7dqdKddKdwdtofdddddde8e7dqdKdddwdtoe7dqdKddKdwdtoe7dqdKddKdwdtoe7dqdKddKdwdtofdddddde8e7dqdKdddwdtoe7dqdKddKdwdtoe7dqdKddKdwdtoe7dqdKddKdwdtofdddddde8e7dqdKdddwdtoe7dqdKddKdwdtoe7dqdKddKdwdtoe7dqdKddKdwdtofdddddeMdde8e7dqdKddddtoe7dqdSddKddtoe7dqdSddKddtoe7dqdKddKddtofdddFddde8e7dqdKddddtoe7dqdSddKddtoe7dqdSddKddtoe7dqdKddKddtofdddFddde8e7dqdKddddtoe7dqdSddKddtoe7dqdSddKddtoe7dqdKddKddtofdddFddd«de8e7dqdKddddtoe7dqdSddKddtoe7dqdSddKddtoe7dqdKddKddtofdddFddd«de8e7dqdKddddtoe7dqdSddKddtoe7dqdSddKddtoe7dqdKddKddtofdddFddd«de8e7dqdSdddtƫe7dqdTddKdtƫe7dqdrddKdtƫe7dqdSddKdtƫfdddFdǫde8e7dqdSdddwdtoe7dqdTddKdwdtoe7dqdrddKdwdtoe7dqdSddKdwdtofdddddde8e7dqdKd|dKddSoe7dqdrddKddSoe7dqddpdKddSoe7dqdKddKddSofdwdFdFdddeMdtΫeMdbdbϫЫ de8e7dqdKd|dKddToe7dqdrddKddToe7dqddpdKddToe7dqdKddKddTofdwdFdFdddeMdtΫeMdbdbϫЫ de8e7dqdSdddwdToe7dqdrdmdKdwdToe7dqdddKdwdToe7dqdSd~dKdwdTofdddFdFdddeMdtΫeMdbdbϫԫ de8e7dqdSddddToe7dqdrdmdKddToe7dqdddKddToe7dqdSd~dKddTofdddFdFdddeMdtΫeMdbdbϫԫ de8e7dqdSddddToe7dqddpdKddToe7dqdd~dKddToe7dqdSddKddTofdddFdFdddeMdtΫeMdbdbϫԫ de8e7dqdKd|dKddSoe7dqdrddKddSoe7dqddpdKddSoe7dqdKddKddSofdwdFdFddee dʬثdeMdtΫeMdbdbϫ٫ de8e7dqdKd|dKddToe7dqdrddKddToe7dqddpdKddToe7dqdKddKddTofdwdFdFddee dʬثdeMdtΫeMdbdbϫ٫ de8e7dqdSddddToe7dqdrdmdKddToe7dqdddKddToe7dqdSd~dKddTofdddFdFddee dʬثdeMdtΫeMdbdbϫݫ de8edeZfdZad͐de=de e=effdZbd͐de=de e=effdZce4idebddebddebddddebdd d!"d#ebdd d!"d$ebdd d!"d%ebdd d!"d&ebdd d!"d'ebdd d!"d(ebdd d!"d)ebdd d!"d*ebdd d!"d+ebdd d!"d,ebdd d!d-d.dn/d0ebdd1d2ddd3dn4d5ecdd6d3dn7d8ecdd9d3dn7id:ecdd;d3dn7d<ecdd=d3dn7d>ecdd?d3dn7d@ecddAdBdCecddDd3dn7dEecddFd3dn7dGecddHd3dn7dIecddJd3dn7dKecddLd3dn7dMecddNd3dn7dOecddPd3dn7dQecddRdSdTdTdUdVdWd3dnX dYecddZdSdTdTdWd-dn[d\ecdd]dTdTd^d-_d`ecddadTdTdWd-dnbdcecddddTdTdWd-dnbdeecdSdTdTdUdVd^d3fidgecddhdSdTdTd^d-idjecddkdTdTd^d-dnbdlebddmdndopdqebddrdndopdsebddrdndopdtebddrdndopduebddrdndopdvecddeedbdUdVdwd!x dyecddeedbdUdVdwd!x dzecddseedbd3d{dwd!x d|ecdd}eedbd~ddwd!x decddeedbdddwd!x decddeedbdUdVdwd!x decddeedbdUdVdwd!x decddeedbdUdVdwd!x decddeedUdVd!decddeedUdVd!ecddeed3d{d!ecddeed~dd!ecddeeddd!ecddeedUdVd!ecddeedUdVd!ecddeedUdVd!ecddecddSdd^ddZde5ddeZfdZee5ddeZfdZfe5ddeZfdZge5ddeZfdZhe5ddeZfdZie5ddeZfdZje5ddeZfdZke5ddeZfdZle5ddeZfdZme5ddeZfdZne5ddeZfdZoe5ddeZfdZpe5ddeZfdZqe5ddeZfdZre5ddeZfdZse5ddeZfdZte5ddeZfdZue5ddeZfdZve5ddeZfdZwe5ddeZfdZxe5ddeZfdZye5ddeZfdZze5ddeZfdZ{e5ddeZfdZ|e5ddeZfdZ}e5ddeZfdZ~e5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfdZe5ddeZfd„Zy(a Bring-Your-Own-Blocks Network A flexible network w/ dataclass based config for stacking those NN blocks. This model is currently used to implement the following networks: GPU Efficient (ResNets) - gernet_l/m/s (original versions called genet, but this was already used (by SENet author)). Paper: `Neural Architecture Design for GPU-Efficient Networks` - https://arxiv.org/abs/2006.14090 Code and weights: https://github.com/idstcv/GPU-Efficient-Networks, licensed Apache 2.0 RepVGG - repvgg_* Paper: `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 Code and weights: https://github.com/DingXiaoH/RepVGG, licensed MIT MobileOne - mobileone_* Paper: `MobileOne: An Improved One millisecond Mobile Backbone` - https://arxiv.org/abs/2206.04040 Code and weights: https://github.com/apple/ml-mobileone, licensed MIT In all cases the models have been modified to fit within the design of ByobNet. I've remapped the original weights and verified accuracies. For GPU Efficient nets, I used the original names for the blocks since they were for the most part the same as original residual blocks in ResNe(X)t, DarkNet, and other existing models. Note also some changes introduced in RegNet were also present in the stem and bottleneck blocks for this model. A significant number of different network archs can be implemented here, including variants of the above nets that include attention. Hacked together by / copyright Ross Wightman, 2021. N) dataclassfieldreplace)partial) TupleListDictOptionalUnionAnyCallableSequenceType)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STDOPENAI_CLIP_MEANOPENAI_CLIP_STD)ClassifierHeadNormMlpClassifierHead ConvNormActBatchNormAct2d DropBlock2d EvoNorm2dS0aAttentionPool2dRotAttentionPool2dDropPathcalculate_drop_path_rates AvgPool2dSame create_conv2d get_act_layerget_norm_act_layerget_attnmake_divisible to_2tuple)build_model_with_cfg)feature_take_indices) named_applycheckpoint_seq)generate_default_cfgsregister_model)ByobNet ByoModelCfg ByoBlockCfgcreate_byob_stem create_blockc eZdZUdZeeejfed<e ed<e ed<dZ e ed<dZ e ee e fed<d Zeed <dZe eed <dZe eeefed <dZe eed <dZe eeefed<dZe eeefed<y)r.zuBlock configuration for Bring-Your-Own-Blocks. Defines configuration for a single block or stage of blocks. typedcsNgs?br attn_layer attn_kwargsself_attn_layerself_attn_kwargs block_kwargs)__name__ __module__ __qualname____doc__r strnnModule__annotations__intr6r7r r r9floatr:r;r r r<r=r>^/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/models/byobnet.pyr.r.Cs RYY  F F AsJ)-BsH}%&-BN!%J $,0K$sCx.)0%)OXc])15htCH~.5-1L(4S>*1rJr.ceZdZUdZeeeeedffdfed<dZe ed<dZ e ed<dZ e e ed <d Z eeeeeedffed <d Zeed <dZeed<dZeed<dZeed<dZe ed<dZe ed<dZe ed<dZe eed<dZe ed<dZe e ed<ed Zeed!<dZe e ed"<ed# Z eed$<ed% Z!e"e e#fed&<y)'r-zlModel configuration for Bring-Your-Own-Blocks network. Defines overall architecture configuration. .blocksconv1x1 downsample3x3 stem_typemaxpool stem_pool stem_chsr8 width_factorr num_featuresTzero_init_lastFfixed_input_sizerelu act_layer batchnorm norm_layeraa_layerNhead_hidden_size classifier head_typer:ctSNdictrIrJrKzByoModelCfg.tsdfrJ)default_factoryr;r<ctSrdrerIrJrKrgzByoModelCfg.vs46rJr=ctSrdrerIrJrKrgzByoModelCfg.wsrJr>)$r?r@rArBrr r.rFrOrCrQrSr rUrGrrVrHrWrXboolrYr[r]r_r`rbr:rr;rfr<r=r>r r rIrJrKr-r-Xs) % U;+;%<<=sB CCJIs(Ix}(79HeCcE#s(O349L%L#ND"d"Is!J!Hc'+hsm*!Is!!%J $n=K=%)OXc])">BdB#(#HL$sCx.HrJr-r3.wfgroupsreturnc d}d}dkDrfd}tt|||Dcgc]\}}}td|||z|c}}}}|Scc}}}w)zCreate RepVGG block configuration. Args: d: Depth (number of blocks) per stage. wf: Width factor per stage. groups: Number of groups for grouped convolution. Returns: Tuple of block configurations. @rc(|dzdzdk(r|zSdS)Nr%r5rrI)chsidxrms rKrgz_rep_vgg_bcfg..sa1}7IcVmqrJrep)r2r3r4r7)tuplezipr.)r3rlrmr4 group_sizebcfgs ` rK _rep_vgg_bcfgr}zsa AJ zP X[\]_`bdXeffHAqRT+5ARJGf gD KgsA rI se_blocksnum_conv_branchescjd}td|d|dz}|xsdt|z}g}t||||D]v\}}}}g} t|D]Y} ||z} t |} i} | ||z k\rd| d<| t d dd |d | d | gz } | t d dd | dt d d d i| d | gz } | }[|| gz }x|S) a,Create MobileOne block configuration. Args: d: Depth (number of blocks) per stage. wf: Width factor per stage. se_blocks: Number of SE blocks per stage. num_conv_branches: Number of conv branches. Returns: List of block configurations per stage. rprqr)r)rser:oner%)r2r3r4r7r> kernel_sizerI)minlenrzrangerfr.)r3rlr~rr4prev_cr|wrscfgiout_cbkaks rK_mobileone_bcfgrs" A QqTBqE\ "F*TCF]I D1aY/  1aq AEE(9:BBAF{#'< [XeqFqrXUWXY YD [^a5QT=Va=VSU=V^Z\^_ _DF    KrJtypeseveryfirstc t|dk(sJt|tr%tt |rdn|||dz}|s|dz g}t |g}t |D]"}||vr|dn|d}|t d|dd|gz }$t|S)a\Interleave 2 block types in stack. Args: types: Two block type names to interleave. d: Total depth of blocks. every: Interval for alternating blocks. first: Whether to start with alternate block. **kwargs: Additional block arguments. Returns: Tuple of interleaved block configurations. r5rr%)r2r3rI)r isinstancerGlistrsetr.ry)rr3rrkwargsrMr block_types rKinterleave_blocksrs& u:??%U15!UQY?@UGEJ F 1X@!"eU1Xq ;>J!>v>??@ =rJstage_blocks_cfgc t|ts|f}g}t|D]6\}}|t|jDcgc]}t |dc}z }8|Scc}w)zExpand block config into individual block instances. Args: stage_blocks_cfg: Block configuration(s) for a stage. Returns: List of individual block configurations. r%)r3)rr enumeraterr3r)r block_cfgsrcfg_s rKexpand_blocks_cfgrsh & 1,.J,-@3%,?Qwsa(?? @ @sAr{channelsc&|sy||zdk(sJ||zS)zCalculate number of groups for grouped convolution. Args: group_size: Size of each group (1 for depthwise). channels: Number of channels. Returns: Number of groups. r%rrI)r{rs rK num_groupsrs* *$))):%%rJceZdZUdZeZeeje d<e Z eeje d<ejZ eeje d<dZeeeje d<dZeeeje d<y)LayerFnz&Container for layer factory functions. conv_norm_actnorm_actactNattn self_attn)r?r@rArBrrrrDrErFrrReLUrrr rrIrJrKrrsm0%0M4 ?0 .Hd299o.77Cbii"&*D(4 ? #*+/IxRYY(/rJrceZdZdZ d dedededededeef fd Zd e jd e jfd Z xZ S) DownsampleAvgz[Average pool downsampling module. AvgPool Downsampling as in 'D' ResNet variants. in_chsout_chsstridedilation apply_actlayersc <||d} t ||xs t}|dk(r|nd} |dkDs|dkDr2| dk(r |dkDrtntj } | d| dd|_nt j|_|j||dfd|i| |_ y) aBInitialize DownsampleAvg. Args: in_chs: Number of input channels. out_chs: Number of output channels. stride: Stride for downsampling. dilation: Dilation rate. apply_act: Whether to apply activation. layers: Layer factory functions. devicedtyper%r5TF) ceil_modecount_include_padrN) super__init__rrrD AvgPool2dpoolIdentityrconv) selfrrrrrrrrdd avg_stride avg_pool_fn __class__s rKrzDownsampleAvg.__init__s* / $79'1}V! A:A+5?x!|-QSQ]Q]K#AzTUZ[DI DI(F((!WyWTVW rJxrncB|j|j|S)zoForward pass. Args: x: Input tensor. Returns: Output tensor. )rrrrs rKforwardzDownsampleAvg.forward&syy1&&rJ)r%r%FNNN) r?r@rArBrGrkr rrtorchTensorr __classcell__rs@rKrrs#(,XXX X  X  XW%X@ ' '%,, 'rJrdownsample_typerrrrrc |dvsJ||k7s|dk7s |d|dk7r6|sy|dk(rt||f||dd|S|j||fd||dd|StjS)aCreate shortcut connection for residual blocks. Args: downsample_type: Type of downsampling ('avg', 'conv1x1', or ''). in_chs: Input channels. out_chs: Output channels. stride: Stride for downsampling. dilation: Dilation rates. layers: Layer factory functions. **kwargs: Additional arguments. Returns: Shortcut module or None. )avgrNr^r%rNrrr)rrr)rrrDr)rrrrrrrs rKcreate_shortcutr2s. 4 44 4 FaK8A;(1++E  % `(ST+`Y_` `'6''vQv`hij`kvouv v{{}rJceZdZdZ ddededededeeefdeeded ed e d e d e d e deffd Z dde fdZ dZxZS) BasicBlockz$ ResNet Basic Block - kxk + kxk rrrrrr{ bottle_ratiorO attn_last linear_outr drop_blockdrop_path_ratec||d}t|| xs t} t||z}t ||}t |||f||d| d||_| j|||f||dd||_| s | jtjn| j||_ | j|||f|d|| dd||_ | r | jtjn| j|fi||_ | dkDr t| ntj|_| rtj|_y| j!d |_y) NrFrrrrrrr%)rrm drop_layerrTinplace)rrrr#rrshortcutr conv1_kxkrrDr conv2_kxkrr drop_pathr)rrrrrrr{rrOrrrrrrrrmid_chsrmrs rKrzBasicBlock.__init__Ysq$ / $79 >,,33 4YY/ (Dt/0%%' (rJc|}|j|}|j|}|j|}|j|}|j |}|j ||j |z}|j |Srd)rrrrrrrrrrs rKrzBasicBlock.forwardw NN1  IIaL NN1  NN1  NN1  == $DMM(++Axx{rJ) r%r%r%Nr8rTFNNrNNF)r?r@rArBrGrr rHrCrkrr rrrrrs@rKrrUs !(.(,"%#"$"#'$&!1M1M1M 1M  1M CHo 1M! 1M 1M1M1M1M1M!1M"1Mf(4( rJrceZdZdZ ddededededeeefdedeed ed e d e d e d e de de deffd Z dde fdZ dZxZS)BottleneckBlockz4 ResNet-like Bottleneck Block - 1x1 - kxk - 1x1 rrrrrrr{rOrr extra_conv bottle_inrrrc|||d}t|| xs t} t| r|n||z}t ||}t |||f||d| d||_| j||dfi||_| j|||f||d||d||_ | r!| j|||f|d|d||_ ntj|_ | s | jtjn| j|fi||_ | j||dfddi||_| r | jtjn| j|fi||_|d kDr t!|ntj|_| rtj|_y| j%d |_y) NrFrr%rrrrmr)rrmrrTr)rrrr#rrrr conv1_1x1r conv2b_kxkrDrr conv3_1x1rrrr)rrrrrrrr{rOrrrrrrrrrrrrmrs rKrzBottleneckBlock.__init__s( / $79 I&7l!RSJ0'          .--fgqGBG---     a[!     2f22"!   DO!kkmDO%.&++2EBKKM;6;;W^KebdKe ---gwYUYVXY.76;;;NT_TZT_T_`gTnkmTn5Cb5H.1bkkm$.2;;=FJJtJ4LrJrXcP|rj|j^t|jjdd=tj j |jjj|j|jfD]}t|ds|j!yr) rrrrrDrrrrrrrrs rKrzBottleneckBlock.init_weightsrrJcV|}|j|}|j|}|j|}|j|}|j |}|j |}|j |}|j||j|z}|j|Srd) rrrrrrrrrrs rKrzBottleneckBlock.forwards NN1  NN1  OOA  IIaL NN1  NN1  NN1  == $DMM(++Axx{rJ)rr%rr8NrFFFFNNrNNrr?r@rArBrGrrHr rCrkrr rrrrrs@rKrrs !(."$(,##$$#"#'$&%?M?M?M ?M  ?M CHo ?M ?M! ?M?M?M?M?M?M?M!?M "!?MB(4( rJrceZdZdZ ddededededeeefdedeed ed e d e d e d e deffd Z dde fdZ dZxZS) DarkBlocka  DarkNet-like (1x1 + 3x3 w/ stride) block The GE-Net impl included a 1x1 + 3x3 block in their search space. It was not used in the feature models. This block is pretty much a DarkNet block (also DenseNet) hence the name. Neither DarkNet or DenseNet uses strides within the block (external 3x3 or maxpool downsampling is done in front of the block repeats). If one does want to use a lot of these blocks w/ stride, I'd recommend using the EdgeBlock (3x3 /w stride + 1x1) for more optimal compute. rrrrrrr{rOrrrrrc ||d}t|| xs t} t||z}t ||}t |||f||d| d||_| j||dfi||_| s | jtjn| j|fi||_ | j|||f||d|| dd||_ | r | jtjn| j|fi||_ | dkDr t| ntj|_| rtj|_y| j!d |_y) NrFrr%rrrrmrrrTr)rrrr#rrrrrrrDrrrrrrrrrrrrrr{rOrrrrrrrrrrmrs rKrzDarkBlock.__init__sl$ / $79 XIOXOFOOGKuKK^bDM0F00 !!%  DM1F00 DM:H"9LQZXn5`b`k`k`mDN%+[[%8BKKMkfkk'>XUW>X ::d:+rJrXcr|jD]s}t|tjstjj |j ddtjj |jddut|jdr|jjyy)Ng?rr) modulesrrD BatchNorm2drnormal_rrrrr)rrXms rKrzRepVggBlock.init_weightss{ /A!R^^,"b12. / 4990 1 II & & ( 2rJc|j/|j|j|j|S|j$|j ||j |z}nJ|j|}|j ||j |z}|j |}||z }|j|}|j|Srd)rrrrrrr)rrrs rKrzRepVggBlock.forwards    (88DIId&7&7&:;< < ==  a 4==#33A}}Q'H a 4==#33Aq!A MA IIaLxx{rJc z|jy|j\}}tj|jj j |jj j|jj j|jj j|jj j|jj j|jj jd|_||jj_||jj_|j!D]\}}d|vr |j#|j%d|j%d|j%d|j%dy)  Following works like `RepVGG: Making VGG-style ConvNets Great Again` - https://arxiv.org/pdf/2101.03697.pdf. We re-parameterize multi-branched architecture used at training time to obtain a plain CNN-like structure for inference. NTrrrrpaddingrrmrrrrrrr_get_kernel_biasrDrrrrrrrr&rrmrdatarnamed_parametersdetach_ __delattr__rkernelrnameparas rKreparameterizezRepVggBlock.reparameterizesT    ( ,,. II **66++88 **66==%%,,MM&&..]]''00==%%,,  )/  %&*#//1 JD$% LLN  $ $ $ %rJrncd}d}|jr|j|j\}}|jjjddz}t j jj|||||g}d}d}|j|j|j\}}|j|j\}}||z|z}||z|z} || fSz Method to obtain re-parameterized kernel and bias. Reference: https://github.com/DingXiaoH/RepVGG/blob/main/repvgg.py#L83 rr5) r_fuse_bn_tensorrrrrrD functionalpadr) r kernel_1x1bias_1x1r6kernel_identity bias_identity kernel_conv bias_conv kernel_final bias_finals rKr(zRepVggBlock._get_kernel_biass   == $#'#7#7 #F J--$$003q8C,,00c3S=QRJ == $-1-A-A$---P *O]"&!5!5dmm!D Y"Z//A )M9 Z''rJct|tr|jj}|jj }|jj }|jj}|jj}|jj}n!t|tjsJt|ds|jjj}||jz} |jjj} t!j"|jjj} t%|D]} d| | | | z| ddz| ddzf<| |_|j&}|j }|j }|j}|j}|j}||zj)} || z j+dddd}||z|||z| z z fS) Method to fuse batchnorm layer with preceding conv layer. Reference: https://github.com/DingXiaoH/RepVGG/blob/main/repvgg.py#L95 id_tensorr%rr5rrrrr running_mean running_varrepsrDrrrrrmrr zeros_likerrAsqrtreshaperbranchr.rDrEgammabetarFr input_dimr kernel_valuerstdts rKr4zRepVggBlock._fuse_bn_tensor0s fk *[[''F!9911L ))//KII$$E99>>D))--Cfbnn5 554-++77"dkk1 "mm00<< $// 0B0B0I0IJ vaA_`LA M;q>Q3F TUZ[H[![\a!-^^F!..L ,,KMME;;D**CS &&( S[ ! !"aA .z4,"6"<<<XIOXOFOOGKuKK^bDME4112  1V11!,!!#  MM%0DM#DOQ"6&"6"6#!"!!# ##:H"9LQZXn5`b`k`k`mDN%+[[%8BKKMkfkk'>XUW>X ::d:+rJrc|j/|j|j|j|Sd}|j|j|}d}|j|j |}|}|j D] }|||z }|j |}||z }|j|j|S)z Apply forward pass. r)rrrrrWrr)rr identity_out scale_outoutcks rKrzMobileOneBlock.forwards    (88DIId&7&7&:;< < == $==+L ?? &*I-- B 2a5LC nnS! |xx #''rJc |jy|j\}}tj|jdj j |jdj j|jdj j|jdj j|jdj j|jdj j|jdj jd|_||jj_||jj_|j!D]\}}d|vr |j#|j%d|j%d|j%d|j%d y) r$NrTr%rrrWrrr'r-s rKr1zMobileOneBlock.reparameterizesw    ( ,,. II a(--99q)..;; a(--99==#((//MM!$))11]]1%**33==#((//)/  %&*#//1 JD$% LLN  $ & $ %rJc"d}d}|ju|j|j\}}|jdjjddz}t j jj|||||g}d}d}|j|j|j\}}d}d}t|jD]-}|j|j|\} } || z }|| z }/||z|z} ||z|z} | | fSr3) rWr4rrrrrDr5r6rrr) r kernel_scale bias_scaler6r9r:r;r<ix_kernel_biasr=r>s rKr(zMobileOneBlock._get_kernel_biass,   ?? &'+';';DOO'L $L*--"''33A6!;C 88..22<#sCQTAUVL == $-1-A-A$---P *O]  ../ B!11$--2CDNGU 7 "K  I  #\1OC +m; Z''rJct|tr|jj}|jj }|jj }|jj}|jj}|jj}n*t|tjsJt|ds|jdjj}||jz} |jdjj} t!j"|jdjj} t%|D]} d| | | | z| ddz| ddzf<| |_|j&}|j }|j }|j}|j}|j}||zj)} || z j+dddd}||z|||z| z z fS)r@rArr%r5rBrCrJs rKr4zMobileOneBlock._fuse_bn_tensors fk *[[''F!9911L ))//KII$$E99>>D))--Cfbnn5 554-q)..::"dkk1 "mmA.33?? $// a0@0E0E0L0LM vaA_`LA M;q>Q3F TUZ[H[![\a!-^^F!..L ,,KMME;;D**CS &&( S[ ! !"aA .z4,"6"<<<=u||U\\/I)J=rJrSc!eZdZdZ ddededededeeefdedeed ed e d e d e d e deeeefde de def fd Z dde fdZ dZxZS) SelfAttnBlockzI ResNet-like Bottleneck Block - 1x1 - optional kxk - self attn - 1x1 rrrrrrr{rOrrr post_attn_na feat_sizerrrc||d}t||Jt| r|n||z}t||}t |||f||d|d||_|j ||dfi||_| r%|j |||f||d||d||_d}ntj|_| in t| }|j|fd|i|||_ | r|j|fi|ntj|_|j ||dfd di||_|d kDr t!|ntj|_| rtj|_y|j%d |_y) NrFrr%rr)rirrrTr)rrr#rrrrrrrDrrfrr post_attnrrrr)rrrrrrrr{rOrrrrhrirrrrrrrrm opt_kwargsrs rKrzSelfAttnBlock.__init__"s* / !!! I&7l!RSJ0'          .--fgqGBG 1V11 !!%  DNF[[]DN$,R$2K )))'U&UJURTU;G7B7R[[]---gwYUYVXY5Cb5H.1bkkm$.2;;=FJJtJ4LrJrXc>|rj|j^t|jjdd=tj j |jjjt|jdr|jjyyr) rrrrrDrrrrrr)rrXs rKrzSelfAttnBlock.init_weights_sl dmm7GDNNDUDUW_ae>,,33 4 4>>#5 6 NN + + - 7rJc4|}|j|}|j|}|j|}|j|}|j |}|j |}|j ||j |z}|j|Srd)rrrrkrrrrrs rKrzSelfAttnBlock.forwardes NN1  NN1  NN1  NN1  NN1  NN1  == $DMM(++Axx{rJ)rr%rr8NrFFFTNNNrNNrrrs@rKrgrgs !(."$(,#$$#!%37"#'$&';M;M;M ;M  ;M CHo ;M ;M! ;M;M;M;M;M;M c3h0;M;M !!;M""#;Mz.4. rJrg)basicbottledarkedgerxrrrblock_fnc|t|<yrd)_block_registry)rrss rKregister_blockrv}s "*OJrJblockc t|tjtfr|di|S|tvs Jd|t |di|S)NzUnknown block type (rI)rrDrErru)rwrs rKr0r0sO%"))W-.v O #C';E7%CC # 5 ! +F ++rJceZdZ ddedeeeeeedffdededededeed e d e ffd Z d ee jee jffd ZxZS)Stemrr.rrrnum_repnum_act chs_decayrc .| | d} t||dvsJ| xs t} t|tt frt |}|} n.t|Dcgc]}t|||zzc}ddd} ||_ g|_ d}dgdg|dz zz}|dk(r|sd|d<||n|}dg||z zd g|zz}|}d}d}tt| ||D]\}\}}}|r | jnt}d |dz}|d kDr2|dkDr-|dz }|jjt!|||d |j#||||f||d | |}||z}|}|r|j%}|dvsJ}|jjt!|||d |dk(r&|j#dt'j(dn|dk(r&|j#dt'j*dnZd|vr)|j#dt'j(dddn-d|vr)|j#dt'j*dddd|dz}d}|d k\r|nd|_|jjt!|||d ||k(sJycc}w)Nr)r5rBr^r5r%rFTrrnum_chs reductionmodulestage)rr)maxrRravgpoolmax2avg2rrrrr)rrr&r)rrr&r)rrrrrryrrroundr feature_inforrzrrrUrf add_modulelowerrD MaxPool2dr last_feat_idx)rrrrrrr{r|r}rrrrrUr prev_feat stem_stridesstem_norm_actsprev_chs curr_striderchr6nalayer_fn conv_namers rKrz Stem.__init__s / $79 ge} -'lGHAFwPAg Q67PQUSUQUVH  saSGaK00 Q;t L $_''Gg$56$'9II  'HlN(ST "NA{Ar/1v++}Hq1ugI1uQ !A !!((h+^gop)qr OOIx"'f+^_'fce'f gH 1 K!I " ::T0T*UrJrzr^rQ pool_type feat_prefixc  ||d}|xs t}|dvsJd|vrd|vrdnd} t||fd| ||d|} nd|vrt|d |zd z|dz|ff||d |} nd |vrt||fd d ||d|} nd|vrt||fd|d|} nd|vrt||fd d|d|} nd|vr-|rt||dfd||d|} nl|j||dfddi|} nTt |t tfrt||d f||d |} n,|rt||d fd||d|} n|j||d fddi|} t | tr<| jD cgc]#} t| dj|| dg%} } | | fSt|d|dg} | | fScc} w)Nr) r^quadquad2tiereddeeprxr7x7rPrrr5r)r{r|rrrr)rrrr8)r{r}rrrx)rrr)rrrrr%)r{rrr.r)rrr) rrzrrSrrryrrrfjoin) rrrQrrrrrrr|stemfrs rKr/r/s U +B  wyF [ [[ [ )+!FGeQiX^ebde Y FQ[A-w!|WEkI^dkhjk 9 FGcQ#IV\c`bc ) 67J1VJrJ ) fg\1Qv\Y[\ )  [AIf[XZ[D'6''K1KKD gt} -P6PRPDFGQ_ RX_\^_+v++FGQOqOBO$VZVgVghQRQsxxak0J'KLh h  W+UVWX  is?(Fr5cL|dSt|Dcgc]}||z c}Scc}wrd)ry)rirr6s rKreduce_feat_sizer s)$4Q%i0Pf0P*QQ0Ps !c||n|}|xsiS)a2 Override model level attn/self-attn/block kwargs w/ block level NOTE: kwargs are NOT merged across levels, block_kwargs will fully replace model_kwargs for the block if set to anything that isn't None. i.e. an empty block_kwargs dict will remove kwargs set at model level for that block rI)r> model_kwargs out_kwargss rKoverride_kwargsrs".!9|J  rJr> block_cfg model_cfgc|d}|jdu}|s |jq|r|jsd}nSt|j|j}|jxs |j}|tt |fi|nd}t ||}|j du}|s |jq|r|j sd}nSt|j|j} |j xs |j }|tt |fi| nd}t ||}||d<|jt|j|jy)Nr)rr) r:r;rrr"rr<r=updater>) r>rr layer_fnsattn_setr:r; self_attn_setr<r=s rKupdate_block_kwargsrsMX&I##4/H9((4 I00J))*?*?AVAVWK"--E1E1EJISI_*!5EEeiJIJ7 --T9M 22> !:!:"O.y/I/I9KeKef '77T9;T;TO".&h&?TCST48 IA &L (>(> @V@VWXrJrrr drop_prob block_size num_stagesc|dk\sJdg|z}|r2tt||dzdz d|d<tt||d|d<|S) awCreate DropBlock layer partials for each stage. DropBlock is applied to the last two stages only, following common practice. The block_size specifies the size for the final stage; the second-to-last stage uses a larger block size scaled to account for 2x larger feature maps. Args: drop_prob: Drop probability for DropBlock. block_size: Block size for the final stage. Second-to-last stage uses `block_size * 2 - 1` to scale with feature map size. num_stages: Number of stages in the model. Returns: List of DropBlock partial instances or None for each stage. r5Nr%?)rr gamma_scaler8rB)rr)rrrdbss rK drop_blocksr?sY( ?? &: C+zTU~XYGYgklB+z_cdB JrJrr output_stride stem_featdrop_block_ratedrop_block_sizeriblock_kwargs_fnc |xs t}g} |jD cgc] } t| } } t| }| Dcgc]%}t |Dcgc]}|j c}'}}}t ||d}t|||}d}|d}|d}|}g}t| D]c\}}|dj}|dk7r|r| j|||k\r |dkDr||z}d}||z}|dvrdnd}g}t|D]\}}t|j|jz} |j}!t|!t r |!| |}!t#|| |dk(r|nd||f|!|j$|j&||||||| | }"|j(d vr||"d <||"|| |t+|j(fi|"gz }|}| }|dkDs|dk(st-||}|t/j0|gz }t#||d ||dz}f| j|t/j0|| |fScc} wcc}wcc}}w)NT) stagewiser%rrr)r%r5r5) rrrrr{rrOrrrrrrri)rrstages.r)rrMrrsumr3rrrr6rUr#r4rVr7rr rfr9rOr2r0rrD Sequential)#rrrrrrrirrrrrr6rr stage_bcsbcdepthsdprrr net_striderrstages stage_idxstage_block_cfgsrfirst_dilationrM block_idxrrr{r>s# rKcreate_byob_stagesr\s wyFL03 ;1#A&;J;ZJ?I J)c),B244,- JF J #NFd KC o CCH;'J#HI F'0'<)t# #!!$&& Q;9    *  &6A:  HFf &&0a$-.>$? @ Iy$Y[[33C3C%CDG"J*h/'; !*avQ((3%&\\>>y>"9~i8 L~~/,5 [) LI M |INNClCD DF%NHzi1n,Y? 7 @: 2==&)**Z'R[Q\H]enqrers S)tV " ==& !< ::m<, JsH3H=H8$ H=8H=Tallow_aact|j}t|j|}|jr*|r(t t |j||j}nt t |j|}|jr)t t|jfi|jnd}|jr)t t|jfi|jnd}t|||||}|S)N)r]r[)r]r[r_)rrrrr) r r[r!r]r_rrr:r"r;r<r=r)rrrrrrrrs rK get_layer_fnsrs  &C!S^^sKH || RU`c`l`lm  RUV CF>>78CNN+ ?s ?W[DRUReRe!4!45N9M9MNkoI]X3UYenoH OrJc*eZdZdZ d%dedededeededeeee eeffd e d e d ed e d e ffd Z e jjd&de deeeffdZe jjd'de ddfdZe jjdej*fdZd(dedeeddfdZ d)de j0deeeeefde de dede de deee j0e e j0ee j0fffdZ d*deeeefde de deefd Zde j0de j0fd!Zd&de j0d"e de j0fd#Zde j0de j0fd$ZxZS)+r,a%Bring-your-own-blocks Network. A flexible network backbone that allows building model stem + blocks via dataclass cfg definition w/ factory functions for module instantiation. Current assumption is that both stem and blocks are in conv-bn-act order (w/ block ending in act). Nr num_classesin_chans global_poolrimg_size drop_raterrrrXc  t|| | d}||_||_||_d|_t |fi|}t|d}t|}|jr |Jd| t|nd}g|_ t|jttfr9|jDcgc]#}tt!||j"z%}}nFtt!|jxs|j$dj&|j"z}t)d|||j*|j,|d|\|_}|jj1|ddt3||dd  }t5|| ||df|| ||d |\|_}}|jj1|dd|dd }|dd }|j8rUtt!|j"|j8z|_|j:||j8d fi||_n ||_t?j@|_|xjtC|j8|dtE|j6gz c_ |jDcgc]}|d c}|_#|j8|_$|jJdvsJ|jJdk(rk|d}tM|j8|f|jH||jN|jP|jd||_)|jRjT|_$n|jJdk(r\|d}|dvsJtW|j8f|jH||||jdd||_)|jRjX|_$n|jJdk(r\|d}|dvsJt[|j8f|jH||||jdd||_)|jRjX|_$n;|d}|jHJt]|j8|f||jd||_)||_/tatctd| |ycc}wcc}w)a Args: cfg: Model architecture configuration. num_classes: Number of classifier classes. in_chans: Number of input channels. global_pool: Global pooling type. output_stride: Output stride of network, one of (8, 16, 32). img_size: Image size for fixed image size models (i.e. self-attn). drop_rate: Classifier dropout rate. drop_block_rate: DropBlock drop rate. drop_block_size: DropBlock block size for final stage (scales up for earlier stages). drop_path_rate: Stochastic depth drop-path rate. zero_init_last: Zero-init last weight of residual path. **kwargs: Extra kwargs overlayed onto cfg. rF)rNz8img_size argument is required for fixed input size modelr)rrrQrrrBr)r)rrrrirr% final_convrr)r^ramlpattn_absattn_rotrr) hidden_sizerr]r[rrtoken)r^rT) embed_dim out_featuresrirr qkv_separater)rr ref_feat_sizerrr)rrrXrI)3rrrrrgrad_checkpointingrrrYr$rrrUrryrGrrVrMr4r/rQrSrextendrrrrWrrrDrrfr stage_endsr`rbrr]r[headrrrrrrr(r _init_weights)rrrrrrrrrrrrXrrrr stem_layers stage_layersrir4rUr stage_featrrrrs rKrzByobNet.__init__s@  /&  ""'c$V$#C%8 $S)   ' c)c c'+3+?Ih'T  cllT5M 2BE,,OQE!c&6&6"678OHO5#,,"A#**Q-//SEUEU!UVWH/ mmmm     9   3B0$Yy}[7QR -?    bM . ,+ .  . * Z   CR1rN;/ b>),    #E#*:*:S=M=M*M$N OD 8l884CTCTVW^[]^DO (D  kkmDO  **i \_`d`k`k\l mo o/3/@/@A!1W:A $ 1 1}} QQQQ ==E !"# -!!  00%>>--..  DI%)II$9$9D ! ]]j ("% -/ //'!! ..(#%..!  DI%)II$7$7D ! ]]j ("% -/ //*!! ..('%..!  DI%)II$7$7D !"# ''/ //&!!&..   DI' GM.I4PGPJBs .(Q= Rcoarsernc0td|rdnddfdg}|S)zGroup matcher for parameter groups. Args: coarse: Whether to use coarse grouping. Returns: Dictionary mapping group names to patterns. z^stemz^stages\.(\d+)z^stages\.(\d+)\.(\d+)N)z ^final_conv)i)rrMre)rrmatchers rK group_matcherzByobNet.group_matcherOs.&,"2JDQ* rJenablec||_y)zEnable or disable gradient checkpointing. Args: enable: Whether to enable gradient checkpointing. N)r)rrs rKset_grad_checkpointingzByobNet.set_grad_checkpointingbs #)rJc.|jjS)zPGet classifier module. Returns: Classifier module. )rfc)rs rKget_classifierzByobNet.get_classifierksyy||rJcJ||_|jj||y)zReset classifier. Args: num_classes: Number of classes for new classifier. global_pool: Global pooling type. N)rrreset)rrrs rKreset_classifierzByobNet.reset_classifierts'  [1rJrindicesnorm stop_early output_fmtintermediates_onlyexclude_final_convcN|dvsJdg}tt|j|\} } | D cgc]} |j| } } |j| } d} t|jdr|jj |\}} n|j |d} }| | vr|j | |n| |jd}tjjs|s |j}n|jd| }|D]t}| dz } |jr+tjjs t||}n||}|s| |k(r|j|}| | vsd|j |v|r|S|r| |k(r|j|}||fScc} w)a9 Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to compatible intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features exclude_final_conv: Exclude final_conv from last intermediate Returns: )NCHWzOutput shape must be NCHW.rrNrBr%)r'rrrrrrUrjit is_scriptingrrr)r)rrrrrrrr intermediates take_indices max_indexrfeat_idxx_interlast_idxrrs rKrzByobNet.forward_intermediates~s.Y&D(DD& "6s4??7KW"U i4@Aq*A AOOI.  4995 688;JAw1twA | #  go7 C??2& 99 ! ! #:[[F[[),F (E MH&&uyy/E/E/G"5!,!H%(h*>OOA&<'$$Q' (  (h"6"A-EBsF" prune_norm prune_headctt|j|\}}|j|}|jd||_||jdkrt j |_|r|jdd|S)aEPrune layers not required for specified intermediates. Args: indices: Indices of intermediate layers to keep. prune_norm: Whether to prune normalization layer. prune_head: Whether to prune the classifier head. Returns: List of indices that were kept. NrBrr^)r'rrrrDrrr)rrrrr r s rKprune_intermediate_layersz!ByobNet.prune_intermediate_layerssu #7s4??7KW"U iOOI. kk*9- tr* * kkmDO   ! !!R (rJc|j|}|jr5tjj st |j |}n|j |}|j|}|S)zForward pass through feature extraction. Args: x: Input tensor. Returns: Feature tensor. )rrrr r r)rrrs rKforward_featureszByobNet.forward_featuressY IIaL  " "599+A+A+Ct{{A.A AA OOA rJ pre_logitscN|r|j||S|j|S)zForward pass through head. Args: x: Input features. pre_logits: Return features before final linear layer. Returns: Classification logits or features. )r)r)rrrs rK forward_headzByobNet.forward_heads(7Atyyzy2RdiiPQlRrJcJ|j|}|j|}|S)zoForward pass. Args: x: Input tensor. Returns: Output logits. )rrrs rKrzByobNet.forwards)  ! !! $   a rJ) rNrTNrrrrTNNrTrd)NFFrFF)r%FT) r?r@rArBr-rGr rCr rrHrkrrr ignorer r rrrDrErrrrrrrrrrrs@rKr,r,s $)-!#>B!%'#$$&#'SQSQSQ SQ "# SQ  SQuS%S/%9:;SQSQ#SQ!SQ"SQ!SQj YYDT#s(^$ YY)T)T)) YY 2C2hsm2W[28<$$',',< ||< eCcN34<  <  <  < !%< !%<  tELL!5tELL7I)I#JJ K< @./$# 3S >*  c 2%,,5<<" Sell S S S  %,, rJr,rr/rXct|tjr|jd|jdz|jz}||j z}|j jjdtjd|z |j%|jjjyyt|tjrctjj|j dd|j*tjj|jyyt|tj rStjj#|j tjj|jyt%|dr|j'| yy) zInitialize weights. Args: module: Module to initialize. name: Module name. zero_init_last: Zero-initialize last layer. rr%@Nrg{Gz?)meanrPrr)rrDrrrrmrr)r mathrHrzero_Linearrrrones_rr)rr/rXfan_outs rKrrs8&"))$$$Q'&*<*? ;; " KK   " " $ # FBII &  CT: ;; " GGNN6;; ' # FBNN +  fmm$ v{{#  (>: )rJgernet_lrorrr8)r2r3r4r6r7r9rpir@rTi )rMrUrSrWgernet_mgernet_s0ri0rs i repvgg_a0)r5rr%)?r2r2@)r3rlrx)rMrQrU repvgg_a1)r%r%r%r3rq repvgg_a2)?r6r6g@ repvgg_b0)r8r8r8r3)rl repvgg_b1)rrr@ repvgg_b1g4)rlrm repvgg_b2)r3r3r3@ repvgg_b2g4 repvgg_b3)r*r*r*r< repvgg_b3g4 repvgg_d2se)rr1r%rg?)rd_ratio rd_divisor)rMrQrUr:r; resnet51qrtirisilu)rMrUrQrSrWr[ resnet61qrr)r2r3r4r6r7r9r>r)r)rMrUrQrSrWr[r> resnext26tsirrR)rMrUrQrSr[ gcresnext26tsgca)rMrUrQrSr[r: seresnext26tseca_resnext26tsecabat_resnext26tsbatr)r)rMrUrQrSr[r:r; resnet32ts resnet33tsi gcresnet33ts)rMrUrQrSrWr[r: seresnet33tseca_resnet33ts gcresnet50t)r2r3r4r6r9)rMrUrQrSr: gcresnext50ts regnetz_b16` )rB)rr) rMrUrSrOrWr[r:r;r> regnetz_c16 regnetz_d32i) rMrUrQrSrOrWr[r:r;r> regnetz_d8 regnetz_e8regnetz_b16_evos)r{) rMrUrSrOrWr[r]r:r;r>regnetz_c16_evosregnetz_d8_evosr) rMrUrQrSrOrWr[r]r:r;r> mobileone_s0)r2r8r8r)rlrr mobileone_s1)r6r6rr3 mobileone_s2)r6rr3r9 mobileone_s3)rr3r*r9 mobileone_s4)r* @rgr9)rrr)r%)rlr~ resnet50_clip)rTrTrqrrr)rMrUrQrSrOr_rbresnet101_clipresnet50x4_clip g?)rMrVrUrQrSrOr_rbresnet50x16_clipr6resnet50x64_clip$r resnet50_mlpr)rMrUrQrSrOr_r`rb test_byobnetrrqrArZ)rMrUrOrSr[r:r;)rhrirkrmrora)rb_gap state_dictmodelprefixcxt|jttf}ddl}d}d}i}|j D]\}} |j |s|j|dd|}|j|dd|}|j|d||}|j|d ||}|j |d rr|s|j|d zd }|jd d }|jdd}|jdd}|jdd}|jdd}| ||<|S)Nrct|jddz }t|jd|jdt|jd}}}d|d|d}ddd d }|||z}||zS) Nr%r5rrrrz conv1_1x1.z conv2_kxk.z conv3_1x1.)r%r5rrGgroup)r!r layer_idx layer_typelayer_id prefix_strid_map suffix_strs rK _stage_subz(_convert_openai_clip.._stage_sub^ s Oa' *-aggaj/1771:s177ST:x: yk9+Q7 !l|DH% 2 J&&rJct|jddz }t|jdt|jd}}d|d|d|dk(rdzSd zS) Nr%r5rrrz .shortcut.rz conv.convzconv.bnrz)r!rr|r~s rK _down_subz'_convert_openai_clip.._down_subf sc Oa' !!''!*os1771:8  1YKz:XYZ]kjj`ijjrJz conv([0-9])zstem.conv\1.convz bn([0-9])zstem.conv\1.bnz'layer([0-9])\.([0-9]+)\.([a-z]+)([0-9])z+layer([0-9])\.([0-9]+)\.downsample\.([0-9])attnpoolrpositional_embedding pos_embedq_projqk_projkv_projvc_projproj) rrrrreitems startswithsubr) rurvrwmodel_has_attn_poolrrrout_dictrrs rK_convert_openai_cliprV sT %UZZ2Do1VW 'k H  "1||F#  FFvhk*,? C FFvhi(*;Q ? FFvhEF TU V FFvhIJIWX Y <<6((+ ,& &:-v6A 0+>A (C(A (C(A (C(A (F+A !$ OrJc&d|vr t||}|S)Nzvisual.conv1.weight)r)rurvs rKcheckpoint_filter_fnr s *)*e< rJvariant pretrainedc Xtt||ft|tt dd|S)zCreate a ByobNet model. Args: variant: Model variant name. pretrained: Load pretrained weights. **kwargs: Additional model arguments. Returns: ByobNet model instance. T)flatten_sequential)rpretrained_filter_fn feature_cfg)r&r, model_cfgsrrf)rrrs rK_create_byobnetr s: * W%1D1     rJurlc 2|dddddttdddd |S) zCreate default configuration dictionary. Args: url: Model weight URL. **kwargs: Additional configuration options. Returns: Configuration dictionary. rrrrr?bilinear stem.convhead.fc apache-2.0 rr input_size pool_sizecrop_pct interpolationr rP first_convralicenserrrrs rK_cfgr s54}SYJ%.B!     rJc 2|dddddttdddd |S) zCreate RepVGG configuration dictionary. Args: url: Model weight URL. **kwargs: Additional configuration options. Returns: Configuration dictionary. rrrsrsrr?bicubiczstem.conv1.convrrrrrs rK_cfgrr s54}SY)%.B'y     rJzgernet_s.idstcv_in1kztimm/) hf_hub_idzgernet_m.idstcv_in1kzgernet_l.idstcv_in1krr)rrrzrepvgg_a0.rvgg_in1k)zstem.conv_kxk.convzstem.conv_1x1.convmit)rrrzrepvgg_a1.rvgg_in1kzrepvgg_a2.rvgg_in1kzrepvgg_b0.rvgg_in1kzrepvgg_b1.rvgg_in1kzrepvgg_b1g4.rvgg_in1kzrepvgg_b2.rvgg_in1kzrepvgg_b2g4.rvgg_in1kzrepvgg_b3.rvgg_in1kzrepvgg_b3g4.rvgg_in1kzrepvgg_d2se.rvgg_in1k)r@r)rlrl)rrrrrrzresnet51q.ra2_in1kzkhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/resnet51q_ra2-d47dcc76.pthz stem.conv1)rrZrZ)rrrrrtest_input_size test_crop_pctzresnet61q.ra2_in1kzkhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/resnet61q_ra2-6afc536c.pth)rrrrzresnext26ts.ra2_in1kzvhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/resnext26ts_256_ra2-8bbd9106.pthzseresnext26ts.ch_in1kzthttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/seresnext26ts_256-6f0d74a3.pthzgcresnext26ts.ch_in1kzthttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/gcresnext26ts_256-e414378b.pthzeca_resnext26ts.ch_in1kzvhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/eca_resnext26ts_256-5a1d030f.pthzbat_resnext26ts.ch_in1kzvhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/bat_resnext26ts_256-fa6fd595.pth)rrmin_input_sizezresnet32ts.ra2_in1kzqhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/resnet32ts_256-aacf5250.pthzresnet33ts.ra2_in1kzqhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/resnet33ts_256-e91b09a4.pthzgcresnet33ts.ra2_in1kzshttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/gcresnet33ts_256-0e0cd345.pthzseresnet33ts.ra2_in1kzshttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/seresnet33ts_256-f8ad44d9.pthzeca_resnet33ts.ra2_in1kzuhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/eca_resnet33ts_256-8f98face.pthzgcresnet50t.ra2_in1kzrhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/gcresnet50t_256-96374d1c.pthzgcresnext50ts.ch_in1kzthttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/gcresnext50ts_256-3e0f515e.pthzregnetz_b16.ra3_in1kzphttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/regnetz_b_raa-677d9606.pthr)rrrrrgGz?) rrrr rPrrrrrzregnetz_c16.ra3_in1kzuhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/regnetz_c_rab2_256-a54bf36a.pth)rrrr rPrrrzregnetz_d32.ra3_in1kzthttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/regnetz_d_rab_256-b8073a89.pthgffffff?)rrr rPrrzregnetz_d8.ra3_in1kzphttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/regnetz_d8_bh-afc03c55.pth)rrr rPrrrzregnetz_e8.ra3_in1kzphttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/regnetz_e8_bh-aace8e6e.pthzregnetz_b16_evos.untrained)rr rPrrrrzregnetz_c16_evos.ch_in1kzuhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-tpu-weights/regnetz_c16_evos_ch-d8311942.pth)rrrr rPrrzregnetz_d8_evos.ch_in1kzthttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-tpu-weights/regnetz_d8_evos_ch-2bc12646.pthzmobileone_s0.apple_in1kr)zstem.conv_kxk.0.convzstem.conv_scale.convzmobileone-license)rrrrzmobileone_s1.apple_in1krzmobileone_s2.apple_in1kzmobileone_s3.apple_in1kzmobileone_s4.apple_in1kzresnet50_clip.openaiz head.proj) rrr rPrYrrrarzresnet101_clip.openaizresnet50x4_clip.openai) rzresnet50x16_clip.openaii)rr.r.)rYrYzresnet50x64_clip.openai)rr)r1r1zresnet50_clip.cc12mzresnet50_clip.yfcc15mzresnet101_clip.yfcc15mzresnet50_clip_gap.openai)rrr rPrrrzresnet101_clip_gap.openai)rr)rr)r)r))rrrrr)zresnet50x4_clip_gap.openaizresnet50x16_clip_gap.openaizresnet50x64_clip_gap.openaizresnet50_clip_gap.cc12mzresnet50_clip_gap.yfcc15mzresnet101_clip_gap.yfcc15mzresnet50_mlp.untrainedztest_byobnet.r160_in1kc tdd|i|S)z GEResNet-Large (GENet-Large from official impl) `Neural Architecture Design for GPU-Efficient Networks` - https://arxiv.org/abs/2006.14090 r)r&rrrs rKr&r&   G* G GGrJc tdd|i|S)z GEResNet-Medium (GENet-Normal from official impl) `Neural Architecture Design for GPU-Efficient Networks` - https://arxiv.org/abs/2006.14090 r)r+rrs rKr+r+ rrJc tdd|i|S)z EResNet-Small (GENet-Small from official impl) `Neural Architecture Design for GPU-Efficient Networks` - https://arxiv.org/abs/2006.14090 r)r,rrs rKr,r, rrJc tdd|i|S)z^ RepVGG-A0 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r0rrs rKr0r0   H: H HHrJc tdd|i|S)z^ RepVGG-A1 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r4rrs rKr4r4 rrJc tdd|i|S)z^ RepVGG-A2 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r5rrs rKr5r5 rrJc tdd|i|S)z^ RepVGG-B0 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r7rrs rKr7r7 rrJc tdd|i|S)z^ RepVGG-B1 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r8rrs rKr8r8& rrJc tdd|i|S)z` RepVGG-B1g4 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r:rrs rKr:r:.   JZ J6 JJrJc tdd|i|S)z^ RepVGG-B2 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r;rrs rKr;r;6 rrJc tdd|i|S)z` RepVGG-B2g4 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r=rrs rKr=r=> rrJc tdd|i|S)z^ RepVGG-B3 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r>rrs rKr>r>F rrJc tdd|i|S)z` RepVGG-B3g4 `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r?rrs rKr?r?N rrJc tdd|i|S)z` RepVGG-D2se `Making VGG-style ConvNets Great Again` - https://arxiv.org/abs/2101.03697 r)r@rrs rKr@r@V rrJc tdd|i|S) r)rDrrs rKrDrD^   H: H HHrJc tdd|i|S)rr)rFrrs rKrFrFe rrJc tdd|i|S)rr)rGrrs rKrGrGl   JZ J6 JJrJc tdd|i|S)rr)rHrrs rKrHrHs   Lz LV LLrJc tdd|i|S)rr)rJrrs rKrJrJz rrJc tdd|i|S)rr)rKrrs rKrKrK   N Nv NNrJc tdd|i|S)rr)rMrrs rKrMrM rrJc tdd|i|S)rr)rOrrs rKrOrO   IJ I& IIrJc tdd|i|S)rr)rPrrs rKrPrP rrJc tdd|i|S)rr)rQrrs rKrQrQ   Kj KF KKrJc tdd|i|S)rr)rRrrs rKrRrR rrJc tdd|i|S)rr)rSrrs rKrSrS   M Mf MMrJc tdd|i|S)rr)rTrrs rKrTrT rrJc tdd|i|S)rr)rUrrs rKrUrU rrJc tdd|i|S)rr)rVrrs rKrVrV rrJc tdd|i|S)rr)r[rrs rKr[r[ rrJc tdd|i|S)rr)r\rrs rKr\r\ rrJc tdd|i|S)rr)r]rrs rKr]r] rrJc tdd|i|S)rr)r^rrs rKr^r^ rrJc tdd|i|S)rr)r_rrs rKr_r_   O* O OOrJc tdd|i|S)rr)r`rrs rKr`r` rrJc tdd|i|S)rr)rarrs rKrara rrJc tdd|i|S)rr)rbrrs rKrbrb rrJc tdd|i|S)rr)rcrrs rKrcrc rrJc tdd|i|S)rr)rdrrs rKrdrd rrJc tdd|i|S)rr)rerrs rKrere rrJc tdd|i|S)rr)rfrrs rKrfrf rrJc tdd|i|S)z0 OpenAI Modified ResNet-50 CLIP image tower r)rhrrs rKrhrh rrJc tdd|i|S)z1 OpenAI Modified ResNet-101 CLIP image tower r)rirrs rKriri" rrJc tdd|i|S)z2 OpenAI Modified ResNet-50x4 CLIP image tower r)rkrrs rKrkrk) rrJc tdd|i|S)z3 OpenAI Modified ResNet-50x16 CLIP image tower r)rmrrs rKrmrm0 rrJc tdd|i|S)z3 OpenAI Modified ResNet-50x64 CLIP image tower r)rorrs rKroro7 rrJc tdd|i|S)zP OpenAI Modified ResNet-50 CLIP image tower w/ avg pool (no attention pool) r)resnet50_clip_gaprrs rKrr> s  P: P PPrJc tdd|i|S)zQ OpenAI Modified ResNet-101 CLIP image tower w/ avg pool (no attention pool) r)resnet101_clip_gaprrs rKrrE s  QJ Q& QQrJc tdd|i|S)zR OpenAI Modified ResNet-50x4 CLIP image tower w/ avg pool (no attention pool) r)resnet50x4_clip_gaprrs rKrrL s  RZ R6 RRrJc tdd|i|S)zS OpenAI Modified ResNet-50x16 CLIP image tower w/ avg pool (no attention pool) r)resnet50x16_clip_gaprrs rKrrS   Sj SF SSrJc tdd|i|S)zS OpenAI Modified ResNet-50x64 CLIP image tower w/ avg pool (no attention pool) r)resnet50x64_clip_gaprrs rKr r Z rrJc tdd|i|S)rr)rrrrs rKrrrra rrJc tdd|i|S)z- Minimal test ResNet (BYOB based) model. r)rsrrs rKrsrsh rrJ))rr(rWr%r8r8r8r8r))r5rrlr%r rIr%)r%F)r^r^rNNN)r5)rrrr)r^F)zvisual.r)r^)rBr! dataclassesrrr functoolsrtypingrrr r r r r rrrtorch.nnrD timm.datarrrr timm.layersrrrrrrrrrrrrr r!r"r#r$_builderr& _featuresr' _manipulater(r) _registryr*r+__all__r.r-rGrHr}rrCrkrrrrrErrrrrr rrSrgrfrurvr0rrzr/rrrrrrr,rrrrrrrrr default_cfgsr&r+r,r0r4r5r7r8r:r;r=r>r?r@rDrFrGrHrJrKrMrOrPrQrRrSrTrUrVr[r\r]r^r_r`rarbrcrdrerfrhrirkrmrorrrrr rrrsrIrJrKrs%< 11TTT dd&++4< W 22 2( II IBU38_%s :Khkk3*+ 0%'!" " c?" %* "c?" "  ${  "P() S#X S$s)^$   ;  @k8K;P.P(QVZ[fVg"&8C=&C&C&$ 00 0/'BII/'d      S/   bii FGGTWbiiWtO OdL L^s="))s=lK=RYYK=\QBIIQh    +s+bii+,c299n-,N2==Nh!//// /  /  /dR !Yd38n!Y!YYd!YJ (7  D"$ #'$(.AE; E;E;E;S> E;  E;  E;C=E;!E;"(+E;P { d LbiiL^ ;"));3;T;VZ;2x  WSA! C WSA! C XcQ1 G XcQ1 D XcQ1 D   x WSA! C WSA! C XcQ1 G XcQ1 D XcQ1 D   x2 WR1r B WR1r B XcQ1 G XcQ1 D XcQ1 D   3xL}1HIMxV}@Wx`}1FGaxj 12kxt 01ux~ 0;xH 34IxR 3A>Sx\ 01]xf 0;gxp~2EF&Q7 qxD XcQ2$ G XcQ2$ G XdaB4 H XdaA# F   Exd VqC1sQUQW X XcQ2$ G XdaB4 H XdaA# F  T* exF XcQ2$ G XcQ2$ G XdaB4 H XdaB4 H   Gx^ XcQ2$ G XcQ2$ G XdaB4 H XdaB4 H   _xx XcQ2$ G XcQ2$ G XdaB4 H XdaB4 H   yxR XcQ2$ G XcQ2$ G XdaB4 H XdaB4 H   Sxl XcQ2$ G XcQ2$ G XdaB4 H XdaB4 H  A& mxL XcQ1 F XcQ1 F XdaA$ G XdaA$ G   Mxj XcQ1 F XcQ1 F XdaA$ G XdaA$ G   kxJ XcQ1 F XcQ1 F XdaA$ G XdaA$ G   Kxf XcQ1 F XcQ1 F XdaA$ G XdaA$ G   gxB XcQ1 F XcQ1 F XdaA$ G XdaA$ G   Cx` XcQ4 @ XcQ4 @ XdaD A XdaD A   axz XcQ2$ G XcQ2$ G XdaB4 H XdaB4 H   {xX  XbA" C XbA" C XsaB1 E XcQ2! D  $'DT:Y xx  XbA" C XbA" C XsaB1 E XcQ2! D  $'DT:y xX  XbA" C XcQ2! D XsaB1 E XcQ2! D  $'DT:Y xz  XbA! B XcQ1 C XsaA! D XcQ1 C  $'DT:{ x\  XbA! B XcQ1 C XsaA! D XcQ1 C  $'DT:] xB ! XbA" C XbA" C XsaB1 E XcQ2! D  4?P?e&DT?PEe&JT? HsKe&X$ yMV%S :Ye&b% y%S:ce&nE E%S:oe&vU C%S:we&~U C%S:e&Fu E%S :Ge&Nu E$ &Oe&Z5 %S:[e&b5 %S:ce&jU B%S:ke&rU B%S:se&zu D%S :{e&DE A%S:Ee&NU C%S:Oe&ZE ~_/ FTS`ps u[e&dE D_/}C Iee&nE C /DR_aoe&v5 ~ /DR_ortwe&~5 ~ /DR_orte&H!%_/ FTS`#bIe&N D_/} !6Oe&Xu C /DR_or tYe&btC#  ce&ntC#  oe&ztC#  {e&FtC#  Ge&RtC#  Se&bE/_-6 ce&pU.O-6 qe&~e.O-6 e&Lu.O-8  Me&Zu/_-8  [e&h5/_-6 ie&vU/_-6 we&De.O-6 Ee&V,/ F !We&b ,/ F "ce&n#(,/ F # $),/ H $ $),/ H $ %,/ F   "',/ F " #(,/ F #$ F$ 46A e&e P HGHHHGHHHGHHIWIIIWIIIWIIIWIIIWIIKwKKIWIIKwKKIWIIKwKKKwKKIWII IWII KwKK MMM MMM O7OO O7OO JgJJ JgJJ LLL LLL N'NN KwKK MMM KwKK KwKK KwKK JgJJ JgJJ PGPP PGPP O7OO LLL LLL LLL LLL LLL MMM N'NN O7OO PGPP PGPP QWQQ RgRR SwSS TTT TTT LLL LLLrJ