^j"ddlZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z mZmZmZddlmZmZmZmZmZmZmZmZmZddlmZddlmZdd lm Z dd l!m"Z"m#Z#d gZ$d Z%Gd de jLZ'Gdde jLZ(e jRddddfde*de*dee jLde+de+de jXf dZ-Gdde jLZ.Gdde jLZ/Gdde jLZ0Gd d!e jLZ1Gd"d#e jLZ2Gd$d%e jLZ3Gd&d'e jLZ4Gd(d)e jLZ5Gd*d+e jLZ6Gd,d e jLZ7dcd-Z8e#id.e8d/0d1e8d/0d2e8d/0d3e8d/0d4e8d/0d5e8d/0d6e8d/d78d9e8d/0d:e8d/0d;e8d/0d<e8d/0d=e8d/0d>e8d/0d?e8d/d78d@e8dAdBd7dCdDdEdFGdHe8dIdJd7dCdDdEdFGdKe8dLdMd7dCdDdEdFGe8d/dNdCdDdEdFOe8d/d7dCdDdEdFOe8d/d7dPeedQdRdFSe8d/d7dPeedQdRdFSdTZ9dUZ:dddVZ;e"dddWZe"dddZZ?e"ddd[Z@e"ddd\ZAe"ddd]ZBe"ddd^ZCe"ddd_ZDe"ddd`ZEe"dddaZFe"dddbZGy)eN)partial)ListOptionalTupleTypeUnion)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STDOPENAI_CLIP_MEANOPENAI_CLIP_STD) DropPathcalculate_drop_path_rates trunc_normal_ create_conv2d ConvNormAct SqueezeExciteuse_fused_attnClassifierHead LayerNorm2d)build_model_with_cfg)feature_take_indices)checkpoint_seq)register_modelgenerate_default_cfgsFastVitc&|sy||zdk(sJ||zS)Nrr) group_sizechannelss ^/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/models/fastvit.py num_groupsr""s( *$))):%%ceZdZdZddddddddej ddf deded ed ed ed ed edededededeejddffd Z de jde jfdZ dZdee je jffdZdeej$ej&fdee je jffdZxZS)MobileOneBlocka#MobileOne building block. This block has a multi-branched architecture at train-time and plain-CNN style architecture at inference time For more details, please refer to our paper: `An Improved One millisecond Mobile Backbone` - https://arxiv.org/pdf/2206.04040.pdf rrFTNin_chsout_chs kernel_sizestridedilationrinference_modeuse_seuse_actuse_scale_branchnum_conv_branches act_layerreturnc | |d}t|||_t|||_||_||_||_||_||_ | |_ |rt|fddi|ntj|_|r#t||f||||jdd||_nd|_||k(r|dk(rtj"d d|i|nd|_| dkDrttj&t)|jDcgc]<}t+|j|jf||j |jd d |>c}|_nd|_d|_|dkDrA| r?t+|j|jfd|j |jd d ||_| r | |_ytj|_ycc}w) aConstruct a MobileOneBlock module. Args: in_chs: Number of channels in the input. out_chs: Number of channels produced by the block. kernel_size: Size of the convolution kernel. stride: Stride size. dilation: Kernel dilation factor. group_size: Convolution group size. inference_mode: If True, instantiates model in inference mode. use_se: Whether to use SE-ReLU activations. use_act: Whether to use activation. Default: ``True`` use_scale_branch: Whether to use scale branch. Default: ``True`` num_conv_branches: Number of linear conv branches. devicedtype rd_divisorrTr(r)r*groupsbiasN num_featuresrFr(r)r8 apply_actr)super__init__r+r"r8r)r*r(r&r'r/rnnIdentityser reparam_conv BatchNorm2didentity ModuleListrangerconv_kxk conv_scaleact)selfr&r'r(r)r*rr+r,r-r.r/r0r4r5dd_ __class__s r!r>zMobileOneBlock.__init__5s@ / , V4    &  !2AG->D))--Cfbnn5 554- KK4;;6 ${{[[)T-=-=t?O?OP ----!==//  t{{+A!1y=$*:*:a*?AQAQUVAVV".^^F!..L ,,KMME;;D**CS &&( S[ ! !"aA .z4,"6"<<<rfTensorrTrbrrXr SequentialrCre __classcell__rMs@r!r%r%+sF#(  %)%&)+`=`=`= `=  `=  `=`=!`=`=`=#`= #`=BII`= !`=D&&%,,&2!#F!(% ell(B"C!(F*="--78*= u||U\\) **=r#r%cpeZdZdZ ddedededededeed ed eejd ed dffd Z de jd e jfdZ d e e je jffdZddZedej"dej$d e e je jffdZxZS)ReparamLargeKernelConvzBuilding Block of RepLKNet This class defines overparameterized large kernel conv block introduced in `RepLKNet `_ Reference: https://github.com/DingXiaoH/RepLKNet-pytorch Nr&r'r(r)r small_kernelr,r0r+r1c V| | d} t |||_t|||_||_||_||_||_| r#t||f||d|jdd| |_ nkd|_ t||f||j|jdd| |_ |7||ksJdt||f||j|jdd| |_ |rt|fd d i| ntj |_| ||_ytj |_y) a!Construct a ReparamLargeKernelConv module. Args: in_chs: Number of input channels. out_chs: Number of output channels. kernel_size: Kernel size of the large kernel conv branch. stride: Stride size. Default: 1 group_size: Group size. Default: 1 small_kernel: Kernel size of small kernel conv branch. act_layer: Activation module. Default: ``nn.GELU`` inference_mode: If True, instantiates model in inference mode. Default: ``False`` r3rTr7NFr;zDThe kernel size for re-param cannot be larger than the large kernel!rd_ratiog?)r=r>r)r"r8r&r'r(rrrBr large_conv small_convrr?r@rArI)rJr&r'r(r)rrr,r0r+r4r5rKrMs r!r>zReparamLargeKernelConv.__init__,s_4 /   V4   &(  - !({{ ! !D !%D )({{{{ DO' K/ZYZ/"-#!-;;;;# ##BH-=$="=R[[]"+"79;R[[]r#rNc|j|j|}n1|j|}|j||j|z}|j|}|j |}|SN)rBrrrArI)rJrNrRs r!rTzReparamLargeKernelConv.forwardvsh    (##A&C//!$C*DOOA..ggclhhsm r#c|j|jj|jj\}}t |dr|j|j j|j j\}}||z }|t jj||j|jz dzgdzz }||fS)zMethod to obtain re-parameterized kernel and bias. Reference: https://github.com/DingXiaoH/RepLKNet-pytorch Returns: Tuple of (kernel, bias) after fusing branches. rrd) _fuse_bnrrzr{r^rr?rgrhr(r)rJeq_keq_bsmall_ksmall_bs r!get_kernel_biasz&ReparamLargeKernelConv.get_kernel_biass]]4??#7#79K9KL d 4 &#}}T__-A-A4??CUCUV GW GOD BMM%%4++d.?.??AEFJ DTzr#c|j\}}t|j|j|j|j |j d|_||jj_ ||jj_ |jdt|dr|jdyy)a Following works like `RepVGG: Making VGG-style ConvNets Great Again` - https://arxiv.org/pdf/2101.03697.pdf. We re-parameterize multi-branched architecture used at training time to obtain a plain CNN-like structure for inference. Tr(r)r8r9rrN) rrr&r'r(r)r8rBrYrZr9r]r^)rJrrs r!rbz%ReparamLargeKernelConv.reparameterizes))+ d) KK LL((;;;;  )-  %&*# & 4 &   \ * 'r#rzr{c|j}|j}|j}|j}|j}|j}||zj }||z j dddd} || z|||z|z z fS)zMethod to fuse batchnorm layer with conv layer. Args: conv: Convolutional kernel weights. bn: Batchnorm 2d layer. Returns: Tuple of (kernel, bias) after fusing batchnorm. rxr)rYr|r}r9r~rr) rzr{r_r|r}rrr~rrs r!rzReparamLargeKernelConv._fuse_bns nn  wwffS &&( S[ ! !"aA .z4,"6"<<<rfrrTrrrb staticmethodConv2drCrrrs@r!rr#s-+/ -1#(HKHKHK HK  HK  HK#3-HKHK  *HK!HK HKT  %,, u||U\\'A!B +.=))== u||U\\) *==r#rFTr&r'r0r+r.r1c||d}tjtd||dd|||d|td||ddd|||d|td||dd|||d|S)a,Build convolutional stem with MobileOne blocks. Args: in_chs: Number of input channels. out_chs: Number of output channels. inference_mode: Flag to instantiate model in inference mode. Default: ``False`` Returns: nn.Sequential object with stem elements. r3rd)r&r'r(r)r0r+r.r)r&r'r(r)rr0r+r.r)r?rr%)r&r'r0r+r.r4r5rKs r!convolutional_stemrs&U +B == )-    )-    )-  -  r#c eZdZUdZej j eed< d de de dede de d df fd Z d ejd ejfd Z xZS) AttentionzMulti-headed Self Attention module. Source modified from: https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py fused_attnNdimhead_dimqkv_bias attn_drop proj_dropr1c||d}t |||zdk(sJd||_||z|_|dz|_t |_tj||dzfd|i||_ tj||_ tj||fi||_ tj||_ y)a}Build MHSA module that can handle 3D or 4D input tensors. Args: dim: Number of embedding dimensions. head_dim: Number of hidden dimensions per head. Default: ``32`` qkv_bias: Use bias or not. Default: ``False`` attn_drop: Dropout rate for attention tensor. proj_drop: Dropout rate for projection tensor. r3rz#dim should be divisible by head_dimgrr9N)r=r>r num_headsscalerrr?LinearqkvDropoutrprojr) rJrrrrrr4r5rKrMs r!r>zAttention.__init__s& / X~"I$II"  % (*99S#'??B?I.IIc3-"- I.r#rNcV|j\}}}}||z}|jdjdd}|j|j ||d|j |j jddddd}|jd\}} } |jrPtjjj|| | |jr|jj nd }nL||j"z}|| jddz} | j%d } |j| } | | z}|jddj |||}|j'|}|j)|}|jddj ||||}|S) Nrdrxrrrr) dropout_p)r)shapeflatten transposerrrrpermuteunbindrrfr?rgscaled_dot_product_attentiontrainingrprsoftmaxrr) rJrNBCHWNrqkvattns r!rTzAttention.forward suWW 1a E IIaL " "2r * HHQK WQ1dnndmm < WQ1a # **Q-1a ??##@@1a.2mm$..**AA DJJAq{{2r**D<>$'DqA KK1  % %aA . IIaL NN1  KKB  ' '1a 3r#) FrrNN)rrrrrfjitFinalr__annotations__rfloatr>rrTrrs@r!rrs  %% """/// /  /  / /@%,,r#rceZdZdZej dddddfdededededeejd e d e d e d dffd Z de jd e jfdZ xZS) PatchEmbedz$Convolutional patch embedding layer.FN patch_sizer)r& embed_dimr0 lkc_use_actr,r+r1c | | d} t |tjt d||||dd||r|nd|d | t d||ddd||d| |_y) a{Build patch embedding layer. Args: patch_size: Patch size for embedding computation. stride: Stride for convolutional embedding layer. in_chs: Number of channels of input tensor. embed_dim: Number of embedding dimensions. inference_mode: Flag to instantiate model in inference mode. Default: ``False`` r3rrN) r&r'r(r)rrr,r0r+F)r&r'r(r)r,r0r+r)r=r>r?rrr%r) rJrr)r&rr0rr,r+r4r5rKrMs r!r>zPatchEmbed.__init__Bs, / MM " !&'2)-     !#-    r#rNc(|j|}|Sr)rrJrNs r!rTzPatchEmbed.forwardss IIaLr#)rrrrr?rrrrrr>rfrrTrrs@r!rr?s.*, % #(/ / /  /  / BII / / / !/  / b%,,r#rc<eZdZ ddededeffd ZdZxZS) LayerScale2dr init_valuesinplacec t|||_tj|t j |dd||z|_y)Nrr3)r=r>rr? Parameterrfonesr)rJrrrr4r5rMs r!r>zLayerScale2d.__init__ys>  \\+ 31V[`0a"ab r#cn|jr|j|jS||jzSr)rmul_rrs r!rTzLayerScale2d.forwards(%)\\qvvdjj!Eq4::~Er#)h㈵>FNN) rrrrrrr>rTrrs@r!rrxs<"&! c c c cFr#rc eZdZdZ d dededeedeffd Zde jde jfd Z d d Z xZ S) RepMixerzReparameterizable token mixer. For more details, please refer to our paper: `FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization `_ rr(layer_scale_init_valuer+c ||d}t|||_||_||_|rXt j |j|jf|jd|jdz|jdd||_yd|_t|||fddddd ||_ t|||fddd ||_ |t||fi||_ yt j|_ y) aBuild RepMixer Module. Args: dim: Input feature map dimension. :math:`C_{in}` from an expected input of size :math:`(B, C_{in}, H, W)`. kernel_size: Kernel size for spatial mixing. Default: 3 layer_scale_init_value: Initial value for layer scale. Default: 1e-5 inference_mode: If True, instantiates model in inference mode. Default: ``False`` r3rrdTr(r)paddingr8r9NFr)rr-r.r/)rr-)r=r>rr(r+r?rrBr%normmixerr layer_scaler@) rJrr(rr+r4r5rKrMs r!r>zRepMixer.__init__s" / &,  "  !!,,((A-xx ! !D !%D & !&"#  DI(   DJ&1#/5K#Rr#R #%;;= r#rNr1c|j|j|}|S||j|j||j|z z}|Sr)rBrrrrs r!rTzRepMixer.forwardsV    (!!!$AD$$TZZ]TYYq\%ABBAr#ch|jry|jj|jjt |j t r|jj|j jjd|jjj|jjjz zz}tj|j j|jjj|jjjz z}n|jj|jjjz|jjjz }|jjj|jjjz }t|j |j |j"d|j d|_ ||jj_||jj_|j'D]\}}d|vr |j)|j+d|j+d|j+d y) ziReparameterize mixer and norm into a single convolutional layer for efficient inference. NrxrTrrBrrr)r+rrbrryrrrvr unsqueezerBrYrfsqueezer9rrr(rZr[r\r])rJwbr`ras r!rbzRepMixer.reparameterizes      !!#   " d&& 5 $$t'7'7'='='G'G'K ''..1G1G1N1NN(A d..445 '',,tyy/E/E/J/JJA  $$**))001))((//0   '',,tyy/E/E/J/JJA) HH HH((88  )*  %&'#//1 JD$% LLN  !   'r#)rrFNNr)rrrrrrrrr>rfrrTrbrrs@r!rrsi !6:#(919191%-UO 91 ! 91v%,,*(r#rc eZdZdZddej dddfdedeedeedeejde d df fd Z d ejd dfd Z d e jd e jfdZxZS)ConvMlpzConvolutional FFN Module.Nrr&hidden_channelsr'r0dropr1cj||d}t ||xs|}|xs|}t||fd|dd||_t j ||fddi||_||_t j ||fddi||_t j||_ |j|jy)a_Build convolutional FFN module. Args: in_chs: Number of input channels. hidden_channels: Number of channels after expansion. Default: None out_chs: Number of output channels. Default: None act_layer: Activation layer. Default: ``GELU`` drop: Dropout rate. Default: ``0.0``. r3F)r(r8r<r(rN) r=r>rrzr?rfc1rIfc2rrapply _init_weights) rJr&rr'r0rr4r5rKrMs r!r>zConvMlp.__init__s& / #V)3V        99V_J!JrJ;99_gK1KKJJt$  4%%&r#mct|tjrOt|jd|j +tj j|j dyyy)N{Gz?rr)ryr?rrrYr9init constant_rJrs r!rzConvMlp._init_weights'sJ a # !(( -vv!!!!&&!," $r#rNc|j|}|j|}|j|}|j|}|j |}|j|}|Sr)rzrrIrrrs r!rTzConvMlp.forward-sU IIaL HHQK HHQK IIaL HHQK IIaLr#)rrrrr?rrrrrrr>rrfrrTrrs@r!rrs# .2%))+#'#'&c]#'c] #' BII #'  #' #'J-ryy-T- %,,r#rc eZdZdZ d dedeedeeeeeffdeddf fd Z d e jde jfd Z d d Z xZS)RepConditionalPosEnca"Implementation of conditional positional encoding. For more details refer to paper: `Conditional Positional Encodings for Vision Transformers `_ In our implementation, we can reparameterize this module to eliminate a skip connection. Nrdim_out spatial_shaper+r1c ||d}t|t|trt |gdz}t|t sJdt |dt|dk(sJdt|d||_||_ |xs||_ ||_ |rQtj|j|jf|jd|ddz|jdd ||_y d |_tj|j|j|dt|ddzf|jdd ||_y ) atBuild reparameterizable conditional positional encoding Args: dim: Number of input channels. dim_out: Number of embedding dimensions. Default: 768 spatial_shape: Spatial shape of kernel for positional encoding. Default: (7, 7) inference_mode: Flag to instantiate block in inference mode. Default: ``False`` r3rdz/"spatial_shape" must by a sequence or int, get z instead.z+Length of "spatial_shape" should be 2, got rrTrN)r8r9)r=r>ryrtuplertypelenrrrr8r?rrBpos_enc) rJrrrr+r4r5rKrMs r!r>zRepConditionalPosEnc.__init__@sj" /  mS )!=/A"56M-/  &'y 2 /=!Q& }%&i 1 & +~#   "   !!..%a(A-{{ ! !D !%D 99 M!$)*  {{  DLr#rNcl|j|j|}|S|j||z}|Sr)rBrrs r!rTzRepConditionalPosEnc.forward{s>    (!!!$A Q!#Ar#c |j|jz}tj|j||jd|jdf|j j j|j j j}t|jD].}d||||z|jddz|jddzf<0|}||j j z}|j j}tj|j|j|jdt|jddz|jd|_||jj _||jj_|j#D]\}}d|vr |j%|j'dy) NrrrwrdTrrBr)rr8rfrrrrYr5r4rFr9r?rrrrBrZr[r\r]) rJrrrrvw_finalb_finalr`ras r!rbz#RepConditionalPosEnc.reparameterizesHH + {{""1%""1%   ,,%%++<<&&--  txx A  I ""1%*""1%*,  ! dll111,,##II HH LL****1-23;; )0  %&-#//1 JD$% LLN  #r#)NrrFNNr)rrrrrrrrrr>rfrrTrbrrs@r!r r 7s&*9?#(99c]9!eCHo!56 9 ! 9 9v%,,+$r#r ceZdZdZddej ddddddf ded ed ed eejd ed edede ffd Z dZ xZ S) RepMixerBlockzImplementation of Metaformer block with RepMixer as token mixer. For more details on Metaformer structure, please refer to: `MetaFormer Is Actually What You Need for Vision `_ r@rrFNrr( mlp_ratior0r drop_pathrr+c R| | d} t |t|f|||d| |_t d|t ||z||d| |_|t||fi| |_ntj|_|dkDrt||_ ytj|_ y)a,Build RepMixer Block. Args: dim: Number of embedding dimensions. kernel_size: Kernel size for repmixer. Default: 3 mlp_ratio: MLP expansion ratio. Default: 4.0 act_layer: Activation layer. Default: ``nn.GELU`` proj_drop: Dropout rate. Default: 0.0 drop_path: Drop path rate. Default: 0.0 layer_scale_init_value: Layer scale value at initialization. Default: 1e-5 inference_mode: Flag to instantiate block in inference mode. Default: ``False`` r3)r(rr+r&rr0rNrr) r=r>r token_mixerrrmlprrr?r@r r) rJrr(rr0rrrr+r4r5rKrMs r!r>zRepMixerBlock.__init__s2 / #  ##9)     i0     " -+C1GN2ND !{{}D 09C),R[[]r#c|j|}||j|j|j|z}|Sr)r!rrr"rs r!rTzRepMixerBlock.forwards=   Q  t// <= =r#)rrrrr?rrrrrrr>rTrrs@r!rrs !")+"",0#(/S/S/S /S BII /S  /S/S%*/S!/Sbr#rceZdZdZdej ej dddddfdedede ejd e ejd ed ed effd Z dZ xZ S)AttentionBlockzImplementation of metaformer block with MHSA as token mixer. For more details on Metaformer structure, please refer to: `MetaFormer Is Actually What You Need for Vision `_ rrrNrrr0 norm_layerrrrc || d} t |||fi| |_tdd|i| |_|t ||fi| |_ntj|_|dkDr t|ntj|_ td|t||z||d| |_ |t ||fi| |_ntj|_|dkDrt||_ytj|_y)aBuild Attention Block. Args: dim: Number of embedding dimensions. mlp_ratio: MLP expansion ratio. Default: 4.0 act_layer: Activation layer. Default: ``nn.GELU`` norm_layer: Normalization layer. Default: ``nn.BatchNorm2d`` proj_drop: Dropout rate. Default: 0.0 drop_path: Drop path rate. Default: 0.0 layer_scale_init_value: Layer scale value at initialization. Default: 1e-5 r3rNrr r)r=r>rrr!r layer_scale_1r?r@r drop_path1rrr" layer_scale_2 drop_path2) rJrrr0r&rrrr4r5rKrMs r!r>zAttentionBlock.__init__s. / s)b) $333 ! -!-c3I!PR!PD !#D 1:S(9-bkkm i0     " -!-c3I!PR!PD !#D 1:S(9-bkkmr#c ||j|j|j|j|z}||j |j |j |z}|Sr)r)r(r!rr+r*r"rs r!rTzAttentionBlock.forward$s^  2 243C3CDIIaL3Q RS S  2 2488A; ?@ @r#)rrrrr?rrCrrrrr>rTrrs@r!r%r%s #)+*,.."",0-T-T-TBII -T RYY -T  -T-T%*-T^r#r%c%eZdZdddddddejej ddd ddddfd ed ed ed ededededede ejdede de ejde ejde de ee e fde e dedef$fd ZdZxZS) FastVitStageTFrrdNrrrrrrdepthtoken_mixer_type downsample se_downsampledown_patch_size down_stride pos_emb_layerr(rr0r&proj_drop_ratedrop_path_raterrr+c<t|||d}d|_|rtd |||||| ||d||_n ||k(sJt j |_| | |fd|i||_nt j |_g}t|D]r}|dk(r'|jt|f| | | |||||d|/|dk(r&|jt|f| | | ||||d |Ztd j|t j||_y) aQFastViT stage. Args: dim: Number of embedding dimensions. depth: Number of blocks in stage token_mixer_type: Token mixer type. kernel_size: Kernel size for repmixer. mlp_ratio: MLP expansion ratio. act_layer: Activation layer. norm_layer: Normalization layer. proj_drop_rate: Dropout rate. drop_path_rate: Drop path rate. layer_scale_init_value: Layer scale value at initialization. inference_mode: Flag to instantiate block in inference mode. r3F)rr)r&rr,r0rr+Nr+repmixer)r(rr0rrrr+ attention)rr0r&rrrz"Token mixer type: {} not supportedr)r=r>grad_checkpointingrr1r?r@pos_embrFappendrr% ValueErrorformatrblocks)rJrrr/r0r1r2r3r4r5r(rr0r&r6r7rrr+r4r5rKr@ block_idxrMs r!r>zFastVitStage.__init__+ssL  /"' ( *"!$#'-  DO'> !> kkmDO  $(VVSUVDL;;=DLu I:- m  +'',,Y7+A#1   "[0 n ''),,Y7+A   !8??@PQ3 8mmV, r#c|j|}|j|}|jr6tjj st |j|}|S|j|}|Sr)r1r<r;rfr is_scriptingrr@rs r!rTzFastVitStage.forwards` OOA  LLO  " "599+A+A+Ct{{A.A AAr#)rrrr?rrCrstrrrrrrrrr>rTrrs@r!r.r.*s8 $"'#$ 15 ")+*,..$'8;6: %#(+\-\-\- \- " \-  \- \-!\-\-$BII.\-\-\-BII\-RYY\-"\- "$u+u"45!\-"%-UO#\-$%\-&!'\-|r#r.c3deZdZUejj eed< ddddddddd d d d d d d ddddddejejdddfde de e dfde e dfde e dfde edfde edfde edfde de de eej dfde d e d!ed"ed#ed$ed%ed&eded'ed(e d)eej d*eej d+ed,df2fd- Zd.ej d,dfd/Zejj(d0Zejj(dCd1Zejj(dDd2Zejj(d,ej fd3ZdEde d(ee fd4Z dFd5ej4d6eee ee fd7ed8ed9e d:ed,eeej4e ej4eej4fffd;Z dGd6ee ee fdZd5ej4d,ej4fd?ZdCd5ej4d@efdAZ d5ej4d,ej4fdBZ!xZ"S)Hr fork_featrrdrdrdr9r9r9r9@rrrr)FTTT)FFFF)NNNNrrdrrFTg@avgNin_chanslayers. token_mixers embed_dims mlp_ratios downsamplesse_downsamplesrepmixer_kernel_size num_classespos_embsr3r4 drop_rater6r7rrstem_use_scale_branch cls_ratio global_poolr&r0r+r1c t(|||d}|rdn| |_||_||_g|_t ||d||fd|i||_|d}d}t||d}g}tt|D]} || xs||| k7}!td$id|d|| d || d |!d || d | d | d| | d|| d|d|| d|d|d|d|| d|d|d||}"|j|"|| }|!r|dz}|xj t|d|zd| gz c_tj||_t|j |_|x|_|_|jrgd|_t+|j(D]c\}#}$|#dk(r6t,j.j1ddr tj2}%n |||#fi|}%d|$}&|j5|&|%enQt7|d |zx|_x|_}'t9d$|d |'d!dd|d|dd" ||_t=|'| f|| d#||_|jA|jBy)%Nr3rr.rT) stagewiserrr/r1r2r3r4r5r0r(rr0r&r6r7rrr+rdrstages.)num_chs reductionmodulerrrdr FORK_LAST3rrxr) r&r'r(r)rr+r,r0r/) pool_typer\r)"r=r>rZrFr_ feature_inforstemrrFrr.r=dictr?rstages num_stagesr:head_hidden_size out_indices enumerateosenvirongetr@ add_modulerr% final_convrheadrr))rJrRrSrTrUrVrWrXrYrZr[r3r4r\r6r7rrr]rFr^r_r&r0r+r4r5rKprev_dimrdprrlrr1stagei_embi_layerlayer layer_namefinal_featuresrMs) r!r>zFastVit.__init__sH:  / )1{"&'  qM    3     a='$Os6{# eA$QD8z!}+DJ "1 Qi&  -Q/  !0 ('qk".a1%Q-$& . #1v (>!"(#$ .'E* MM% !!}H    $x1u9W^_`^aUb"c!d d 7 e8mmV, dkk*4<<D1 >> ,D "+D,<,<"= 3wA:"**..t"DKKME&z%'8?B?E#G9-  E2 3JMZXZ^^gMgIh hD  h 5, !"~&-#"#  DO'&#   DI 4%%&r#rc t|tjrjt|jdt|tjr8|j +tj j|j dyyyy)zInit. for classificationrrNr)ryr?rrrYr9rr r s r!rzFastVit._init_weightssZ a # !(( -!RYY'AFF,>!!!&&!,-?' $r#ctSr)setrJs r!no_weight_decayzFastVit.no_weight_decays u r#c2td|rdSgdS)Nz^stemz^stages\.(\d+)))z^stages\.(\d+).downsampler)z^stages\.(\d+).pos_embr)z^stages\.(\d+)\.\w+\.(\d+)N)rjr@)rk)rJcoarses r! group_matcherzFastVit.group_matcher!s'(.$  5  r#c4|jD] }||_ yr)rlr;)rJenabless r!set_grad_checkpointingzFastVit.set_grad_checkpointing,s *A#)A  *r#c.|jjSr)rvfcrs r!get_classifierzFastVit.get_classifier1syy||r#cJ||_|jj||yr)rZrvreset)rJrZr_s r!reset_classifierzFastVit.reset_classifier5s&  [1r#rNindicesr stop_early output_fmtintermediates_onlyc|dvsJdg}tt|j|\}} |j|}|jdz } t j js|s |j} n|jd| dz} d} t| D]#\} } | |}| |vs|j|%|r|S| | k(r|j|}||fS)a Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to compatible intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )NCHWzOutput shape must be NCHW.rNr) rrrlrjrmrfrrCrpr=ru)rJrNrrrrr intermediates take_indices max_indexlast_idxrlfeat_idxrys r!forward_intermediateszFastVit.forward_intermediates9s*Y&D(DD& "6s4;;7G"Q i IIaL??Q& 99 ! ! #:[[F[[)a-0F(0 (OHeaA<'$$Q' (  x "A-r# prune_norm prune_headctt|j|\}}|jd|dz|_|r|jdd|S)z@ Prune layers not required for specified intermediates. Nrr)rrrlr)rJrrrrrs r!prune_intermediate_layersz!FastVit.prune_intermediate_layersgsM#7s4;;7G"Q ikk.9q=1   ! !!R (r#c<|j|}g}t|jD]Q\}}||}|js||jvs*t |d|}||}|j |S|jr|S|j|}|S)Nr)rjrprlrFrogetattrr=ru)rJrNoutsidxblockr&x_outs r!forward_featureszFastVit.forward_featuresus IIaL#DKK0 'JCaA~~$***!(cU|!>K OOA r# pre_logitscN|r|j|dS|j|S)NT)r)rv)rJrNrs r! forward_headzFastVit.forward_heads$0:tyyty,L ! Lr#cf|j|}|jr|S|j|}|Sr)rrFrrs r!rTzFastVit.forwards3  ! !! $ >>H   a r#F)Tr)NFFrF)rFT)#rrrrfrrrrr?rCrrrrDrrrrr>rignorerrrrrrrrrrrrrTrrs@r!rrsyyt$$ &2,\*=,4,E/K()#8C#$ "$'$',0 %*.#"$*,..)+#(7z'z'#s(Oz' S/ z' c3h z' eSj) z'tSy)z'"$),z'#&z'z'HRYY/45z'!z'z'z'"z' "!z'"%*#z'$%z'&$('z'()z'*+z',-z'.RYY/z'0BII1z'2!3z'8 9z'x-ryy-T- YY YY   YY** YY 2C2hsm28<$$',, ||, eCcN34,  ,  ,  , !%,  tELL!5tELL7I)I#JJ K, `./$# 3S >*   %,,5<<"MellMM%,,r#c 2|dddddtdtddd |S) NrP)rrMrM)rg?bicubiczfastvit-license)stem.0.conv_kxk.0.convzstem.0.conv_scale.convzhead.fc) urlrZ input_size pool_sizecrop_pct interpolationmeanlicenser first_conv classifier)r r )rkwargss r!_cfgrs7#"%$#J    r#zfastvit_t8.apple_in1kztimm/) hf_hub_idzfastvit_t12.apple_in1kzfastvit_s12.apple_in1kzfastvit_sa12.apple_in1kzfastvit_sa24.apple_in1kzfastvit_sa36.apple_in1kzfastvit_ma36.apple_in1kgffffff?)rrzfastvit_t8.apple_dist_in1kzfastvit_t12.apple_dist_in1kzfastvit_s12.apple_dist_in1kzfastvit_sa12.apple_dist_in1kzfastvit_sa24.apple_dist_in1kzfastvit_sa36.apple_dist_in1kzfastvit_ma36.apple_dist_in1kzfastvit_mci0.apple_mclipzapple/mobileclip_s0_timmzXhttps://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_s0.ptrN)rrr)?rrz apple-amlr)rrrrZrrrzfastvit_mci1.apple_mclipzapple/mobileclip_s1_timmzXhttps://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_s1.ptzfastvit_mci2.apple_mclipzapple/mobileclip_s2_timmzXhttps://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_s2.ptr)rrrZrrr)rrr)rrrZrrrrr)z!fastvit_mci0.apple_mclip2_dfndr2bz!fastvit_mci2.apple_mclip2_dfndr2bz!fastvit_mci3.apple_mclip2_dfndr2bz!fastvit_mci4.apple_mclip2_dfndr2bcd|vr|Sd|vrF|jDcic]*\}}|jds|jdd|,c}}S|jd|}d|vrd}nd}d d l}d d l}g}|jD]C\}}|j d |}|s|jt|jd Ettt|}i} |jD]d\}}|r||vr|j|d}|jd d}|jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jdd |}|jd!r|jd!d"}|jd#d$}|jd%r|d&k(rt|j d'rrt#|j j$t&j(rD|jd&d(}|j*}t-j.|j0d | d)<n|jd%d*}|j d+|}d,\} } |r,t|jd-} |j3|| } | _d.| } d/| } | d0z|vr|j| d0z| d1z}n5| d2z|vr|j| d2z| d3z}n|j| | d4z}|| |<g| Scc}}w)5z$ Remap original checkpoints -> timm zstem.0.conv_kxk.0.conv.weightz1module.visual.trunk.stem.0.conv_kxk.0.conv.weightzmodule.visual.trunkzmodule.visual.trunk.r state_dictz8image_encoder.model.patch_embed.0.rbr_conv.0.conv.weightzimage_encoder.model.rNz^(.*?)network\.(\d+)\.proj.*rd patch_embedrjrbr_convrG rbr_scalerHrbr_skiprDconv_expru lkb_originrconvffnr"z se.reducezse.fc1z se.expandzse.fc2zlayer_scale_([0-9])zlayer_scale_\1.gammarzlayer_scale.gamma dist_head head_distzhead.z head.projrzhead.fc.weightz head.fc.biaszhead.fc.z^network\.(\d+))NNrznetwork.rbz.projz.downsample.projz.pez.pos_emb.pos_encz.blocks)items startswithreplacersrebisectmatchr=rgrouplistsortedrsubendswithr^rvryrr?rTrfrr bisect_right)rmodelrrprefixrr stage_endsrout_dict stage_idxnet_idx net_prefix stage_prefixs r!checkpoint_filter_fnrsT&*4:jHEOEUEUEWTQ[\[g[gh}[~ 0"5q8 j9JAZO' J  "318!<    c%++a.1 23fS_-.JH  "-1 Q &"%A IImV , IIj* - IIk< 0 IIj* - IIj, / IIlL 1 IIi ' IIk8 , IIk8 , FF)+BA F ::m $ -)<=A IIk; / << KGEJJ$=*UZZ]]\^\e\eBfIIk+;<CC+0;;qwwqz+B(IIgz2+Q/' 7 %++a.)G++J@I  #G9-J$YK0LG#q(IIj72LCU4UVe#q(IIj50,AS2STIIj,*BC [-\ OG@s M%M%c r|jdd}tt||ftt d|d|}|S)NrorfT)flatten_sequentialro)pretrained_filter_fn feature_cfg)poprrrrk)variant pretrainedrrors r!_create_fastvitrMsJ**]L9K  2DkJ     E Lr#c Ntdddd}tdd|it|fi|S)z%Instantiate FastViT-T8 model variant.)rdrdrrd)0`rrrrrIrSrUrVrTr) fastvit_t8rkrrr model_argss r!rrZs:%E J  ]J ]$zB\U[B\ ]]r#c Ntdddd}tdd|it|fi|S)z&Instantiate FastViT-T12 model variant.rGrJrrIrr) fastvit_t12rrs r!rrf:&E J  ^Z ^4 C]V\C] ^^r#c Ntdddd}tdd|it|fi|S)z&Instantiate FastViT-S12 model variant.rGrJrOrIrr) fastvit_s12rrs r!rrrrr#c vtddddddttdfd}td d |it|fi|S) z'Instantiate FastViT-SA12 model variant.rGrJrONrrr9r9r9r:rSrUrVr[rTr) fastvit_sa12rkrr rrs r!rr~sO&dG,@PV$WXF J  _j _DD^W]D^ __r#c vtddddddttdfd}td d |it|fi|S) z'Instantiate FastViT-SA24 model variant.)rr rrJrONrrrrr) fastvit_sa24rrs r!rrO&dG,@PV$WXF J  _j _DD^W]D^ __r#c vtddddddttdfd}td d |it|fi|S) z'Instantiate FastViT-SA36 model variant.rHrHrHrJrONrrrrr) fastvit_sa36rrs r!rrrr#c vtddddddttdfd}td d |it|fi|S) z'Instantiate FastViT-MA36 model variant.r)Li0i`rONrrrrr) fastvit_ma36rrs r!r r rr#c ztdddddddttdfdd }td d |it|fi|S) zInstantiate MCi0 model variant.)rdrH rdrJrFFTTNrrrTrSrUrVrXr[rTrr) fastvit_mci0rrs r!rrsU&1dG,@PV$WXFJ  _j _DD^W]D^ __r#c ztdddddddttdfdd }td d |it|fi|S) zInstantiate MCi1 model variant.)rrrrJrrNrrrTrr) fastvit_mci1rrs r!rrU&1dG,@PV$WXFJ  _j _DD^W]D^ __r#c ztdddddddttdfdd }td d |it|fi|S) zInstantiate MCi2 model variant.)rrr)Pi@irrNrrrTrr) fastvit_mci2rrs r!rrrr#c tddddddddttdttdfd d ttd d  }t dd|it|fi|}|S)zInstantiate L model variant.)rdrrrrd)rrrrirrrrrFFFFFFTTTTNrrr9r9r9r:r:Trr~F rSrUrVrXrWr[rTrr&r]r) fastvit_mci3rkrr rrrrrrs r!r"r"s{ ,":3    ( ? ( ?  T;D1#!J$  `z `T*E_X^E_ `E Lr#c tddddddddttdttdfd d ttd d  }t dd|it|fi|}|S)zInstantiate XL model variant.)rdrrrr)rLrMrNiirrrNrrrTrr Fr!r) fastvit_mci4r#r$s r!r&r&s{ .":3    ( ? ( ?  T;D1#!J&  `z `T*E_X^E_ `E Lr#)rr)Hrq functoolsrtypingrrrrrrftorch.nnr? timm.datar r r r timm.layersr rrrrrrrr_builderr _featuresr _manipulater _registryrr__all__r"rr%rrrrrrrrrrrr rr%r.rr default_cfgsrrrrrrrrr rrrr"r&rr#r!r2s 55 dd   ++'< +&u=RYYu=p[=RYY[=B&(WW$!%444 ?4 4  4]]4nD DN66rF299F"s(ryys(l5bii5pv$299v$r;BII;|9RYY9xf299fR{bii{|"%\&T\&d\& d \&t \&t \&t \&t \&&!$#'\&*"4$+\&0"4$1\&4#D%5\&8#D%9\&<#D%=\&B#D%C\&L, f |\ !M\&Z, f |\ ![\&h, f |\ !i\&x*. | **. | **. ?+**. ?+*g\&\ ~IX ^^____ ` ` ` ` ` ` ` ` ` ` ` ` ` `0r#