^jՃ dZddlZddlmZddlmZmZmZmZm Z m Z m Z ddl Z ddl mZddlmZmZddlmZmZmZmZmZmZmZddlmZdd lmZdd lmZm Z m!Z!dd l"m#Z#m$Z$m%Z%d d gZ&Gdd ejNZ(GddejNZ)GddejNZ*GddejNZ+GddejNZ,Gdd ejNZ-didejNde.de/de0ddf dZ1dZ2djde-fdZ3dkdee.effd Z4e#id!e4d"e4d#e4d$e4d%d&'d(e4d%d)d*+d,e4d-e4d%d.'d/e4d%d0d*+d1e4d%d2d3d4d5d6d78d9e4d%d:d3d4d5d6;d<e4ee=d>e4d%d?ee@dAe4d%dBee@dCe4d%dDee@dEe4d%dFee@dGe4d%dHee@dIe4d%dJee@e4d%dKee@e4d%dLee@e4d%dMee@e4d%dNee@e4d%dOee@e4d%dPee@e4e4d%dQ'e4dR Z5e$djde-fdSZ6e$djde-fdTZ7e$djde-fdUZ8e$djde-fdVZ9e$djde-fdWZ:e$djde-fdXZ;e$djde-fdYZe$djde-fd\Z?e$djde-fd]Z@e$djde-fd^ZAe$djde-fd_ZBe$djde-fd`ZCe$djde-fdaZDe%eEd$d-d9d1dIdbdcdddedfdgdh y)luj MLP-Mixer, ResMLP, and gMLP in PyTorch This impl originally based on MLP-Mixer paper. Official JAX impl: https://github.com/google-research/vision_transformer/blob/linen/vit_jax/models_mixer.py Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 @article{tolstikhin2021, title={MLP-Mixer: An all-MLP Architecture for Vision}, author={Tolstikhin, Ilya and Houlsby, Neil and Kolesnikov, Alexander and Beyer, Lucas and Zhai, Xiaohua and Unterthiner, Thomas and Yung, Jessica and Keysers, Daniel and Uszkoreit, Jakob and Lucic, Mario and Dosovitskiy, Alexey}, journal={arXiv preprint arXiv:2105.01601}, year={2021} } Also supporting ResMlp, and a preliminary (not verified) implementations of gMLP Code: https://github.com/facebookresearch/deit Paper: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 @misc{touvron2021resmlp, title={ResMLP: Feedforward networks for image classification with data-efficient training}, author={Hugo Touvron and Piotr Bojanowski and Mathilde Caron and Matthieu Cord and Alaaeldin El-Nouby and Edouard Grave and Armand Joulin and Gabriel Synnaeve and Jakob Verbeek and Hervé Jégou}, year={2021}, eprint={2105.03404}, } Paper: `Pay Attention to MLPs` - https://arxiv.org/abs/2105.08050 @misc{liu2021pay, title={Pay Attention to MLPs}, author={Hanxiao Liu and Zihang Dai and David R. So and Quoc V. Le}, year={2021}, eprint={2105.08050}, } A thank you to paper authors for releasing code and weights. Hacked together by / Copyright 2021 Ross Wightman N)partial)AnyDictListOptionalTypeUnionTuple)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpGluMlpGatedMlpDropPath lecun_normal_ to_2tuple)build_model_with_cfg)feature_take_indices) named_apply checkpointcheckpoint_seq)generate_default_cfgsregister_modelregister_model_deprecations MixerBlockMlpMixerc,eZdZdZdeeejdejddddfde de d e e e e e ffd e ejd e ejd e ejd e de ddffd Zdej"dej"fdZxZS)rzResidual Block w/ token mixing and channel MLPs. Based on: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 ?@ư>epsNdimseq_len mlp_ratio mlp_layer norm_layer act_layerdrop drop_pathreturnc X| | d} t|t|D cgc]} t| |zc} \} }||fi| |_||| f||d| |_|dkDr t |ntj|_ ||fi| |_ |||f||d| |_ ycc} w)aInitialize MixerBlock. Args: dim: Dimension of input features. seq_len: Sequence length. mlp_ratio: Expansion ratios for token mixing and channel MLPs. mlp_layer: MLP layer class. norm_layer: Normalization layer. act_layer: Activation layer. drop: Dropout rate. drop_path: Drop path rate. devicedtyper,r-r&N) super__init__rintnorm1 mlp_tokensrnnIdentityr.norm2 mlp_channels)selfr'r(r)r*r+r,r-r.r2r3ddx tokens_dim channels_dim __class__s `/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/models/mlp_mixer.pyr6zMixerBlock.__init__@s2 / :CI:N#OQCCL#O L*r* #GZ^9SW^[]^09B),BKKM*r* %c<^9SW^[]^ $PsB'r@c ||j|j|j|jddjddz}||j|j |j |z}|S Forward pass.r)r.r9r8 transposer=r<r>r@s rDforwardzMixerBlock.forwardbsm ttzz!}/F/Fq!/LMWWXY[\]^ ^ t00A?@ @)__name__ __module__ __qualname____doc__rrr: LayerNormGELUr7r floatr rModuler6torchTensorrK __classcell__rCs@rDrr;sr'r2r3r?rCs rDr6zAffine.__init__lsb  / \\%**aC["?B"?@ LLaC[!?B!?@ rLr@cXtj|j|j|S)zApply affine transformation.)rUaddcmulr`r^rJs rDrKzAffine.forwardws}}TYY A66rLNN) rMrNrOrPr7r6rUrVrKrWrXs@rDrZrZis7& AC AT A77%,,7rLrZceZdZdZdeeejdddddf dedede d e ejd e ejd e ejd e d e de ddffd Z de jde jfdZxZS)ResBlockzResidual MLP block w/ LayerScale and Affine 'norm'. Based on: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 g-C6?r&Nr'r(r)r*r+r, init_valuesr-r.r/c | | d} t|t||z} ||fi| |_t j ||fi| |_| dkDr t| nt j|_ ||fi| |_ ||| f||d| |_ t j|tj|fi| z|_t j|tj|fi| z|_y)aInitialize ResBlock. Args: dim: Dimension of input features. seq_len: Sequence length. mlp_ratio: Channel MLP expansion ratio. mlp_layer: MLP layer class. norm_layer: Normalization layer. act_layer: Activation layer. init_values: Initial values for layer scale. drop: Dropout rate. drop_path: Drop path rate. r1r&r4N)r5r6r7r8r:Linear linear_tokensrr;r.r<r=r\rUr]ls1ls2)r>r'r(r)r*r+r,rgr-r.r2r3r? channel_dimrCs rDr6zResBlock.__init__s6 / # /* *r* YYw>2>09B),BKKM*r* %c;])RV]Z\]<< ejj.C.C CD<< ejj.C.C CDrLr@c B||j|j|j|j|j ddj ddzz}||j|j |j |j|zz}|SrF)r.rkrjr8rIrlr=r<rJs rDrKzResBlock.forwards txx$*<*r'r(r+r2r3r?gate_dimrCs rDr6zSpatialGatingUnit.__init__sK / !8x.2. IIgw5"5 rLctjj|jjdtjj |jj y)z'Initialize weights for projection gate.r#stdN)r:initnormal_rsweightones_biasr>s rD init_weightszSpatialGatingUnit.init_weightss:  ((d3  diinn%rLr@c|jdd\}}|j|}|j|jdd}||jddzS)zApply spatial gating.rHr')chunkrrrsrI)r>r@uvs rDrKzSpatialGatingUnit.forwardsTwwqbw!1 IIaL IIakk"b) *1;;r2&&&rL)r/N)rMrNrOrPr:rQr7rrTr6r~rUrVrKrWrXs@rDrprpsj+-,, 666RYY 6 6*& ''%,,'rLrpceZdZdZdeeejdejddddfde de d e d e ejd e ejd e ejd e de ddffd Z dejdejfdZxZS)SpatialGatingBlockzpResidual Block w/ Spatial Gating. Based on: `Pay Attention to MLPs` - https://arxiv.org/abs/2105.08050 rfr#r$r&Nr'r(r)r*r+r,r-r.r/c  | | d} t|t||z} ||fi| |_t t fd|i| } ||| f|| |d| |_|dkDrt||_ ytj|_ y)aInitialize SpatialGatingBlock. Args: dim: Dimension of input features. seq_len: Sequence length. mlp_ratio: Channel MLP expansion ratio. mlp_layer: MLP layer class. norm_layer: Normalization layer. act_layer: Activation layer. drop: Dropout rate. drop_path: Drop path rate. r1r()r, gate_layerr-r&N) r5r6r7rrrrpr=rr:r;r.)r>r'r(r)r*r+r,r-r.r2r3r?rmsgurCs rDr6zSpatialGatingBlock.__init__s2 / # /* s)b) '??B?%        1:B),BKKMrLr@cj||j|j|j|z}|SrG)r.r=rrrJs rDrKzSpatialGatingBlock.forwards- t001>? ?rL)rMrNrOrPrrr:rQrRr7rSrrTr6rUrVrKrWrXs@rDrrs !)1*1",,D*I)+!&R&R&R &R BII &R RYY &RBII&R&R&R &RP%,,rLrc%&eZdZdZdddddddeeeejd ejd d d d d d ddfde de de de de de de e e e e ffdeejdeejdeejdeejde de de dededed df$fd! Zej(j*d7ded dfd"Zej(j*d7d#ed eeeffd$Zej(j*d8d%ed dfd&Zej(j*d ejfd'Zd9de deed dfd(Z d:d)ej<d*ee e ee fd+ed,ed-ed.ed e eej<e ej<eej<fffd/Z d;d*e e ee fd0ed1ed ee fd2Z!d)ej<d ej<fd3Z"d7d)ej<d4ed ej<fd5Z#d)ej<d ej<fd6Z$xZ%S)rrrrrrr)rr*r+r,rrrrrrr2r3r?r_irCs rDr6zMlpMixer.__init__sV  /&  &ENNND1DN"' !%.zD     /6dii.NDII((*T^ mm:& &(  %% $%##(   &( ) Y^^hXikSTD'! yI Nky/B/ I.DORSOBIIi)9)9@R@Y[YdYdYf  t$' &(ks .,F 0Fc|r tj|j nd}tt t ||y)zrInitialize model weights. Args: nlhb: Use negative log bias initialization for head. r&) head_bias)rN)mathlogrrr _init_weights)r>rrs rDr~zMlpMixer.init_weightsas048TXXd..//R GMY?MrLcoarsec tdddgS)zCreate regex patterns for parameter grouping. Args: coarse: Use coarse grouping. Returns: Dictionary mapping group names to regex patterns. z^stem)z^blocks\.(\d+)N)z^norm)i)rr)r)r>rs rD group_matcherzMlpMixer.group_matcherks-/CD  rLenablec||_y)zEnable or disable gradient checkpointing. Args: enable: Whether to enable gradient checkpointing. N)r)r>rs rDset_grad_checkpointingzMlpMixer.set_grad_checkpointingzs #)rLc|jS)zGet the classifier module.)rr}s rDget_classifierzMlpMixer.get_classifiersyyrLcn||_| |dvsJ||_|jjjt |jdr |jjj nd}}|dkDr)tj|j||||_ytj|_y)zReset the classifier head. Args: num_classes: Number of classes for new classifier. global_pool: Global pooling type. N)rrzrcrr1) rrrrzr2rr3r:rirr;)r>rrr2r3s rDreset_classifierzMlpMixer.reset_classifiers'  "+- --*D  ((//7SWS\S\^fKg1A1A1G1GmyZehiZiBIIdnnk&PUV oqozozo| rLr@indicesrr stop_early output_fmtintermediates_onlyc $|dvsJd|dk(}g}tt|j|\} } |j\} } } }|j |}t j js|s |j}n|jd| dz}t|D]m\}}|jr+t j js t||}n||}|| vsJ|j|r|j|n|o|ra|jj| |f\}}|Dcgc]6}|j| ||djdddd j!8}}|r|S|j|}||fScc}w) aVForward features that returns intermediates. Args: x: Input image tensor. indices: Take last n blocks if int, all if None, select matching indices if sequence. norm: Apply norm layer to all intermediates. stop_early: Stop iterating over blocks when last desired intermediate hit. output_fmt: Shape of intermediate feature outputs ('NCHW' or 'NLC'). intermediates_only: Only return intermediate features. Returns: List of intermediate features or tuple of (final features, intermediates). )NCHWNLCz)Output format must be one of NCHW or NLC.rNrrrrrH)rlenrshaperrUjit is_scripting enumeraterrappendrrdynamic_feat_sizereshapepermute contiguous)r>r@rrrrrrr intermediates take_indices max_indexBrheightwidthrrblkHWys rDforward_intermediateszMlpMixer.forward_intermediatess,_,Y.YY,& "6s4;;7G"Q i gg1fe IIaL 99 ! ! #:[[F[[)a-0F' BFAs&&uyy/E/E/GsA&FL $$TTYYq\qA B 99..?DAq^klYZQYYq!Q3;;Aq!QGRRTlMl  IIaL-ms7;F  prune_norm prune_headctt|j|\}}|jd|dz|_|rtj|_|r|j dd|S)aEPrune layers not required for specified intermediates. Args: indices: Indices of intermediate layers to keep. prune_norm: Whether to prune normalization layer. prune_head: Whether to prune the classifier head. Returns: List of indices that were kept. Nrrr)rrrr:r;rrr)r>rrrrrs rDprune_intermediate_layersz"MlpMixer.prune_intermediate_layerss] #7s4;;7G"Q ikk.9q=1  DI   ! !!R (rLc|j|}|jr5tjj st |j |}n|j |}|j|}|S)z/Forward pass through feature extraction layers.)rrrUrrrrrrrJs rDforward_featureszMlpMixer.forward_featuressV IIaL  " "599+A+A+Ct{{A.A AA IIaLrL pre_logitsc|jdk(r|jd}|j|}|r|S|j|S)zForward pass through classifier head. Args: x: Feature tensor. pre_logits: Return features before final classifier. Returns: Output tensor. rrr)rmeanrr)r>r@rs rD forward_headzMlpMixer.forward_headsD   u $1 A NN1 q0DIIaL0rLcJ|j|}|j|}|Sr)rrrJs rDrKzMlpMixer.forwards'  ! !! $   a rLF)T)N)NFFrF)rFT)&rMrNrOrPrrrr:rQrRr7r rSr rrTboolstrr6rUrignorer~rrrrrrrrVrrrrrrKrWrXs@rDrr sf $  ;E+5),*1",,D*I)+!$&$&#$)P%P%P% P%  P%  P%P%UE%,$778P%biiP%BIIP%RYYP%BIIP%P%"P%"P% !P%"#P%$%P%* +P%d YYNN$NN YY  D  T#s(^     YY)T)T)) YY  }C }hsm }W[ }"8<$$',7 ||7 eCcN347  7  7  7 !%7  tELL!5tELL7I)I#JJ K7 v./$# 3S >*  c 0%,,5<< 1ell 1 1 1%,,rLrnamerflaxr/ct|tjrC|jdrTtjj |j tjj|j|y|rKt|j |jtjj |jytjj|j |jZd|vr,tjj|jdytjj |jyyyt|tjrLt|j |j*tjj |jyyt|tjtjtjfrStjj!|j tjj |jyt#|dr|j%yy)zMixer weight initialization (trying to match Flax defaults). Args: module: Module to initialize. name: Module name. head_bias: Bias value for head layer. flax: Use Flax-style initialization. rNmlpr#rvr~) isinstancer:ri startswithrxzeros_rz constant_r|rxavier_uniform_ryConv2drQ BatchNorm2d GroupNormr{rr~)rrrrs rDrrsw&"))$ ??6 " GGNN6== ) GG  fkk9 5fmm,;;*GGNN6;;/'' 6;;*} >v{{3 + + FBII &fmm$ ;; " GGNN6;; ' # FR\\2>>2<<H I  fmm$ v{{#  (  )rLcJd|vri}|jD]\}}|jdd}|jdd}|jdd}|jdd }|jd s|jd r|jd d d }|||<|S|S)z Remap checkpoints if needed zpatch_embed.proj.weightz patch_embed.zstem.zattn.zlinear_tokens.zmlp.z mlp_channels.gamma_lsz.alphaz.betarr)itemsreplaceendswithr) state_dictmodelout_dictkrs rDcheckpoint_filter_fnr /s J.$$& DAq .'2A '#34A &/2A (D)Azz(#qzz'':IIaB'HQK  rLc r|jdd}tt||ftt |dd|}|S)N out_indicesrgetter)r  feature_cls)pretrained_filter_fn feature_cfg)poprrr r)variant pretrainedkwargsr rs rD _create_mixerr@sJ**]A.K  2[hG     E LrLc $|ddddddddddd d |S) Nr)rrr?bicubicT)r!r!r!z stem.projrz apache-2.0) urlr input_size pool_sizecrop_pct interpolationfixed_input_sizerrw first_conv classifierlicenser)rrs rD_cfgr!Ms4=tI4!    rLzmixer_s32_224.untrainedzmixer_s16_224.untrainedzmixer_b32_224.untrainedz mixer_b16_224.goog_in21k_ft_in1kztimm/zlhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_mixer_b16_224-76587d61.pth) hf_hub_idrzmixer_b16_224.goog_in21kzrhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_mixer_b16_224_in21k-617b3de2.pthiSU)r"rrzmixer_l32_224.untrainedz mixer_l16_224.goog_in21k_ft_in1kzlhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_mixer_l16_224-92f9adc4.pthzmixer_l16_224.goog_in21kzrhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-vitjx/jx_mixer_l16_224_in21k-846aa33c.pthzmixer_b16_224.miil_in21kzvhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-tresnet/mixer_b16_224_miil_in21k-2a558a71.pth)r&r&r&)?r#r#rbilineari+)r"rrrwrrrz mixer_b16_224.miil_in21k_ft_in1kzphttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-tresnet/mixer_b16_224_miil-9229a591.pth)r"rrrwrrzgmixer_12_224.untrained)rrwzgmixer_24_224.ra3_in1kzohttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/gmixer_24_224_raa-7daf7ae6.pth)r"rrrwzresmlp_12_224.fb_in1kz9https://dl.fbaipublicfiles.com/deit/resmlp_12_no_dist.pthzresmlp_24_224.fb_in1kz9https://dl.fbaipublicfiles.com/deit/resmlp_24_no_dist.pthzresmlp_36_224.fb_in1kz9https://dl.fbaipublicfiles.com/deit/resmlp_36_no_dist.pthzresmlp_big_24_224.fb_in1kz:https://dl.fbaipublicfiles.com/deit/resmlpB_24_no_dist.pthzresmlp_12_224.fb_distilled_in1kz6https://dl.fbaipublicfiles.com/deit/resmlp_12_dist.pthz6https://dl.fbaipublicfiles.com/deit/resmlp_24_dist.pthz6https://dl.fbaipublicfiles.com/deit/resmlp_36_dist.pthz7https://dl.fbaipublicfiles.com/deit/resmlpB_24_dist.pthz6https://dl.fbaipublicfiles.com/deit/resmlpB_24_22k.pthz6https://dl.fbaipublicfiles.com/deit/resmlp_12_dino.pthz6https://dl.fbaipublicfiles.com/deit/resmlp_24_dino.pthznhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/gmlp_s16_224_raa-10536d42.pth) resmlp_24_224.fb_distilled_in1kresmlp_36_224.fb_distilled_in1k#resmlp_big_24_224.fb_distilled_in1k"resmlp_big_24_224.fb_in22k_ft_in1kzresmlp_12_224.fb_dinozresmlp_24_224.fb_dinozgmlp_ti16_224.untrainedzgmlp_s16_224.ra3_in1kzgmlp_b16_224.untrainedc >tddddd|}tdd|i|}|S)zv Mixer-S/32 224x224 Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 rrrrrrr) mixer_s32_224rrrr model_argsrs rDr,r,1 KSKFKJ  Oj OJ OE LrLc >tddddd|}tdd|i|}|S)zw Mixer-S/16 224x224 Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 rrrr+rr) mixer_s16_224r-r.s rDr2r2r0rLc >tddddd|}tdd|i|}|S)zw Mixer-B/32 224x224 Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 r* r+rr) mixer_b32_224r-r.s rDr6r61 LcLVLJ  Oj OJ OE LrLc >tddddd|}tdd|i|}|S)z Mixer-B/16 224x224. ImageNet-1k pretrained weights. Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 rr4r5r+rr) mixer_b16_224r-r.s rDr9r9r7rLc >tddddd|}tdd|i|}|S)zx Mixer-L/32 224x224. Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 r*r+rr) mixer_l32_224r-r.s rDr=r=1 MdMfMJ  Oj OJ OE LrLc >tddddd|}tdd|i|}|S)z Mixer-L/16 224x224. ImageNet-1k pretrained weights. Paper: 'MLP-Mixer: An all-MLP Architecture for Vision' - https://arxiv.org/abs/2105.01601 rr;r<r+rr) mixer_l16_224r-r.s rDr@r@r>rLc htdddddttjd|}t dd|i|}|S) zV Glu-Mixer-12 224x224 Experiment by Ross Wightman, adding SwiGLU to MLP-Mixer rr4r#r"rrrr)r*r,rr) gmixer_12_224rrr:SiLUrr.s rDrErEF 7"zBGG7/57J  Oj OJ OE LrLc htdddddttjd|}t dd|i|}|S) zV Glu-Mixer-24 224x224 Experiment by Ross Wightman, adding SwiGLU to MLP-Mixer rr;rBrCrDrr) gmixer_24_224rFr.s rDrJrJrHrLc Ttdddddttd|}tdd|i|}|S) zx ResMLP-12 Paper: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 rr4rBrfrrrr)rr+rr) resmlp_12_224)rrerZrr.s rDrMrMsC u"qhciumsuJ  Oj OJ OE LrLc jtd ddddttdtd|}t d d|i|}|S) zx ResMLP-24 Paper: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 rr;rBrfgh㈵>rgrLrr) resmlp_24_224rrrerZrr.s rDrPrPL V"qH$7FVNTVJ  Oj OJ OE LrLc jtd ddddttdtd|}t d d|i|}|S) zx ResMLP-36 Paper: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 r$rBrfr#rOrLrr) resmlp_36_224rQr.s rDrUrU(rRrLc jtd ddddttdtd|}t d d|i|}|S) zz ResMLP-B-24 Paper: `ResMLP: Feedforward networks for image classification...` - https://arxiv.org/abs/2105.03404 rr;r5rfr#rOrLrr)resmlp_big_24_224rQr.s rDrWrW4sL VsaH$7FVNTVJ  S* S SE LrLc Ttdddddttd|}tdd|i|}|S) zU gMLP-Tiny Paper: `Pay Attention to MLPs` - https://arxiv.org/abs/2105.08050 rrrrr)rr*rr) gmlp_ti16_224rrrrr.s rDr]r]@sC &"qN`&$&J  Oj OJ OE LrLc Ttdddddttd|}tdd|i|}|S) zV gMLP-Small Paper: `Pay Attention to MLPs` - https://arxiv.org/abs/2105.08050 rrYr[r\rr) gmlp_s16_224r^r.s rDraraLC &"qN`&$&J  NZ N: NE LrLc Ttdddddttd|}tdd|i|}|S) zU gMLP-Base Paper: `Pay Attention to MLPs` - https://arxiv.org/abs/2105.08050 rrYrr[r\rr) gmlp_b16_224r^r.s rDrdrdXrbrLr%r&r'r(rMrP) mixer_b16_224_in21kmixer_l16_224_in21kmixer_b16_224_miilmixer_b16_224_miil_in21kresmlp_12_distilled_224resmlp_24_distilled_224resmlp_36_distilled_224resmlp_big_24_distilled_224resmlp_big_24_224_in22ft1kresmlp_12_224_dinoresmlp_24_224_dino)r&Fr)r)FrPr functoolsrtypingrrrrrr r rUtorch.nnr: timm.datar r timm.layersr rrrrrr_builderr _featuresr _manipulaterrr _registryrrr__all__rTrrZrerprrrrSrrr rr! default_cfgsr,r2r6r9r=r@rErJrMrPrUrWr]rardrMrrLrDr{sK'P @@@ A]]]*+@@YY  $++\7RYY7&/ryy/d%' %'P00f{ryy{|%"))%3%5%T%^b%P" (  d38n %a&tva&tva&tva&' z) a& A!a&tva&' z)a&$ A!%a&2 E |e:ch!3a&<' ~ |e:)=a&Ht)>DXYIa&Jd } "(<>Ka&TT G "(<>Ua&\T G "(< >]a&fT G "(<>ga&n  H "(<">oa&x&t D "(<(>ya&@(, D "(<(>(, D "(<(>,0 E "(<,> +/ D "(<+> " D "(<>" D "(<> $v! |#fAa&a HXH==< :@@@#H"F)) ' rL