^jc dZddlZddlmZddlmZmZmZmZm Z ddl Z ddl m cm Zddl m Z ddlmZmZddlmZmZmZmZmZmZmZmZdd lmZdd lmZdd lm Z dd l!m"Z"m#Z#dd l$m%Z%m&Z&dgZ'e Gdde jPZ)Gdde jPZ*Gdde jPZ+Gdde jPZ,Gdde jPZ-Gdde jPZ.d,dZ/dZ0d-dZ1d.dZ2e&e2dd d!e2dd d!e2dd"d#d$e2dd"d#d$e2dd"d#d$e2dd#d!d%Z3e%d-d&e.fd'Z4e%d-d&e.fd(Z5e%d-d&e.fd)Z6e%d-d&e.fd*Z7e%d-d&e.fd+Z8y)/a# EdgeNeXt Paper: `EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications` - https://arxiv.org/abs/2206.10589 Original code and weights from https://github.com/mmaaz60/EdgeNeXt Modifications and additions for timm by / Copyright 2022, Ross Wightman N)partial)ListOptionalTupleTypeUnion)nnIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)DropPathcalculate_drop_path_rates LayerNorm2dMlp create_conv2dNormMlpClassifierHeadClassifierHeadtrunc_normal_tf_)build_model_with_cfg)feature_take_indices)register_notrace_module) named_applycheckpoint_seq)register_modelgenerate_default_cfgsEdgeNeXtcPeZdZ ddededeffd ZdeeeeffdZxZS)PositionalEncodingFourier hidden_dimdim temperaturec||d}t|tj|dz|fddi||_dt j z|_||_||_ ||_ y)Ndevicedtype kernel_sizer) super__init__r Conv2dtoken_projectionmathpiscaler"r r!)selfr r!r"r%r&dd __class__s _/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/models/edgenext.pyr*z"PositionalEncodingFourier.__init__)sa /  " *q.# S1 SPR S[ &$shapec n|jjj}|jjj}t j |j |tj}|jdtj}|jdtj}d}||ddddddf|zz |jz}||ddddddf|zz |jz}t j|jtj|j tj}|jdt j|dd z|jz z}|dddddddf|z } |dddddddf|z } t j | ddddddd ddfj#| dddddddddfj%fd j'd } t j | ddddddd ddfj#| dddddddddfj%fd j'd } t j(| | fd j+d d dd} |j| j |} | S)Nr$r)r&r'ư>)r&r%floor) rounding_moderr!)r,weightr%r&torchzerostoboolcumsumfloat32r/aranger int64r"divstacksincosflattencatpermute) r0r5r%r&inv_masky_embedx_embedepsdim_tpos_xpos_yposs r3forwardz!PositionalEncodingFourier.forward9s^&&--44%%,,22KK&))uzz)JJ//!5==/9//!5==/9WQQY/#56CWQ23Y/#56C T__EKKORRSXS`S`a  Q5!7)S%SVZVeVe%ef1a &.1a &. 1aADqD= ! % % ' 1aADqD= ! % % ' )./118  1aADqD= ! % % ' 1aADqD= ! % % ' )./118 iiA.66q!QB##CFF5M2 r4) ig@NN) __name__ __module__ __qualname__intfloatr*rrV __classcell__r2s@r3rr'sL!!'   U3S=1r4rceZdZddddddeej dej dddf d ed eed ed ed e de de de ejde ejde ffd Z dZxZS) ConvBlockNrTr;r7rQr!dim_outr(stride conv_bias expand_ratio ls_init_value norm_layer act_layer drop_pathc | | d} t||xs|}|dkDxs||k7|_t||f||d|d| |_||fi| |_t |t||zfd| i| |_|dkDr,tj|tj|fi| znd|_ | dkDrt| |_ytj|_y)Nr$rT)r(re depthwisebiasrjrrc)r)r*shortcut_after_dwrconv_dwnormrr[mlpr Parameterr?onesgammar Identityrk)r0r!rdr(rerfrgrhrirjrkr%r&r1r2s r3r*zConvBlock.__init__Us / .S!'!!=sg~$   $    w-"-    w& '      Q^`aPaR\\-%**W2K2K"KLgk 09B),BKKMr4c:|}|j|}|jr|}|jdddd}|j|}|j |}|j |j |z}|jdddd}||j |z}|S)Nrr'r=r)rprorMrqrrrurk)r0xshortcuts r3rVzConvBlock.forward|s LLO  ! !H IIaAq ! IIaL HHQK :: ! QA IIaAq ! t~~a( (r4)rXrYrZrr LayerNormGELUr[rrBr\rModuler*rVr]r^s@r3r`r`Ts&* ""##'*1",,D*I)+!%R%Rc]%R %R  %R  %R %R!%RRYY%RBII%R%RNr4r`c eZdZ d dededededef fd ZdZejjdZ xZ S) CrossCovarianceAttnr! num_headsqkv_bias attn_drop proj_dropc|||d}t |||_tjt j |ddfi||_tj||dzfd|i||_ tj||_ tj||fi||_ tj||_ y)Nr$rr=rn)r)r*rr rsr?rtr"LinearqkvDropoutrprojr) r0r!rrrrr%r&r1r2s r3r*zCrossCovarianceAttn.__init__s / "<< 9a(Ib(IJ99S#'??B?I.IIc3-"- I.r4cH|j\}}}|j|j||d|jdj ddddd}|j d\}}}t j|dt j|djddz|jz} | jd} |j| } | |z}|j ddddj|||}|j|}|j|}|S) Nr=r8r'rr;rr<)r5rreshaperrMunbindF normalize transposer"softmaxrrr) r0rxBNCrqkvattns r3rVzCrossCovarianceAttn.forwards''1ahhqk!!!Q4>>2>FFq!QPQSTU**Q-1a A2&QB)?)I)I"b)QQUYUeUee|||#~~d# AX IIaAq ! ) )!Q 2 IIaL NN1 r4cdhS)Nr"r0s r3no_weight_decayz#CrossCovarianceAttn.no_weight_decays r4)FrcrcNN) rXrYrZr[rBr\r*rVr?jitignorerr]r^s@r3r~r~so"!!/// /  /  /(  YYr4r~ceZdZdddddddeej dej dddddfd ed ed ed ed e de de dede ejde ejdededeffd Z dZ xZS)SplitTransposeBlockrrr;Tr7rbrcNr! num_scalesrrg use_pos_embrfrrhrirjrkrrc ||d}t|ttt j ||z tt j ||z}||_td|dz |_g}t|jD]"}|jt||fdd|d|$tj||_d|_|rt!d d|i||_| |fi||_|dkDr,tj$|t'j(|fi|znd|_t-|f||| | d||_| |fd d i||_t3|t||zfd | i||_|dkDr,tj$|t'j(|fi|znd|_| d kDrt9| |_ytj:|_y)Nr$rr=T)r(rmrnr!r)rrrrrQr7rjrcr)r)r*maxr[r-ceilr9widthrrangeappendrr ModuleListconvspos_embdrnorm_xcarsr?rt gamma_xcar~xcarqrrrrur rvrk)r0r!rrrgrrfrrhrirjrkrrr%r&r1rrir2s r3r*zSplitTransposeBlock.__init__s$ / C # "234c$**SJEV:W6XY aa0t' kA LLueidYbifhi j k]]5)  5D#DDDM"3-"- P]`aPamejj6K6K&KLgk&      s333    s" #      MZ\]L]R\\-%**S2GB2G"GHcg 09B),BKKMr4c |}|jt|jdzd}g}|d}t|jD]+\}}|dkDr|||z}||}|j |-|j |dt j |d}|j\}} } } |j|| | | zjddd}|jH|j|| | fj|d|jdjddd} || z}||j|j|j|j|zz}|j|| | | }|j|}|j!|}|j"|j"|z}|jdddd}||j|z}|S)Nrr<rr8r'r=)chunklenr enumeraterr?rLr5rrMrrkrrrrqrrru) r0rxryspxsposprconvrrHW pos_encodings r3rVzSplitTransposeBlock.forwardsggc$**o)qg1 V , GAt1u#a&[bB JJrN   3r7 IIc1 WW 1a IIaAE " * *1a 3 == $==!Q3;;Ar1771:NVVWXZ[]^_LL A t~~q9I0JJK K IIaAq ! IIaL HHQK :: ! QA IIaAq ! t~~a( (r4)rXrYrZrr rzr{r[r\rBrr|r*rVr]r^s@r3rrs "# $"!#'*1",,D*I)+!!!!3R3R3R 3R  3R  3R3R3R!3RRYY3RBII3R3R3R3Rj!r4rc$ eZdZddddddddddddeeej d ejddfd ed ed edededededede de de de de de e e de ejde ejde ejf"fd ZdZxZS) EdgeNeXtStager'rr;raFT?Nr7rbin_chsout_chsredepthnum_global_blocksrscalesr(rgrdownsample_blockrfrhdrop_path_ratesri norm_layer_clrjc||d}t|d|_| s|dk(rtj|_n=tj ||fi|tj||fdd| d||_|}g}t|D]f}|||z kr2|jtd ||| r|dk(r|nd| || | ||||d |n(|jtd |||| | | | ||||d ||}htj ||_ y) Nr$Frr'r(rernr) r!rdrerfr(rgrhrkrirj) r!rrrgrrfrhrkrirjr) r)r*grad_checkpointingr rv downsample Sequentialr+rrr`rblocks)r0rrrerrrrr(rgrrrfrhrrirrjr%r&r1 stage_blocksrr2s r3r*zEdgeNeXtStage.__init__sV, / "' v{ kkmDO mm6(R( &'YqYVXYDOF u! A5,,,## " ')9a1fv!"+$/%1&3"1!"4#0"+   ##' "#)"+%1$/"+&3"1!"4#0"+  FC! Dmm\2 r4c|j|}|jr6tjj st |j |}|S|j |}|SN)rrr?r is_scriptingrrr0rxs r3rVzEdgeNeXtStage.forwardZsS OOA   " "599+A+A+Ct{{A.A AAr4)rXrYrZrrr rzr{r[r\rBrrrr|r*rVr]r^s@r3rrs- %& "# %%*##&59*5-4R\\t-L)+)F3F3F3 F3  F3 # F3F3F3F3 F3F3#F3F3!F3&d5k2F3 RYY!F3"  ?#F3$BII%F3Pr4rc)eZdZddddddddd d d d d ddddejddddfdedededeedfdeedfdeedfdeedfdeedfdeedfdeedfde de d e d!ed"ed#ed$ed%e ejd&e d'e f(fd( Z e jjddedeefd-Z d?d.e j*d/eeeeefd0ed1ed2ed3ed+eee j*ee j*ee j*fffd4Z d@d/eeeefd5ed6efd7Zd8Zd\} } | |}| |vs|r| | k(r|j|}n|}|j|@|r|S | k(r|j|}||fS)a Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence norm: Apply norm layer to compatible intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features Returns: )NCHWzOutput shape must be NCHW.rN) rrrrr?rrrrr)r0rxr rqr r r intermediates take_indices max_indexlast_idxrfeat_idxstagex_inters r3forward_intermediateszEdgeNeXt.forward_intermediatess*Y&D(DD& "6s4;;7G"Q i IIaLt{{#a' 99 ! ! #:[[F[[)a-0F(0 .OHeaA<'H0"mmA.GG$$W- .  x  a A-r4 prune_norm prune_headctt|j|\}}|jd|dz|_|rtj|_|r|j dd|S)z@ Prune layers not required for specified intermediates. Nrr)rrrr rvrr )r0r rrrrs r3prune_intermediate_layersz"EdgeNeXt.prune_intermediate_layerss]#7s4;;7G"Q ikk.9q=1 KKMDM   ! !!R (r4cl|j|}|j|}|j|}|Sr)rrrrs r3forward_featureszEdgeNeXt.forward_features&s/ IIaL KKN MM! r4 pre_logitscN|r|j|dS|j|S)NT)r )r)r0rxr s r3 forward_headzEdgeNeXt.forward_head,s$0:tyyty,L ! Lr4cJ|j|}|j|}|Sr)rr"rs r3rVzEdgeNeXt.forward/s'  ! !! $   a r4F)Tr)NFFrF)rFT)rXrYrZr r{r[strrrBr\rr|r*r?rrrrrrr Tensorrrrrrr"rVr]r^s@r3rrcs#$$5&23?,8%1)5,G#'%'"#%*"$$))+$&!/fSfSfS fS S/ fS #s(O fS"'sCxfS S/fSc?fSS#XfStSy)fS!fS#fS fS#fS !fS"#fS$"%fS&BII'fS(")fS*+fSP YY   YY** YY 2C2hsm28<$$',0 ||0 eCcN340  0  0  0 !%0  tELL!5tELL7I)I#JJ K0 h./$# 3S >*   M$Mr4rc.t|tjrNt|jd|j *tj j|j yyt|tjrt|jdtj j|j |rPd|vrK|jjj||j jj|yyyy)Ng{Gz?)stdhead.) isinstancer r+rr>rninitzeros_rdatamul_)rnamers r3rr5s&"))$C0 ;; " GGNN6;; ' # FBII &C0 v{{# GtO MM   # #O 4 KK   ! !/ 2$4 'r4c>d|vsd|vr|Sd|vr|d}nd|vr|d}n d|vr|d}i}ddl}|jD]\}}|jdd }|jd d |}|jd d |}|jdd}|jdd}|jdd}|j dr|jdd}|j dk(r2d|vr.|j |j}|j|}|||<|S)z Remap FB checkpoints -> timm zhead.norm.weightznorm_pre.weight model_emamodel state_dictrNzdownsample_layers.0.zstem.zstages.([0-9]+).([0-9]+)zstages.\1.blocks.\2z#downsample_layers.([0-9]+).([0-9]+)zstages.\1.downsample.\2dwconvrppwconvzmlp.fcr)zhead.fc.znorm.rqz head.normr'r) reitemsreplacesub startswithndimr3r5r)r3r2out_dictr6rr model_shapes r3checkpoint_filter_fnr>BsBZ'+< +Jj  , J (  # - H   " 1 II,g 6 FF.0F J FF9;UWX Y IIh * IIh ) IIgz * <<  &+.A 66Q;6?**,Q/55K +&A   Or4c Ntt||fttddd|}|S)N)rrr'r=T) out_indicesflatten_sequential)pretrained_filter_fn feature_cfg)rrr>r)variant pretrainedkwargsr2s r3_create_edgenextrGas6 ': 1\dK    E Lr4c 2|dddddttdddd |S) Nr)r=rI)rrg?bicubiczstem.0zhead.fcmit) urlr input_size pool_sizecrop_pct interpolationmeanr( first_conv classifierlicenser )rLrFs r3_cfgrUjs3=v)%.Bi    r4ztimm/)r= rV) hf_hub_idtest_input_size test_crop_pctgffffff?)r=@rZ)rWrOrXrY)zedgenext_xx_small.in1kzedgenext_x_small.in1kzedgenext_small.usi_in1kzedgenext_base.usi_in1kzedgenext_base.in21k_ft_in1kzedgenext_small_rw.sw_in1krc Ltddd}tdd|it|fi|S)N)r'r'r'rr;r;r;r;rrrrE)edgenext_xx_smallrrGrErF model_argss r3r_r_s1\0AVJ  eJ e$zJd]cJd eer4c Ltddd}tdd|it|fi|S)Nr)rW@dr]r^rE)edgenext_x_smallr`ras r3rgrgs1\0B,WJ  d: djIc\bIc ddr4c Jtdd}tdd|it|fi|S)Nr)r`i0rrrE)edgenext_smallr`ras r3rlrls/\0BCJ  b btJGaZ`Ga bbr4c Rtgdgd}tdd|it|fi|S)Nr)PrjrViHrkrE) edgenext_baser`ras r3roros/\0CDJ  a ad:F`Y_F` aar4c Ptddddd}tdd|it|fi|S) Nr)rrirfiTFr)rrrrfrrE)edgenext_small_rwr`ras r3rqrqs:"4)EJ  eJ e$zJd]cJd eer4)Nrr$)r)9__doc__r- functoolsrtypingrrrrrr?torch.nn.functionalr functionalr timm.datar r timm.layersr rrrrrrr_builderr _featuresr _features_fxr _manipulaterr _registryrr__all__r|rr`r~rrrrr>rGrU default_cfgsr_rgrlrorqrr4r3rs 55 A   ++14< ,) ))X6 6r'"))'TW"))WtOBIIOdOryyOd 3> %"%S:"%S: $}C #}C$(}C$"&%S"'& 4fXffeHeec(ccbbbfXffr4