^j!dZddlmZmZmZddlZddlmZddlmcm Z ddl m Z ddl mZmZmZGddej"Ze Gd d ej"ZGd d ej"Ze Gd dej"Zy)a Convolution with Weight Standardization (StdConv and ScaledStdConv) StdConv: @article{weightstandardization, author = {Siyuan Qiao and Huiyu Wang and Chenxi Liu and Wei Shen and Alan Yuille}, title = {Weight Standardization}, journal = {arXiv preprint arXiv:1903.10520}, year = {2019}, } Code: https://github.com/joe-siyuan-qiao/WeightStandardization ScaledStdConv: Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 Official Deepmind JAX code: https://github.com/deepmind/deepmind-research/tree/master/nfnets Hacked together by / copyright Ross Wightman, 2021. )OptionalTupleUnionN)register_notrace_module) get_paddingget_padding_valuepad_sameceZdZdZ d dededeeeeeffdeeeeeffdeeeeeeffdeeeeeffded ed e ffd Z d Z xZ S) StdConv2dzConv2d with Weight Standardization. Used for BiT ResNet-V2 models. Paper: `Micro-Batch Training with Batch-Channel Normalization and Weight Standardization` - https://arxiv.org/abs/1903.10520v2 in_channel out_channels kernel_sizestridepaddingdilationgroupsbiasepsc d| t|||}t | ||||||||| |  | |_y)Nrrrrrdevicedtype)rsuper__init__r) selfr rrrrrrrrrr __class__s _/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/layers/std_conv.pyrzStdConv2d.__init__#sL ?!+vx@G   k&hvDQW_d  fc ftj|jjd|jddddd|j j |j}tj|||j|j|j|j|j}|SNrT)trainingmomentumr) F batch_normweightreshaperr reshape_asconv2drrrrrrxr(s rforwardzStdConv2d.forward8s KK  4#4#4b 9     *T[[ !  HHQ 4;; dmmUYU`U` ar)rNrrFư>NN) __name__ __module__ __qualname____doc__intrrrboolfloatrr. __classcell__rs@rr r s34=A45sE#s(O34  #uS#X./  eCsCx$89: CsCx01* rr ceZdZdZ d dededeeeeeffdeeeeeffdedeeeeeffded ed e ffd Z d Z xZ S) StdConv2dSamezConv2d with Weight Standardization. TF compatible SAME padding. Used for ViT Hybrid model. Paper: `Micro-Batch Training with Batch-Channel Normalization and Weight Standardization` - https://arxiv.org/abs/1903.10520v2 r rrrrrrrrc xt||||\}} t | ||||||||| |  | |_| |_y)Nrrr)r rrsame_padr)rr rrrrrrrrrr is_dynamicrs rrzStdConv2dSame.__init__LsU0V^fg   k&'\dV5  B# rc |jr,t||j|j|j}t j |jjd|jddddd|jj|j}t j|||j|j|j|j|j}|Sr!)r=r rrrr&r'r(r)rrr*r+rrrr,s rr.zStdConv2dSame.forwardas ==D,,dkk4==IA KK  4#4#4b 9     *T[[ !  HHQ 4;; dmmUYU`U` ar)rSAMErrFr/NN) r0r1r2r3r4rrstrr5r6rr.r7r8s@rr:r:Es34!45sE#s(O34  #uS#X./   CsCx01* rr:ceZdZdZ ddededeeeeeffdeeeeeffdeeeeeefefdeeeeeffded e d e d e d e ffd Z dfd Z dZ xZS)ScaledStdConv2da4Conv2d layer with Scaled Weight Standardization. Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 NOTE: the operations used in this impl differ slightly from the DeepMind Haiku impl. The impact is minor. in_channelsrrrrrrrgammar gain_initc b| | d}| t|||}t||||f|||||d|| |jdj dzz|_| |_| |_tjtj|jdddffi||_ |jy)Nrrrrrrrrr)rrrr(numelscalerrFnn Parametertorchemptyrgainreset_parameters)rrDrrrrrrrrErrFrrddrs rrzScaledStdConv2d.__init__ys  / ?!+vx@G  { ,;A7]e ,(* ,T[[^113t;; "LLd.?.?Aq-I!PR!PQ  rct|drMtjjj |j |j t|!yyNrQ hasattrrOrMinit constant_rQrFrrRrrs rrRz ScaledStdConv2d.reset_parameters= 4 HHMM # #DIIt~~ > G $ & !rc tj|jjd|jddd|j |j zjddd|jj|j}tj|||j|j|j|j|jSNrr"Tr#)r(r$r%r)r&r'r(r)rrQrLviewrr*r+rrrrrr,s rr.zScaledStdConv2d.forwards KK  4#4#4b 9  II *004  *T[[ ! xx6499dkk4<<X\XcXcddr) rNrrT?r/r_NNreturnN)r0r1r2r3r4rrrrAr5r6rrRr.r7r8s@rrCrCps34BF45"   sE#s(O34  #uS#X./  eCsCx#$=>?  CsCx01      <' errCceZdZdZ ddededeeeeeffdeeeeeffdedeeeeeffded ed e d e d e ffd Z dfd Z dZ xZ S)ScaledStdConv2dSamea\Conv2d layer with Scaled Weight Standardization and Tensorflow-like SAME padding support Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 NOTE: the operations used in this impl differ slightly from the DeepMind Haiku impl. The impact is minor. rDrrrrrrrrErrFc v| | d}t||||\}}t||||f|||||d|| |jdj dzz|_||_| |_| |_tjtj|jdddffi||_|jy)NrHr<rIrrJr)r rrr(rKrLr=rrFrMrNrOrPrrQrR)rrDrrrrrrrrErrFrrrSr>rs rrzScaledStdConv2dSame.__init__s  //V^fg  { ,;A7]e ,(* ,T[[^113t;; " "LLd.?.?Aq-I!PR!PQ  rct|drMtjjj |j |j t|!yyrUrVrZs rrRz$ScaledStdConv2dSame.reset_parametersr[rc  |jr,t||j|j|j}t j |jjd|jddd|j|jzjddd|jj|j}t j|||j |j|j"|j|j$Sr])r=r rrrr&r'r(r)rrQrLr^rr*r+rrrr,s rr.zScaledStdConv2dSame.forwards ==D,,dkk4==IA KK  4#4#4b 9  II *004  *T[[ ! xx6499dkk4<<X\XcXcddr) rr@rrTr_r/r_NNr`)r0r1r2r3r4rrrAr5r6rrRr.r7r8s@rrcrcs34!45"   sE#s(O34  #uS#X./    CsCx01      <' errc)r3typingrrrrOtorch.nnrMtorch.nn.functional functionalr&_fxrrrr r Conv2dr r:rCrcrrrns$*) (==% %P'BII''T9ebii9ex;e"));e;er