^joJdZddlmZddlmZmZddlZddlmZddlm Z m Z ddl m Z ddl mZmZmZd d lmZd d lmZmZmZd gZGd d eZd\dZd]dZeidedddedddedddedddddeddd d!edd"d d#edd$d d%edd&ddd'd(edd)d*edd+ddd,edd-d.edd/d0edd1ddd2edd3d4edd5ddd6edd7d8edd9d:edd;ddeddddedd?d:edd@ddeddAd:dBZed^dCefdDZed^dCefdEZed^dCefdFZed^dCefdGZed^dCefdHZ ed^dCefdIZ!ed^dCefdJZ"ed^dCefdKZ#ed^dCefdLZ$ed^dCefdMZ%ed^dCefdNZ&ed^dCefdOZ'ed^dCefdPZ(ed^dCefdQZ)ed^dCefdRZ*ed^dCefdSZ+ee,d8dTdUdVdWdXdYdZd[y)_a[ DeiT - Data-efficient Image Transformers DeiT model defs and weights from https://github.com/facebookresearch/deit, original copyright below paper: `DeiT: Data-efficient Image Transformers` - https://arxiv.org/abs/2012.12877 paper: `DeiT III: Revenge of the ViT` - https://arxiv.org/abs/2204.07118 Modifications copyright 2021, Ross Wightman )partial)OptionalTypeN)nnIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)resample_abs_pos_embed)VisionTransformer trunc_normal_checkpoint_filter_fn)build_model_with_cfg)generate_default_cfgsregister_modelregister_model_deprecationsVisionTransformerDistilledc8eZdZdZfdZdfd ZejjddZ ejjde jfdZ dde deefd Zejjdd Zd Zdd edej(fd ZxZS)rz Vision Transformer w/ Distillation Token and Head Distillation token & head support for `DeiT: Data-efficient Image Transformers` - https://arxiv.org/abs/2012.12877 c|jdd}t||i|ddi|jdvsJ|j dd|j ddd}d|_t jtjd d |jfi||_ t jtjd |jj|j z|jfi||_|jd kDr+t j |j|jfi|nt j"|_d |_|dk(rd n||_|dk7r|j+d yy)N weight_initskip)tokendevicedtype)rrrrFreset) needs_reset)popsuper__init__ global_poolgetnum_prefix_tokensr Parametertorchempty embed_dim dist_token patch_embed num_patches pos_embed num_classesLinearIdentity head_distdistilled_trainingweight_init_mode init_weights)selfargskwargsrdd __class__s [/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/models/deit.pyr!z#VisionTransformerDistilled.__init__#sDjj3  $=&=f=:--- 8T2VZZQU=V W!",,u{{1a'N2'NO KK4++77$:P:PPRVR`R` gdf giNRN^N^abNb4>>43C3CJrJhjhshshu"'+6&+@k &   %  0 !cv|xs |j}t|jdt|||y)Ng{Gz?)std)moder)r2r r)r r3)r4r=rr8s r9r3z'VisionTransformerDistilled.init_weights5s3,t,,doo3/ $K@r:c tdddgS)Nz+^cls_token|pos_embed|patch_embed|dist_token)z^blocks\.(\d+)N)z^norm)i)stemblocks)dict)r4coarses r9 group_matcherz(VisionTransformerDistilled.group_matcher:s?)$&  r:returnc2|j|jfSNheadr0)r4s r9get_classifierz)VisionTransformerDistilled.get_classifierCsyy$..((r:r-r"c*||_|dkDr tj|j|ntj|_|dkDr0tj|j|j|_ytj|_y)Nr)r-rr.r(r/rHr0)r4r-r"s r9reset_classifierz+VisionTransformerDistilled.reset_classifierGse&>IAoBIIdnnk:SUS^S^S` HSVW4>>43C3CD]_]h]h]jr:c||_yrF)r1)r4enables r9set_distilled_trainingz1VisionTransformerDistilled.set_distilled_trainingLs "(r:c|jrn|j\}}}}|jj}t |j ||f||j rdn |j}|j|d|}n |j }|j rp||z}tj|jj|jddd|jj|jddd|fd}notj|jj|jddd|jj|jddd|fd}||z}|j|S)Nr)new_sizeold_sizer$r)dim)dynamic_img_sizeshaper* grid_sizer r,no_embed_classr$viewr&cat cls_tokenexpandr)pos_drop)r4xBHWCprev_grid_sizer,s r9 _pos_embedz%VisionTransformerDistilled._pos_embedPsV  JAq!Q!--77N.Q''+':':!@V@V I q"a AI   I A %%aggaj"b9&&qwwqz2r: A %%aggaj"b9&&qwwqz2r: A I A}}Qr: pre_logitsc|dddf|dddf}}|r||zdz S|j|}|j|}|jr.|jr"tj j s||fS||zdz S)Nrrr)rHr0r1trainingr&jit is_scripting)r4r]rdx_dists r9 forward_headz'VisionTransformerDistilled.forward_headqsadGQq!tW6 J!# # IIaL'  " "t}}UYY=S=S=Uf9 J!# #r:)rTFrF)T)__name__ __module__ __qualname____doc__r!r3r&rgignorerCrModulerIintrstrrKrNrcboolTensorrj __classcell__)r8s@r9rrs 1$A  YY   YY) ))kCkhsmk  YY)) B $$ $5<< $r:c |jdd}|rtnt}t|||ft t dt |dd|}|S)N out_indicesT)adapt_layer_scalegetter)rx feature_cls)pretrained_filter_fn feature_cfg)rrr rrr rA)variant pretrained distilledr6rx model_clsmodels r9 _create_deitrs]**]A.K.7*=NI  %%9TR[hG     E Lr:c 4|ddddddttdddd |S) Ni)ryrg?bicubicTzpatch_embed.projrHz apache-2.0) urlr- input_size pool_sizecrop_pct interpolationfixed_input_sizemeanr< first_conv classifierlicenser)rr6s r9_cfgrs5=t%.B(    r:zdeit_tiny_patch16_224.fb_in1kztimm/zFhttps://dl.fbaipublicfiles.com/deit/deit_tiny_patch16_224-a1311bcf.pth) hf_hub_idrzdeit_small_patch16_224.fb_in1kzGhttps://dl.fbaipublicfiles.com/deit/deit_small_patch16_224-cd65a155.pthzdeit_base_patch16_224.fb_in1kzFhttps://dl.fbaipublicfiles.com/deit/deit_base_patch16_224-b5f2ef4d.pthzdeit_base_patch16_384.fb_in1kzFhttps://dl.fbaipublicfiles.com/deit/deit_base_patch16_384-8de9b5d1.pth)ryrg?)rrrrz'deit_tiny_distilled_patch16_224.fb_in1kzPhttps://dl.fbaipublicfiles.com/deit/deit_tiny_distilled_patch16_224-b40b3cf7.pthrG)rrrz(deit_small_distilled_patch16_224.fb_in1kzQhttps://dl.fbaipublicfiles.com/deit/deit_small_distilled_patch16_224-649709d9.pthz'deit_base_distilled_patch16_224.fb_in1kzPhttps://dl.fbaipublicfiles.com/deit/deit_base_distilled_patch16_224-df68dfff.pthz'deit_base_distilled_patch16_384.fb_in1kzPhttps://dl.fbaipublicfiles.com/deit/deit_base_distilled_patch16_384-d0272ac0.pth)rrrrrzdeit3_small_patch16_224.fb_in1kz;https://dl.fbaipublicfiles.com/deit/deit_3_small_224_1k.pthzdeit3_small_patch16_384.fb_in1kz;https://dl.fbaipublicfiles.com/deit/deit_3_small_384_1k.pthz deit3_medium_patch16_224.fb_in1kzhttps://dl.fbaipublicfiles.com/deit/deit_3_huge_224_21k_v1.pth)(deit3_small_patch16_384.fb_in22k_ft_in1k)deit3_medium_patch16_224.fb_in22k_ft_in1k'deit3_base_patch16_224.fb_in22k_ft_in1k'deit3_base_patch16_384.fb_in22k_ft_in1k(deit3_large_patch16_224.fb_in22k_ft_in1k(deit3_large_patch16_384.fb_in22k_ft_in1k'deit3_huge_patch14_224.fb_in22k_ft_in1krDc Rtdddd}tdd|it|fi|}|S)z DeiT-tiny model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit.  ry patch_sizer(depth num_headsr)deit_tiny_patch16_224rArrr6 model_argsrs r9rrs7 s"JJ  fZ f4PZKe^dKe fE Lr:c Rtdddd}tdd|it|fi|}|S)z DeiT-small model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrr)deit_small_patch16_224rrs r9rrs7 s"JJ  gj gDQ[Lf_eLf gE Lr:c Rtdddd}tdd|it|fi|}|S)z DeiT base model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrr)deit_base_patch16_224rrs r9rr7 s"KJ  fZ f4PZKe^dKe fE Lr:c Rtdddd}tdd|it|fi|}|S)z DeiT base model @ 384x384 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrr)deit_base_patch16_384rrs r9rrrr:c Vtdddd}t d|ddt|fi|}|S) z DeiT-tiny distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrryrTrr)deit_tiny_distilled_patch16_224rrs r9rr"sG s"JJ ) p6@D pTXYcTngmTn pE Lr:c Vtdddd}t d|ddt|fi|}|S) z DeiT-small distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrrTr) deit_small_distilled_patch16_224rrs r9rr-sG s"JJ * q7AT qUYZdUohnUo qE Lr:c Vtdddd}t d|ddt|fi|}|S)z DeiT-base distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrTr)deit_base_distilled_patch16_224rrs r9rr8G s"KJ ) p6@D pTXYcTngmTn pE Lr:c Vtdddd}t d|ddt|fi|}|S)z DeiT-base distilled model @ 384x384 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrTr)deit_base_distilled_patch16_384rrs r9rrCrr:c Vtdddddd}td d|it|fi|}|S) z DeiT-3 small model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrTư>rr(rrrW init_valuesr)deit3_small_patch16_224rrs r9rrN= s"Z^lpqJ  hz hTR\Mg`fMg hE Lr:c Vtdddddd}td d|it|fi|}|S) z DeiT-3 small model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrTrrr)deit3_small_patch16_384rrs r9rrXrr:c Vtdddddd}td d|it|fi|}|S) z DeiT-3 medium model @ 224x224 (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rirTrrr)deit3_medium_patch16_224rrs r9rrbs= s"Z^lpqJ  i idS]NhagNh iE Lr:c Vtdddddd}tdd|it|fi|}|S) z DeiT-3 base model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrTrrr)deit3_base_patch16_224rrs r9rrl= s"[_mqrJ  gj gDQ[Lf_eLf gE Lr:c Vtdddddd}tdd|it|fi|}|S) DeiT-3 base model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrTrrr)deit3_base_patch16_384rrs r9rrvrr:c Vtdddddd}tdd|it|fi|}|S) z DeiT-3 large model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rTrrr)deit3_large_patch16_224rrs r9rr= t2\`nrsJ  hz hTR\Mg`fMg hE Lr:c Vtdddddd}tdd|it|fi|}|S) z DeiT-3 large model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrTrrr)deit3_large_patch16_384rrs r9rrrr:c Vtdddddd}td d|it|fi|}|S) ri rTrrr)deit3_huge_patch14_224rrs r9rrs= t2\`nrsJ  gj gDQ[Lf_eLf gE Lr:rrrrrrr) deit3_small_patch16_224_in21ft1k deit3_small_patch16_384_in21ft1k!deit3_medium_patch16_224_in21ft1kdeit3_base_patch16_224_in21ft1kdeit3_base_patch16_384_in21ft1k deit3_large_patch16_224_in21ft1k deit3_large_patch16_384_in21ft1kdeit3_huge_patch14_224_in21ft1k)FF)rrk)-ro functoolsrtypingrrr&r timm.datarr timm.layersr timm.models.vision_transformerr r r _builderr _registryrrr__all__rrr default_cfgsrrrrrrrrrrrrrrrrrlr:r9rs ! A.aa*YY ' (`$!2`$F  %^&#T T&V^& %d U'W ^&$T T&V^&$T T 3&0^& .t ^(0*!^&(/ _(1*)^&0.t ^(0*1^&8.t ^ 3( 0*9^&D&t I(KE^&J&t I 3(0K^&R' J)LS^&X%d H'JY^&^%d H 3'0_^&f&t I(Kg^&l&t I 3(0m^&t%d H'Ju^&|/ J1}^&D15 J 31026 K204 I004 I 30015 J115 J 31004 L0u^&^ B9J:K9J9JC]D^C]C];L;L