^jI>dZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmcmZddlmZmZddlmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#ddl$m%Z%dd l&m'Z'dd l(m)Z)dd l*m+Z+m,Z,d gZ-d5d e e.e.fdej^fdZ0GddejbZ2GddejbZ3GddejbZ4Gdd ejbZ5d6de6dee6effdZ7e+e7de7ddde7dde7de7ddde7dd de7dde7dee!e7dee!e7ddee"e7dd#ee$e7dd#ee$e7ddee"d% Z8d7d&ee6ej^fd'ejbd(e6d)e9dee6ej^ff d*Z:d8d+e6d,e9de5fd-Z;e,d8d,e9de5fd.Ze,d8d,e9de5fd1Z?e,d8d,e9de5fd2Z@e,d8d,e9de5fd3ZAe,d8d,e9de5fd4ZBy)9a BEiT: BERT Pre-Training of Image Transformers (https://arxiv.org/abs/2106.08254) Model from official source: https://github.com/microsoft/unilm/tree/master/beit @inproceedings{beit, title={{BEiT}: {BERT} Pre-Training of Image Transformers}, author={Hangbo Bao and Li Dong and Songhao Piao and Furu Wei}, booktitle={International Conference on Learning Representations}, year={2022}, url={https://openreview.net/forum?id=p-BhZSz59o4} } BEiT-v2 from https://github.com/microsoft/unilm/tree/master/beit2 @article{beitv2, title={{BEiT v2}: Masked Image Modeling with Vector-Quantized Visual Tokenizers}, author={Zhiliang Peng and Li Dong and Hangbo Bao and Qixiang Ye and Furu Wei}, year={2022}, eprint={2208.06366}, archivePrefix={arXiv}, primaryClass={cs.CV} } At this point only the 1k fine-tuned classification weights and model configs have been added, see original source above for pre-training models and procedure. Modifications by / Copyright 2021 Ross Wightman, original copyrights below N)partial) AnyCallableDictListOptionalSetTupleTypeUnion)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD) PatchEmbedMlpSwiGLU LayerNormDropPathcalculate_drop_path_rates trunc_normal_use_fused_attnresample_patch_embedresample_abs_pos_embedresize_rel_pos_bias_tablendgrid)build_model_with_cfg)feature_take_indices) checkpoint)generate_default_cfgsregister_modelBeit window_sizereturnc 2d|dzdz d|dzdz zdz}|d|dz}tjttj|d|tjtj|d|tj}tj |d}|dddddf|dddddfz }|j dddj}|dddddfxx|ddz z cc<|dddddfxx|ddz z cc<|dddddfxxd|dzdz zcc<tj|dzfdz||j}|jd|ddddf<|dz |dddf<|dz |dddf<|dz |d <|S) aGenerate relative position index for window-based attention. Creates a lookup table for relative position indices between all pairs of positions within a window, including special handling for cls token interactions. Args: window_size: Height and width of the attention window. Returns: Relative position index tensor of shape (window_area+1, window_area+1) where +1 accounts for the cls token. rrdevicedtypeN)sizer(r))rr) torchstackrarangelongflattenpermute contiguouszerosr)sum)r"r(num_relative_distance window_areacoordscoords_flattenrelative_coordsrelative_position_indexs [/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/models/beit.pygen_relative_position_indexr<IsQ/!3KN8JQ8NORSSa.;q>1K [[ [^F%**E [^F%**EF]]61-N$Q4Z0>!T1*3MMO%--aA6AACOAq!G A 22Aq!G A 22Aq!GKN 2Q 66#kk a/AA/Ef\k\q\qr&5&9&9"&=ABF#%:Q%>AqrE"%:Q%>ABE"$9A$=D! ""c8eZdZUdZej j eed< dde de dedede de d e e e e fd e e ffd Z d ejfd Zddejde ejd ejfdZddZddZddZxZS) AttentionzMulti-head attention module with optional relative position bias. Implements multi-head self-attention with support for relative position bias and fused attention operations. Can use either standard or custom head dimensions. fused_attndim num_headsqkv_biasqkv_bias_separate attn_drop proj_dropr" attn_head_dimc h| | d} t|||_||z} ||} | |jz} | dz|_t |_||_tj|| dzfddi| |_ |rtjtj| fi| |_ |jdtj| fi| dtjtj| fi| |_nd|_ d|_d|_|r||_d |d zd z d |d zd z zdz|_|d |d z}tjtj|j$|fi| |_|jd tj|d z|d zf| tj(dnd|_d|_d|_tj,||_tj| |fi| |_tj,||_|j5y) akInitialize attention module. Args: dim: Input feature dimension. num_heads: Number of attention heads. qkv_bias: If True, add learnable bias to query, key, value projections. qkv_bias_separate: If True, use separate bias for q, k, v projections. attn_drop: Dropout rate for attention weights. proj_drop: Dropout rate for output projection. window_size: Window size for relative position bias. If None, no relative position bias. attn_head_dim: Dimension per attention head. If None, uses dim // num_heads. r'Ngr&biasFk_bias persistentr%rrr:)super__init__rBscalerr@rDnnLinearqkv Parameterr,emptyq_biasregister_bufferv_biasrJr"r5relative_position_bias_tabler/r:DropoutrEprojrFreset_parameters)selfrArBrCrDrErFr"rGr(r)ddhead_dim all_head_dimr6 __class__s r;rNzAttention.__init__ts2 / ")#  $$H$..0 % (*!299S,"2EE"E ,,u{{<'F2'FGDK  5;;|+Jr+JW\ ],,u{{<'F2'FGDKDKDKDK *D *+k!n*9K02  D66 HRH1JD -  ) [1_kAo>vUZU_U_`  !  $D 04D -+/D (I.IIlC626 I. r=r#cX|j|jjdj|jd|jdzdz|jd|jdzdzd}|j dddj }|j dS)zGet relative position bias for the attention window. Returns: Relative position bias tensor of shape (1, num_heads, window_area+1, window_area+1). r+rrr%)rXr:viewr"r1r2 unsqueezer\relative_position_biass r;_get_rel_pos_biaszAttention._get_rel_pos_biass "&!B!B  ( ( - -b 1"3374   Q $"2"21"5 5 9   Q $"2"21"5 5 924? "8!?!?1a!H!S!S!U%//22r=xshared_rel_pos_biascN|j\}}}|j|j|}ntj|j|j |j f}|jr|j|}||z }n,tj||jj|}|j||d|jdjddddd}|jd\}} } |jrgd} |j |j#} | | |z} n||} tj$|| | | |j&r|j(j*nd }nr||j,z}|| j/d dz} |j | |j#z} || |z} | j1d } |j)| } | | z}|j/ddj|||}|j3|}|j5|}|S) a-Forward pass of attention module. Args: x: Input tensor of shape (batch_size, num_tokens, dim). shared_rel_pos_bias: Optional shared relative position bias from parent module. Returns: Output tensor of shape (batch_size, num_tokens, dim). N)weightrIr&r+r%rr) attn_mask dropout_prA)shaperUrRr,catrJrWrDFlinearrjreshaperBr1unbindr@rXrfscaled_dot_product_attentiontrainingrEprO transposesoftmaxrZrF) r\rgrhBNCrRrCqkv rel_pos_biasattns r;forwardzAttention.forwards''1a ;; ((1+Cyy$++t{{DKK!HIH%%hhqkxhhqxHkk!Q4>>26>>q!Q1M**Q-1a ??L00<#557 &2#/2E#EL$02 ..1a&.2mm$..**A DJJA B++D00<d4466".11<>$'DqA KK1  % %aA . IIaL NN1 r=cJ|jRtjj|jtjj|j|j )tjj|j |j yz"Initialize parameters and buffers.N)rUrPinitzeros_rWrX _init_buffersr\s r;r[zAttention.reset_parameterssb ;; " GGNN4;; ' GGNN4;; '  , , 8 GGNN4<< = r=c|j|jj|jE|jjt |j |jj yy).Compute and fill non-persistent buffer values.Nr()rJzero_r:copy_r<r"r(rs r;rzAttention._init_bufferss] ;; " KK     ' ' 3  ( ( . .+D,<,eZdZdZddddddddej eddddfdededed e d ed ed e d e de de e de ejde ejde e eefde effd ZddZddej"de ej"dej"fdZxZS)BlockzTransformer block with attention and MLP. Standard transformer block consisting of multi-head self-attention and MLP with residual connections and layer normalization. Supports layer scale and stochastic depth regularization. F@rlNrArBrC mlp_ratio scale_mlp swiglu_mlprFrE drop_path init_values act_layer norm_layerr"rGc ||d}t|| |fi||_t|f||||| |d||_| dkDr t | nt j|_| |fi||_ |r&td|t||z|r| nd|d||_ n&td|t||z| |r| nd|d||_ | dkDr t | nt j|_| |_| r]t j t#j$|fi||_t j t#j$|fi||_nd\|_|_|j+y) aInitialize transformer block. Args: dim: Input feature dimension. num_heads: Number of attention heads. qkv_bias: If True, add learnable bias to query, key, value projections. mlp_ratio: Ratio of MLP hidden dimension to input dimension. scale_mlp: If True, apply layer normalization in MLP. swiglu_mlp: If True, use SwiGLU activation in MLP. proj_drop: Dropout rate for projections. attn_drop: Dropout rate for attention. drop_path: Drop path rate for stochastic depth. init_values: Initial values for layer scale. If None, no layer scale. act_layer: Activation function class. norm_layer: Normalization layer class. window_size: Window size for relative position bias in attention. attn_head_dim: Dimension per attention head. r')rBrCrErFr"rGrlN) in_featureshidden_featuresrdrop)rrrrrNN)rMrNnorm1r?rrrPIdentity drop_path1norm2rrmlpr drop_path2rrSr,rTgamma_1gamma_2r[)r\rArBrCrrrrFrErrrrr"rGr(r)r]r`s r;rNzBlock.__init__sxJ / *r*    #'     2;R(9-R[[]*r*  #C)O 4)2:   DH #C)O 4#)2:   DH2;R(9-R[[]& << C(>2(>?DL<< C(>2(>?DL)3 &DL$, r=r#c|jitjj|j|jtjj|j |jyy)zInitialize parameters.N)rrPr constant_rrrs r;r[zBlock.reset_parameterspsM << # GG  dllD,<,< = GG  dllD,<,< = $r=rgrhc |jh||j|j|j||z}||j |j |j |z}|S||j|j|j|j||zz}||j |j|j |j |zz}|S)aForward pass of transformer block. Args: x: Input tensor of shape (batch_size, num_tokens, dim). shared_rel_pos_bias: Optional shared relative position bias. Returns: Output tensor of shape (batch_size, num_tokens, dim). rh)rrrrrrrr)r\rgrhs r;rz Block.forwardvs << DOODIIdjjmQdI$effADOODHHTZZ]$;< HU\\KN:!"[^!3a!7A AZ!d4ej<d(ej<fd?Z"dEd4ej<d@e d(ej<fdAZ#d4ej<d(ej<fdBZ$xZ%S)Ir!aBEiT: BERT Pre-Training of Image Transformers. Vision Transformer model with support for relative position bias and shared relative position bias across layers. Implements both BEiT v1 and v2 architectures with flexible configuration options. r&avg TrFrlNgMbP?img_size patch_sizein_chans num_classes global_pool embed_dimdepthrBrCrrr drop_rate pos_drop_rateproj_drop_rateattn_drop_ratedrop_path_raterruse_abs_pos_embuse_rel_pos_biasuse_shared_rel_pos_biashead_init_scalec||d}t |||_||_||_|x|_x|_|_d|_d|_ td||||d||_ |jj}t|jdr|jjn|}tj t#j$dd|fi||_|r.tj t#j$d|dz|fi|nd|_tj*||_|r(t/d|jj0|d||_nd|_t5||}tj6t9|Dcgc]5}t;d||| | | | |||||||r|jj0ndd |7c}|_t9|Dcgc]}t?d ||| c}|_ |jd k(}|rtjBn||fi||_"|r ||fi|ntjB|_#tj*| |_$|d kDrtjJ||fi|ntjB|_&||_'|jQdycc}wcc}w)aInitialize BEiT model. Args: img_size: Input image size. patch_size: Patch size for patch embedding. in_chans: Number of input image channels. num_classes: Number of classes for classification head. global_pool: Type of global pooling ('avg' or ''). embed_dim: Embedding dimension. depth: Number of transformer blocks. num_heads: Number of attention heads. qkv_bias: If True, add learnable bias to query, key, value projections. mlp_ratio: Ratio of MLP hidden dimension to embedding dimension. swiglu_mlp: If True, use SwiGLU activation in MLP. scale_mlp: If True, apply layer normalization in MLP. drop_rate: Dropout rate. pos_drop_rate: Dropout rate for position embeddings. proj_drop_rate: Dropout rate for projections. attn_drop_rate: Dropout rate for attention. drop_path_rate: Stochastic depth rate. norm_layer: Normalization layer class. init_values: Initial values for layer scale. use_abs_pos_emb: If True, use absolute position embeddings. use_rel_pos_bias: If True, use relative position bias in attention. use_shared_rel_pos_bias: If True, share relative position bias across layers. head_init_scale: Scale factor for head initialization. r'rF)rrrr feat_ratioN)ry)r"rB) rArBrCrrrrFrErrrr"zblocks.)modulenum_chs reductionrr needs_resetr))rMrNrrr num_featureshead_hidden_sizernum_prefix_tokensgrad_checkpointingr patch_embed num_patcheshasattrrrPrSr,rT cls_token pos_embedrYpos_dropr grid_sizerr ModuleListrangerblocksdict feature_infornormfc_norm head_droprQheadr init_weights)!r\rrrrrrrrBrCrrrrrrrrrrrrrrr(r)r]rrdpri use_fc_normr`s! r;rNz Beit.__init__sn / &  &ENNND1DN!""'% !    &&22 -4T5E5E|-TD   ' ' )Zdekk!Q &HR&HI[jekk![1_i&VSU&VWpt ]3 " 4! ,,66#!!D  !%D '>mm 5\!%#   #!##%((a&%':JD,,66PT  %#$ $QVV[P\^KLD'! yA F^&&%/ %0BKKMj6Qb6Q 6Az)2r2r{{} I.?JQBIIi;;TVT_T_Ta . e,9%#"^s :J8J=rr#c|jt|j||jt |jdt |j d|j |jrt|jtjrt |jjdtj5|jjj|j|jj j|jdddyyy#1swYyxYw)zInitialize model weights. Args: needs_reset: If True, call reset_parameters() on modules that have it. Set to False when modules have already self-initialized in __init__. rN{Gz?std)applyr _init_weightsrrrfix_init_weightr isinstancerrPrQrjr,no_gradmul_rI)r\rs r;rzBeit.init_weights?s 74--;GH >> % $..c 2dnn#.    Jtyy"))$D $))** 4 :   %%d&:&:; ##D$8$89 : :%E  : :s AEE ctj5t|jD]~\}}t j d|dzz}|j jjj||jjjj| dddy#1swYyxYw)zFix initialization weights according to BEiT paper. Rescales attention and MLP weights based on layer depth to improve training stability. g@rN) r,r  enumeratermathsqrtrrZrjdiv_rfc2)r\layer_idlayerrOs r;rzBeit.fix_init_weightSs ]]_ 1#,T[[#9 1% #A"67 &&++E2 $$))%0 1 1 1 1s BB66B?mct|tjrOt|jd|j +tj j|j dyy|rt|dr|jyyy)zInitialize model weights. Args: m: Module to initialize. needs_reset: If True, call reset_parameters() on modules that have it. rrNrr[) rrPrQrrjrIrrrr[)r\rrs r;rzBeit._init_weights_sg a # !(( ,vv!!!!&&!," WQ(:;   <[r=cjddh}|jD]\}}d|vs |j||S)zGet parameter names that should not use weight decay. Returns: Set of parameter names to exclude from weight decay. rrrX)named_parametersadd)r\nwdn_s r;no_weight_decayzBeit.no_weight_decaymsDK())+ DAq-2   r=enablec||_y)z}Enable or disable gradient checkpointing. Args: enable: If True, enable gradient checkpointing. N)r)r\rs r;set_grad_checkpointingzBeit.set_grad_checkpointingzs #)r=coarsec$tdddg}|S)zCreate parameter group matcher for optimizer parameter groups. Args: coarse: If True, use coarse grouping. Returns: Dictionary mapping group names to regex patterns. z-^cls_token|pos_embed|patch_embed|rel_pos_bias)z^blocks\.(\d+)N)z^norm)i)stemr)r)r\rmatchers r; group_matcherzBeit.group_matchers!A-/CD r=c|jS)z_Get the classifier head. Returns: The classification head module. )rrs r;get_classifierzBeit.get_classifiersyyr=c||_|||_|dkDr&tj|j||_ytj |_y)zReset the classification head. Args: num_classes: Number of classes for new head. global_pool: Global pooling type. Nr)rrrPrQrrr)r\rrs r;reset_classifierzBeit.reset_classifiersF'  "*D >IAoBIIdnnk: SUS^S^S` r=rgindicesreturn_prefix_tokensr stop_early output_fmtintermediates_onlyc `|dvsJd|dk(}g} tt|j|\} } |j\} } }}|j |}t j |jj|jddd|fd}|j||jz}|j|}|j|jnd}t jjs|s |j}n|jd| dz}t|D]q\}}|jr-t jjst!||| }n ||| }|| vsN| j#|r|j%|n|s|j&rD| Dcgc]}|ddd|j&f}}| Dcgc]}|dd|j&df} }|ra|jj)||f\}}| Dcgc]6}|j+| ||dj-dd dd j/8} }t jjs|rt1t3| } |r| S|j%|}|| fScc}wcc}wcc}w) a Forward pass that returns intermediate feature maps. Args: x: Input image tensor of shape (batch_size, channels, height, width). indices: Block indices to return features from. If int, returns last n blocks. return_prefix_tokens: If True, return both prefix and spatial tokens. norm: If True, apply normalization to intermediate features. stop_early: If True, stop at last selected intermediate. output_fmt: Output format ('NCHW' or 'NLC'). intermediates_only: If True, only return intermediate features. Returns: If intermediates_only is True, returns list of intermediate tensors. Otherwise, returns tuple of (final_features, intermediates). )NCHWNLCz)Output format must be one of NCHW or NLC.r.rr+rrpNrr&r%)rlenrrqrr,rrrexpandrrrr is_scriptingr rrappendrrdynamic_feat_sizerur1r2listzip)r\rgr(r)rr*r+r,ru intermediates take_indices max_indexr|rheightwidthrrrblky prefix_tokensHWs r;forward_intermediateszBeit.forward_intermediatess2_,Y.YY,& "6s4;;7G"Q i gg1fe   Q  IIt~~,,QWWQZR@!D! L >> %DNN"A MM! .2.?.?.Kt((*QU 99 ! ! #:[[F[[)a-0F' BFAs&&uyy/E/E/GsA<H|<L $$TTYYq\qA B  ! !ERSQq!D$:$:"::;SMSDQRqQq$"8"8"99:RMR ##55vuoFDAq^klYZQYYq!Q3;;Aq!QGRRTlMlyy%%',@ ]M!BCM  IIaL-TRms0J!J&;J+ prune_norm prune_headctt|j|\}}|jd|dz|_|rtj|_|r+tj|_|jdd|S)a=Prune layers not required for specified intermediate outputs. Args: indices: Indices of blocks to keep. prune_norm: If True, remove final normalization. prune_head: If True, remove classification head. Returns: List of indices that were kept. Nrr)rr0rrPrrrr')r\r(rBrCr8r9s r;prune_intermediate_layerszBeit.prune_intermediate_layerssj #7s4;;7G"Q ikk.9q=1  DI ;;=DL  ! !!R (r=c|j|}tj|jj |j ddd|fd}|j ||j z}|j|}|j|jnd}|jD]E}|jr-tjjst|||}<|||}G|j|}|S)zForward pass through feature extraction layers. Args: x: Input tensor of shape (batch_size, channels, height, width). Returns: Feature tensor of shape (batch_size, num_tokens, embed_dim). rr+rrpNr)rr,rrrr1rqrrrrrrr2rr)r\rgrr<s r;forward_featureszBeit.forward_features s   Q  IIt~~,,QWWQZR@!D! L >> %DNN"A MM! .2.?.?.Kt((*QU ;; =C&&uyy/E/E/GsA<H|<  = IIaLr= pre_logitsc|jr=|jdk(r%|dd|jdfjdn|dddf}|j|}|j |}|r|S|j |S)a7Forward pass through classification head. Args: x: Feature tensor of shape (batch_size, num_tokens, embed_dim). pre_logits: If True, return features before final linear layer. Returns: Logits tensor of shape (batch_size, num_classes) or pre-logits. rNrrpr)rrmeanrrr)r\rgrIs r; forward_headzBeit.forward_head!s   =A=M=MQV=V!T++,,-22q29\]^_ab^b\cA LLO NN1 q0DIIaL0r=cJ|j|}|j|}|S)zForward pass through the model. Args: x: Input tensor of shape (batch_size, channels, height, width). Returns: Logits tensor of shape (batch_size, num_classes). )rHrL)r\rgs r;rz Beit.forward1s)  ! !! $   a r=)TrFr)NFFFr.F)rFT)&rrrrrr rr strrrr rPrrrNrrrr,rignorer rrrrr#r%r'rrrArFrHrLrrrs@r;r!r!s5868#$ !!$#!#%$&$&$&*3+/$(%*,1%*5u-CsCx01u-c5c?23u- u-  u-  u-u-u-u-u-u-u-u-u-!u- "!u-""#u-$"%u-&RYY'u-("%)u-*"+u-,#-u-.&*/u-0#1u-n:::( 1 !ryy !t ! YY S   YY)T)) YY D T#s(^   YY  aC ahsm a8<).$$',F ||F eCcN34F #' F  F  F F !%F  tELL!5tELL7I)I#JJ KF T./$# 3S >*  c 2%,,5<<01ell111  %,, r=urlc $|dddddddddd d d |S) zCreate a default configuration dictionary for BEiT models. Args: url: Model weights URL. **kwargs: Additional configuration parameters. Returns: Configuration dictionary. r)r&rrNg?bicubicT)?rTrTzpatch_embed.projrz apache-2.0) rQr input_size pool_sizecrop_pct interpolationfixed_input_sizerKr first_conv classifierlicenser)rQkwargss r;_cfgr^?s6=t(    r=ztimm/) hf_hub_id)r&r`g?)r_rUrWiQU)r_r)r&ra)r_rKr)r_rrKrgffffff?)r_rWrKr) z)beit_base_patch16_224.in22k_ft_in22k_in1kz)beit_base_patch16_384.in22k_ft_in22k_in1kz$beit_base_patch16_224.in22k_ft_in22kz*beit_large_patch16_224.in22k_ft_in22k_in1kz*beit_large_patch16_384.in22k_ft_in22k_in1kz*beit_large_patch16_512.in22k_ft_in22k_in1kz%beit_large_patch16_224.in22k_ft_in22kz*beitv2_base_patch16_224.in1k_ft_in22k_in1kz$beitv2_base_patch16_224.in1k_ft_in1kz%beitv2_base_patch16_224.in1k_ft_in22kz+beitv2_large_patch16_224.in1k_ft_in22k_in1kz%beitv2_large_patch16_224.in1k_ft_in1kz&beitv2_large_patch16_224.in1k_ft_in22k state_dictmodelrX antialiasc @|jd|}|jd|}i}|jD]c\}}d|vr d|vrf|jjjj \}}} } |j d| k7s|j d| k7rt || | f||d}n|d k(rQ|j d |jj d k7r(d } t||jj| ||d }n|jd r|j|d d} |j | jj k7s| jd| jd k7r,t|| j| jj }|||<f|S)aFilter and process checkpoint state dict for loading. Handles resizing of patch embeddings, position embeddings, and relative position bias tables when model size differs from checkpoint. Args: state_dict: Checkpoint state dictionary. model: Target model to load weights into. interpolation: Interpolation method for resizing. antialias: If True, use antialiasing when resizing. Returns: Filtered state dictionary. rcrr:zpatch_embed.proj.weightr+roT)rXrdverboserr)new_sizerrXrdrfrXNir)new_window_sizenew_bias_shape)getitemsrrZrjrqrrrrendswith get_submodulerXr"r) rbrcrXrdout_dictrrOIr?r@rrs r;checkpoint_filter_fnrqs4J*5JH  " 1 $ )  $ )**//66<>>!--PQBRVWVcVcdeVfBf-$%MM#$#A#A#G#G  A B Or=variant pretrainedc r|jdd}tt||ftt |dd|}|S)zCreate a BEiT model. Args: variant: Model variant name. pretrained: If True, load pretrained weights. **kwargs: Additional model arguments. Returns: BEiT model instance. out_indicesr&getter)ru feature_cls)pretrained_filter_fn feature_cfg)poprr!rqr)rrrsr]rurcs r; _create_beitr{sH**]A.K  gz 1[hG    E Lr=c Ztdddddddd}td d |it|fi|}|S) z0BEiT base model @ 224x224 with patch size 16x16.rrrrkFT皙?rrrrBrrrrrs)beit_base_patch16_224rr{rsr] model_argsrcs r;rrsFB"#GJ  fZ f4PZKe^dKe fE Lr=c Ztdddddddd}td d |it|fi|}|S) z0BEiT base model @ 384x384 with patch size 16x16.r`rrrFTr}rrrrrBrrrrs)beit_base_patch16_384rrs r;rrsFs"#GJ  fZ f4PZKe^dKe fE Lr=c Xtddddddd}td d|it|fi|}|S) z1BEiT large model @ 224x224 with patch size 16x16.rFTh㈵>rrrrBrrrrs)beit_large_patch16_224rrs r;rrsDR2$HJ  gj gDQ[Lf_eLf gE Lr=c Ztdddddddd}td d |it|fi|}|S) z1BEiT large model @ 384x384 with patch size 16x16.r`rrrFTrrrs)beit_large_patch16_384rrs r;rrFt2$HJ  gj gDQ[Lf_eLf gE Lr=c Ztdddddddd}td d |it|fi|}|S) z1BEiT large model @ 512x512 with patch size 16x16.rarrrFTrrrs)beit_large_patch16_512rrs r;rr rr=c Ztdddddddd}td d |it|fi|}|S) z3BEiT v2 base model @ 224x224 with patch size 16x16.rrrrkFTrr~rs)beitv2_base_patch16_224rrs r;rrsFB"$HJ  hz hTR\Mg`fMg hE Lr=c Xtddddddd}td d|it|fi|}|S) z4BEiT v2 large model @ 224x224 with patch size 16x16.rrrFTrrrs)beitv2_large_patch16_224rrs r;rrsDR2$HJ  i idS]NhagNh iE Lr=r)rE)rSTrN)Crr  functoolsrtypingrrrrrr r r r r,torch.nnrPtorch.nn.functional functionalrs timm.datar r timm.layersrrrrrrrrrrrr_builderr _featuresr _manipulater _registryrr __all__rrr<rr?rrr!rOr^ default_cfgsrrqr{rrrrrrrrr=r;rsNP OOO A    ++#< ( #U38_ #ell #Ff fRqBIIqh4D2994Dn|299|~ cT#s(^*%15226 32 -1- 37337 33 37 33 .2. 37 "(<3 -1 "(<- .2 5;O. 4817K4 .217K. /3 5;O/u?&? D5T#u||*;%<5RYY5_b5{5LPQTV[VbVbQbLc5p#4d*ddt$t$t$4Dr=