^jfbdZddlZddlZddlmZmZmZmZmZm Z ddl Z ddl m Z ddl m cm ZddlmZmZddlmZddlmZej,eZGd d e j2ZGd d eZ d"d eedededefdZe j@Z!e!fdeeefd eeefdedede jDde jFde jHfdZ%e!fde jHde jHdeeefde jFde jFde jHf dZ& d"de jHd eedededef dZ'Gdde j2Z(Gd d!e j2Z)y)#a? Image to Patch Embedding using Conv2d A convolution based approach to patchifying a 2D image w/ embedding projection. Based on code in: * https://github.com/google-research/vision_transformer * https://github.com/google-research/big_vision/tree/main/big_vision Hacked together by / Copyright 2020 Ross Wightman N)CallableDictListOptionalTupleUnion)nn)Formatnchw_to) to_2tuple)_assertceZdZUdZeed<ejje ed< dde e e e e e ffde de de de ed e de ed e d e de ffd Zde e e e e fffd Z dde e e e e e ffde e e e e e fffdZdde e e e fe ffdZde e e fde e e ffdZdZxZS) PatchEmbed! 2D Image to Patch Embedding output_fmtdynamic_img_padimg_size patch_sizein_chans embed_dim norm_layerflattenbiasstrict_img_sizec | | d} t|t||_|j |\|_|_|_|d|_t||_ n||_tj|_ | |_ | |_ tj||f|||d| |_|r||fi| |_ytj"|_y)NdevicedtypeF) kernel_sizestrider)super__init__r r_init_img_sizer grid_size num_patchesrr rNCHWrrr Conv2dprojIdentitynorm)selfrrrrrrrrrrrrdd __class__s b/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/layers/patch_embed.pyr#zPatchEmbed.__init__ s / #J/:>:M:Mh:W7 t~t'7  ! DL$Z0DO#DL$kkDO..IIh nzR\cgnkmn 3=Jy/B/ 2;;= c|jsJ|yt|}tt||jDcgc] \}}||z c}}}|d|dz}|||fScc}}w)N)NNNrr )rr tuplezip)r,rspr%r&s r/r$zPatchEmbed._init_img_sizeAsn  #X&c(DOO.LMda16MN lYq\1 K//NsA" c nd}| t|}|Q||jk7rAtj5t j |j j|j j|||j jdu|j jj|j jj}|jjt|j j|d|j j/|jj|j j||_ddd||_|xs |j}||jk7s|%|j!|\|_|_|_yy#1swYWxYw)N)r r!rrrT)verbose)r rtorchno_gradr r(r) in_channels out_channelsrweightrrcopy_resample_patch_embedrr$r%r&)r,rrnew_patch_sizenew_projs r/set_input_sizezPatchEmbed.set_input_sizeJsQ   !&z2N  %.DOO*K %99II))II** .)t399++22))**00%%&:499;K;K^ei&jk99>>-MM'' 7$  %-DO,t}} t}} $(B>B>Q>QRZ>[ ;DM4>4+;)C! % %s DF++F4returncH|rt|jS|jSN)maxr)r, as_scalars r/ feat_ratiozPatchEmbed.feat_ratiofs t' '?? "r0c|jrPtj|d|jdz tj|d|jdz fS|d|jdz|d|jdzfS)z Get grid (feature) size for given image size taking account of dynamic padding. NOTE: must be torchscript compatible so using fixed tuple indexing rr )rmathceilr)r,rs r/dynamic_feat_sizezPatchEmbed.dynamic_feat_sizels   99Xa[4??1+==> (ST+X\XgXghiXjJj@kk kA;$//!"44hqkT__UVEW6WW Wr0c|j\}}}}|j|jrat||jdk(d|d|jddt||jdk(d|d|jddnr|jsft||j dzdk(d|d|j ddt||j dzdk(d|d|j dd|jr~|j d||j dzz |j dz}|j d||j dzz |j dz}t j|d|d|f}|j|}|jr"|jdjdd}n3|jtjk7rt||j}|j|}|S) NrzInput height (z) doesn't match model ().r z Input width (z%) should be divisible by patch size ()shaperrrrrFpadr)r transposerr r'r r+)r,xBCHWpad_hpad_ws r/forwardzPatchEmbed.forwardusWW 1a == $##T]]1--sBYZ^ZgZghiZjYkkm/noT]]1--qcAXY]YfYfghYiXjjl/mn))**a/$QC'LT__]^M_L``bc**a/#A3&KDOO\]L^K__ab   __Q'!dooa.@*@@DOOTUDVVE__Q'!dooa.@*@@DOOTUDVVEa!UAu-.A IIaL << ! &&q!,A __ +4??+A IIaLr0) NTNTTFNN)NN)T)__name__ __module__ __qualname____doc__r __annotations__r8jitFinalboolrrintrrstrr#r$rArGrKrZ __classcell__r.s@r/rrsYY__T**?B  -1 (,$($)QuS%S/%9:;QQ Q  Q !* QQ! QQ"Q"QB0uS%S/-A'B0?C@D\uS%S/%9:;\!sE#s(O';!<=\8#E%S/32F,G# X%S/XeCHoXr0rceZdZUdZeed< d deeee eeffdedededee de dee d e ffd Z d e ejeeffd ZxZS)PatchEmbedWithSizerrrrrrrrrc 8t |||||||||| |  y)N) rrrrrrrrrr)r"r#) r,rrrrrrrrrrr.s r/r#zPatchEmbedWithSize.__init__s5 !!!  r0rBc4|j\}}}}|jft||jdzdk(d|d|jddt||jdzdk(d|d|jdd|j |}|jdd}|j r"|j dj dd}n3|jtjk7rt||j}|j|}||fS) NrzInput image height (z#) must be divisible by patch size (rMr zInput image width (rN) rOrrrr)rrRrr r'r r+)r,rSrTrUrVrW feat_sizes r/rZzPatchEmbedWithSize.forwards+WW 1a == $ A**a/3GsJmnrn}n}~oAnBBD2E F A**a/3FqcIlmqm|m|}~mmAAC2D E IIaLGGBCL << ! &&q!,A __ +4??+A IIaL)|r0) r[r\r]r^NTNTNN)r_r`rarbr rcrrrgrrrfrhr#r8TensorrrZrirjs@r/rlrls?B  -1 (, uS%S/%9:;      !*   !   4 E%,,S "9: r0rlnew_size interpolation antialiasr7c  ddl ddlm}t |j dk(sJdt dk(sJd|j dd}t|tk(r|S|r)tjd |j d d d fd  fd}||}tj jj|j|jfd} ||| dddd} |j} |j!}| |}|j#| }|S#t$r ddlm}YAwxYw)a/Resample the weights of the patch embedding kernel to target resolution. We resample the patch embedding kernel by approximately inverting the effect of patch resizing. Code based on: https://github.com/google-research/big_vision/blob/b00544b81f8694488d5f36295aeb7972f3755ffe/big_vision/models/proj/flexi/vit.py With this resizing, we can for example load a B/8 filter into a B/16 model and, on 2x larger input image, the result will match. Args: patch_embed: original parameter to be resized. new_size (tuple(int, int): target shape (height, width)-only. interpolation (str): interpolation for resize antialias (bool): use anti-aliasing filter in resize verbose (bool): log operation Returns: Resized patch embedding kernel. rN)vmapzFour dimensions expectedrNzNew shape should only be hwrozResize patch embedding z to z, w/ z interpolation.ctj|d}tj||dj }|S)N)NN.)sizemodert)rr.)r8rqrP interpolatenumpy)x_np _new_sizex_tf x_upsampledrtrss r/resizez(resample_patch_embed_old..resizesI||D!/2mm y} KKTVV[V[V] r0cg}tj|D]O}j|}d|j||<|j ||j dQj |jS)Ng?)rangeprodzeros unravel_indexappendreshapestackT) _old_sizer~mati basis_vecnprs r/get_resize_matz0resample_patch_embed_old..get_resize_mats~rwwy)* AA+I8:Ib&&q)4 5 JJvi3;;B? @ Axx}r0)rcN|jdz}|jS)Nr)r)kernelresampled_kernelrrresize_mat_pinvs r/resample_kernelz1resample_patch_embed_old..resample_kernels(*V^^B-??''11r0r )r|r8rv ImportError functorchlenrOr2_loggerinfotensorlinalgpinvrrrfloatto) patch_embedrrrsrtr7rvold_sizer resize_matrv_resample_kernel orig_dtyperrrs ``` @@@r/resample_patch_embed_oldrsM4# {  !Q &B(BB & x=A <<<   %H X%/) .{/@/@.AhZuUbTccrst  (3Jll299>>*,,#? HZHZ[O2T/1a8!Q?""J##%K#K0K..,K M #""#sEEErrrrBc|\}}|\}} ||z} || z} tj| ||} | j| d||} tj| |||d}|j dj dddj| | }|S)zKComputes the resize matrix basis vectors and interpolates them to new_size.rr F)ryrzrt align_cornersrNr)r8eyerrPr{squeezepermute)rrrrsrtrrold_hold_wnew_hnew_w old_total new_total eye_matrixbasis_vectors_batchresized_basis_vectors_batch resize_matrixs r/_compute_resize_matrixr sLE5LE5 I I9V5AJ$,,Y5%H"#--   #077:BB1aKSST]_hiM r0r pinv_matrixnew_size_tuplerintermediate_dtypec|j^}}}|j||dj|}|j|}||z}|j||g|j|}|S)zW Simplified resampling w/o vmap use. As proposed by https://github.com/stas-sl r)r)rOrr) rrrrrc_outc_in_resampled_patch_embeds r/_apply_resamplingr%s"''OE4!%%eT2699@R9SK..'9.:K'+59199%WWZZakZl  r0c~t|jdk(sJdt|dk(sJdt|jdd}t|}||k(r|S|j}|j}t |||||t } tjj| } t|| ||t } | S)z5 Standalone function (computes matrix on each call). rwz/Input tensor should be 4D (out_ch, in_ch, h, w)rNz+New shape should only be hw (height, width)roN) rrOr2rrrDTYPE_INTERMEDIATEr8rrr) rrrrsrtr7old_size_tuplerrrrrrs r/r>r>7s {  !Q &Y(YY & x=A LLL &+K,=,=bc,B&CN&+HoN'   F""J' y&J\J,,##J/K-[.*>P ! r0c eZdZdZ d deeefdedeffd Ze fdeeefde jde jd e jfd Zd e jdeed e jfd ZxZS) PatchEmbedResamplerFixedOrigSizez Resample patch embedding weights from a fixed original size, caching the pseudoinverse matrix based on the target size. orig_sizersrtct|t|trt |dk(sJd||_||_||_i|_y)z Args: orig_size (Tuple[int, int]): The expected original (height, width) of input patch_embed tensors. interpolation (str): Interpolation mode. antialias (bool): Use anti-aliasing filter in resize. rNz.`orig_size` must be a tuple of (height, width)N) r"r# isinstancer2rrrsrt_pinv_cache_map)r,rrsrtr.s r/r#z)PatchEmbedResamplerFixedOrigSize.__init__ZsS )U+I!0C = < =C"*";=r0rrrrrBc|}|jj|}|r8t||r,t||}|j|k(r|j |k(r|St |j||j|j||}tjj|}d|dd|d}t||r t|||j||||j|<|S)zRRetrieves the cached pinv matrix or computes and caches it for the given new_size.pinv_rrSr )rgethasattrgetattrrrrrrsrtr8rrdelattrregister_buffer)r,rrrr cache_key buffer_namerrs r/_get_or_create_pinv_matrixz;PatchEmbedResamplerFixedOrigSize._get_or_create_pinv_matrixos **..y9 745!$ 4K!!V+ 0A0AU0J##, NNHd&8&8$..&RW ll'' 3 hqk]!HQK=9 4 % T; ' [+6*5Y'r0rc~t|jdk(sJt|dk(sJt|jdd}||jk(sJd|d|jt|}|j|k(r|S|j}|j }|j ||}t||||}|S)a Resamples the patch embedding weights to new_size. Args: patch_embed (torch.Tensor): Original weights (out_ch, in_ch, H_orig, W_orig). new_size (List[int]): Target [height, width]. Returns: torch.Tensor: Resampled weights. rwrNroNzInput patch_embed spatial size z0 does not match module's expected original size )rrOr2rrrrr) r,rrr input_sizerrrrrs r/rZz(PatchEmbedResamplerFixedOrigSize.forwards;$$%***8}!!!;,,RS12 T^^+ @-j\://3~~.> @ @++0/ >>^ + ## && 55nfM !2+{N\f g$$r0)bicubicT)r_r`rarbrrgrhrfr#rr8rrrqrrrZrirjs@r/rrUs"+" >S#X>> >2"4 CHoLL;;   <"%5<<"%49"%"%r0rc leZdZdZ ddeeefdededededef fd Zd e jd eeefd e jfd Z d e jd eeefd e jfd Z dde jde jde e jde eeefded e jf dZxZS)PatchEmbedInterpolatoraDynamically interpolates patch embedding weights for variable patch sizes. This module wraps patch embedding weight resampling functionality to support on-the-fly patch size variation during training. It handles both Conv2d and Linear patch embeddings. Args: base_patch_size: The original patch size the model was initialized with in_chans: Number of input channels embed_dim: Embedding dimension interpolation: Interpolation mode for resampling antialias: Whether to use antialiasing during interpolation channels_last: Per-patch flat layout of the linear weight / patches: True -> (ph, pw, C) [NaFlex default], False -> (C, ph, pw). base_patch_sizerrrsrt channels_lastcvt|||_||_||_||_||_||_yrD)r"r#rrrrsrtr)r,rrrrsrtrr.s r/r#zPatchEmbedInterpolator.__init__s= .  "*"*r0r<target_patch_sizerBc||jk(r|S|jd}|j\}}|\}}|jr1|j||||jj dddd}n|j||j||}t |||g|j|jd} |jr&| j ddddj|d} | S| j|d} | S)a/Resample linear patch embedding weights for a new patch size. Args: weight: Linear weight tensor of shape [embed_dim, patch_h * patch_w * in_chans] target_patch_size: Target (patch_h, patch_w) to resample to Returns: Resampled weight tensor rr]r rNFrrrsrtr7r) rrOrrrrr>rsrt) r,r<rrbase_phbase_pw target_ph target_pw weight_convweight_conv_resampledweight_resampleds r/resample_linear_weightz-PatchEmbedInterpolator.resample_linear_weights   4 4 4MLLO //0 9    ..GWdmmT\\]^`acdfghK ..DMM7GTK!5 +,,nn !    4<listrsrt)r,r<rrs r/resample_conv_weightz+PatchEmbedInterpolator.resample_conv_weightsK  4 4 4M0 +,,,nn   r0patches proj_weight proj_biasr is_linearc| |j}|r||jk7rz|jdk(sJd|jdd\}}|j||}|j ||d} t j jj| ||} | S|jdk(r-|j |jd|jdd}t j jj|||} | S||jk7rB|j||}t j jj||||d} | St j jj||||d} | S) aWApply patch embedding with dynamic weight resampling. Args: patches: Input patches - For linear mode with resampling: [B, N, Ph, Pw, C] - For linear mode without resampling: [B, N, Ph*Pw*C] - For conv mode: [B, C, H, W] proj_weight: Original projection weight proj_bias: Optional projection bias patch_size: Current patch size (if None, uses base_patch_size) is_linear: Whether using linear (True) or conv2d (False) projection Returns: Embedded patches NzKPatches must be 5-D ([B, N, Ph, Pw, C] or [B, N, C, Ph, Pw]) for resamplingrNrrr )r!padding) rndimrOrrr8r functionallinearrconv2d) r,rrrrrrTNr patches_flatoutputs r/rZzPatchEmbedInterpolator.forwards.  --J T111||q(bab(}}Ra(1$(#>#>{J#W  'q!R8 ,,33LBRT]^( #<<1$%oogmmA.> a@PRTUG,,33G[)T T111#'#<#<[*#U ,,33-y%q4 ,,33[)%q4  r0)r]r^rTT)NNT)r_r`rarbrrgrhrfr#r8rqrrrrZrirjs@r/rrs0& !*""&+"38_++ +  +  + +"+ LL+  %S#X+   + Z LL  %S#X    @1548" <\\<<  - < !sCx1 <  < <r0r)rTF)*rbloggingrItypingrrrrrrr8r torch.nn.functionalrrPformatr r helpersr trace_utilsr getLoggerr_rModulerrlrgrhrfrfloat32rrrrqrrr>rrr0r/rs  ?? # '  H %ttn,,f' Cs)CC C  CL]]0 S/S/    {{  \\@+= !\\!\\!c3h!KK ! "KK !  \\ !*' !\\!s)!! !  !