^jdZddlZddlZddlmZmZmZddlmZddl m Z m Z m Z m Z mZmZmZmZmZddlZddlmZddlmcmZddlmZmZddlmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(dd l)m*Z*dd l+m,Z,dd l-m.Z.m/Z/dd l0m1Z1m2Z2dd l3m4Z4m5Z5ddl6m7Z7ddl8m9Z9m:Z:ddgZ;ejxe=Z>eGddZ?de?de?fdZ@ dHdejdeeBeBfdeCdeCdeejeeBeBfff dZDdejfdZEGddZFde?de fd ZGe/Gd!d"ejZIe.dddejfd#ejd$eBd%eCd&e eBd'ejde ejf d(ZLe. dIdejd#e ejd)eMd$eBd*eCdejf d+ZNGd,dejZOd-ZPdJd.eMd/eQd0eCde fd1ZRd2e eMefd3eOde eMeffd4ZSdKd5eMde eMeffd6ZTe5eTd78eTd78eTd78eTeTeTeTd7d9eTd7d9d:ZUdLd;eMdZW dLd;eMd<dZ8eeed?<dZ9eeed@<dZ:eeedA<dZ;eeedB<dZeedF<d Z?eedG<d Z@eedH<d ZAeedI<d ZBeedJ<y)Kr+zConfiguration for FlexVit model. This dataclass contains the bulk of model configuration parameters, with core parameters (img_size, in_chans, num_classes, etc.) remaining as direct constructor arguments for API compatibility.  patch_size embed_dim depth num_headsg@ mlp_ratioFscale_mlp_normTqkv_biasqk_norm proj_biasattn_drop_ratescale_attn_inner_normN init_values drop_rate pos_drop_ratepatch_drop_rateproj_drop_ratedrop_path_rate class_tokenr reg_tokenslearned pos_embed)r.r.pos_embed_grid_sizebicubicpos_embed_interp_modepos_embed_ar_preservingpos_embed_use_grid_sample rope_type@rope_temperaturerope_ref_feat_shaperope_grid_offsetijrope_grid_indexingrope_rotate_halfrope_mrope_sectiondynamic_img_padpatchify_channels_lastpre_norm final_normfc_normmap global_poolpool_include_prefixattn_pool_num_headsattn_pool_mlp_ratio weight_initfix_initlinearembed_proj_typeinput_norm_layerembed_norm_layer norm_layer act_layerblock_fn mlp_layer attn_layerstandard attn_type attn_gated swiglu_mlp qkv_fusedenable_patch_interpolator)C__name__ __module__ __qualname____doc__r/rintr __annotations__r1r3r4r5floatr6boolr7r8r9r;r<r=r r>r?r@rArBrCrDrFstrrGrIrJrKrMrOrPrQrSrTrUrVrWrXrYrZr\r]r^r_r`rarcrdrerfrgrhrirjrlrmrnrorp`/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/models/naflexvit.pyr+r+:s/1Jc5c?*+0IsE3OIsIu ND HdGTItNE"'4'$(K%'IuM5 OU NENEKJIs5=%S/2=!*3*$)T)&+t+Is%e%59%S/29 e """d"9=sC}!56="OT!#'D'HdJ"GXd^"K %%)-#-+/%/KHd$OS#&*hsm*&*hsm*!%J $#Ix}#"Hhsm"#Ix}# $J $ IsJJIt',t+r{cfgreturnc t|jj}|jDcic] \}}||vs ||}}}|r t |fi|}|Scc}}w)zIOverlay kwargs onto config, replacing config values with provided kwargs.)set__dataclass_fields__keysitemsr)r}kwargs config_fieldskv config_kwargss r|_overlay_kwargsrsb005578M&,llnKda]8JQTKMKc+]+ J Ls AATxr/pad channels_lastc|j\}}}}|\}} |rQ||zdk7s|| zdk7rA|||zz |z} | || zz | z} tj|d| d| f}|j\} } }}||z|| z}} |r+|j||| ||| j dddddd}n*|j||| ||| j dddddd}|j || |z|| z|z}|| |ffS)uPatchify a batch of images. Args: x: Input tensor of shape ``[B, C, H, W]``. patch_size: Patch dimensions ``(patch_h, patch_w)``. pad: Whether to pad images to be divisible by patch size. channels_last: Per-patch flat layout. ``True`` (default) yields ``P-P-C`` flat (channel index varies fastest); ``False`` yields ``C-P-P`` flat (channel index varies slowest — HF / Gemma4 native layout). Returns: Tuple of ``(patches, grid_size)`` where ``patches`` has shape ``[B, N, P*P*C]`` and ``grid_size`` is ``(num_patches_h, num_patches_w)``. Grid order within ``N`` is row-major over ``(y, x)`` regardless of ``channels_last``. rr)shapeFrviewpermutereshape)rr/rrBCHWphpwpad_hpad_w_nhnwpatchess r|batch_patchifyrs"*JAq!Q FB B! q2v{a"f"a"f" EE!a5) *WW 1a "Wa2gB&&Ar2r2.66q!Q1aH&&Ar2r2.66q!Q1aHooab"r'A+6G RH r{_coordc (|dddddfjddz}|dddddfjddz}t||Dcgc]7\}}t|jt|jf9c}}Scc}}w)Nrrdim)amaxzipruitem)rmax_ymax_xhws r|calculate_naflex_grid_sizesrs~ 1a7O Q '! +E 1a7O Q '! +E7:5%7H Itq!S]CM * II IsI$..Y.?J3=D % %i 0 >r{cd|_|S)Nr)rrs r|__iter__zNaFlexRopeIterator.__iter__s r{c |j|jk\rttj|j |j |j|j|j|j}|jD]Y}|\}}||z}|j|}|j||j}|D]}|ddd|ddf||ddd|ddf<[|xjdz c_|S)Nrrr)rr3 StopIterationtorchzerosrr4rrrrrrr) r batch_embedrrr actual_len batch_indicesembedbis r|__next__zNaFlexRopeIterator.__next__s ??djj ( kk OOT^^T\\4==**T[[ ** NIDAqQJ 00;M--i8IE$ N5:1kzk1;L5M B;J;12 N N 1r{N)rqrrrsrtrr rur rrrrrrrrzr{r|rrsb>eCHotCy89>5c?+ >  >  > >{{>:r{rc |jdvxs*|jdvxs|jxs |j}|r|j}|dk(r|jdvrd}|jrdnd|j z}t t||j|j|j|j||jxs|jdk(|j S|jxst}i}|js |jr|j|d <|j|d <|jr|j|d <|r t |fi|}|S) zGet appropriate block function based on configuration. Returns a partially applied block constructor with EVA-specific or conflicting parameters pre-configured if needed. )evar)rLnonerkrrrmrope)rlrn scale_mlpscale_attn_innerronum_prefix_tokens rotate_half gated_attnr6scale_attn_normrj)rlrMrnrmrCrDrr(r6r<rorTrhr)rj)r}use_eva_featuresrlrrh block_kwargss r| get_block_fnrsE (  \)     MM  "s}}L'HI"%//QqCNNJ ~~(( 66mm/,,H 0H~~  <<(5   !:!:-0-?-?L) *.1.G.GL* + >>),L & x8<8Hr{c+eZdZdZ d*deeeeeffdededeede de d ed e d eeeeeeffd ed eeeefdede de dee e jdee ee e jfdee e jde de de ddf*fd Zd+dZdeeeffdZd,de deeeeefffdZdeeefdeeeffdZedej.d ej.ddfd!Zedej.d ej.ddfd"Zdej.d#eeddfd$Zedej.d ej.ddfd%Zedej.d ej.ddfd&Zdej.d#eeddfd'Z d-dej.d eej.d(eej.deej.eeeefffd)ZxZ S). NaFlexEmbedsaNaFlex Embedding module for Vision Transformers. This module encapsulates the complete embedding process for Vision Transformers, supporting both standard and NaFlex (NaViT + FlexiViT) functionality: 1. Patch embedding (via Conv2d or Linear) 2. Class and register token preparation 3. Position embedding addition with interpolation support 4. Pre-normalization (if requested) 5. Dropout application NaFlex capabilities include: - Variable aspect ratio and resolution via patch coordinates - Patch type indicators for handling padding tokens in attention - Flexible position embedding interpolation for arbitrary grid sizes - Support for factorized position embeddings The patch embedding can be one of two types: - Conv2d-based (default): For standard image inputs [B, C, H, W] - Linear-based: For pre-patchified inputs [B, N, P*P*C] Args: patch_size: Size of patches for patch embedding in_chans: Number of input image channels embed_dim: Dimensionality of patch embedding proj_type: Type of embedding projection layer ('conv' or 'linear') input_norm_layer: Normalization layer applied to input (linear mode only) proj_norm_layer: Normalization layer applied after projection pos_embed: Type of position embedding ('learned', 'factorized', 'none') pos_drop_rate: Dropout rate for position embeddings class_token: Whether to include a class token reg_tokens: Number of register tokens to include bias: Whether to use bias in projection layers dynamic_img_pad: Whether to enable dynamic padding for variable resolution pos_embed_grid_size: Grid size for position embedding initialization pos_embed_interp_mode: Interpolation mode for position embedding resizing pos_embed_ar_preserving: Whether to preserve aspect ratio during position embedding interpolation default_img_size: Default image size for position embedding grid calculation Nr/in_chansr1 proj_typer9rCrDrVdefault_img_sizerFrGrIrJrKrdproj_norm_layerrfr?rprr~c||d}t|||_||_| |_| |_||_t||_||_ ||_ ||_ ||_ ||_ |rdnd|_|xj|z c_|r+tj t#j$dd|fi|nd|_|r+tj t#j$d||fi|nd|_d|_d|_| | |_nV| Tt| |_t/t1|j*|jDcgc] \}}||z c}}|_|dk(ry|jd|jdz|z}|dur |Jd|dur|n|xsd}|r||nd|_tj4||fd|i||_d |_d|_n9|rJd|_tj<||f|||d ||_d|_d |_|jr!t?|j||| d| |_ nd|_ |dur |Jd |dur|n|xsd}|r||ntjB|_"| d vr|j, tGdd|_$d|_%d|_&| r| dk(rd|_'n| dk(r|j,J|j,\}}d|_'tj t#j$d||fi||_%tj t#j$d||fi||_&nU|j,J|j,\}}tj t#j$d|||fi||_$d|_'tjP||_)|jUycc}}w)aInitialize NaFlexEmbeds module. Args: patch_size: Size of patches for patch embedding. in_chans: Number of input image channels. embed_dim: Dimensionality of patch embedding. proj_type: Type of embedding projection layer ('conv' or 'linear'). proj_bias: Whether to use bias in projection layers. class_token: Whether to include a class token. reg_tokens: Number of register tokens to include. dynamic_img_pad: Whether to enable dynamic padding for variable resolution. default_img_size: Default image size for position embedding grid calculation. pos_embed: Type of position embedding ('learned', 'factorized', 'none'). pos_embed_grid_size: Grid size for position embedding initialization. pos_embed_interp_mode: Interpolation mode for position embedding resizing. pos_embed_ar_preserving: Whether to preserve aspect ratio during interpolation. input_norm_layer: Normalization layer applied to input (linear mode only). proj_norm_layer: Normalization layer applied after projection. norm_layer: Default normalization layer. pos_drop_rate: Dropout rate for position embeddings. enable_patch_interpolator: Enable dynamic patch size support. channels_last: Per-patch flat layout: True=(ph,pw,C) [NaFlex default], False=(C,ph,pw). rrrrNrbTz5`norm_layer` must be given when input_norm_layer=TruebiasF) kernel_sizestrider)base_patch_sizerr1 interpolation antialiasrz4`norm_layer` must be given when proj_norm_layer=True) factorizedrEzgCannot initialize position embeddings without grid_size.Please provide img_size or pos_embed_grid_size.rrrE)p)+superrhas_class_tokennum_reg_tokensrIrJrKrr/rr1rVrprrnn Parameterrempty cls_token reg_tokenrrGtupler norm_inputLinearprojflatten is_linearConv2drpatch_interpolatorIdentitynorm ValueErrorrF pos_embed_y pos_embed_xpos_embed_typeDropoutpos_dropreset_parameters)rr/rr1rr9rCrDrVrrFrGrIrJrKrdrrfr?rprrrddsr patch_dimrr __class__s r|rzNaFlexEmbeds.__init__os` / *(%:"'>$)B&#J/  ".)B&*'2q *,NYekk!Q &HR&HI^bV`ekk!Z&Qb&QRfj<@>B  *':D $  )$-.>$?D !',TEZEZ\`\k\kAl-mAa1f-m'nD $  *T__Q-??(JI(D0Z5G HG HH-=-EzL\Ld`d =M.y9SWDO )YMYM"MDI DL!DN( ''"DO '!   DI DL"DN  ) )&< $!#3+ 'D #'+D #$t+ 0B C B CC(74(?*oF]Y]2AOI.r{{}  1 1d6N6N6VBC C263737I/"(D  , &++7 77++DAq".D !||EKK1i,N2,NOD !||EKK1i,N2,NOD ++7 77++DAq\\%++aAy*OB*OPDN"+D  ]3  U.nsO c,|j+tjj|jd|j+tjj|jd|j +tjj|j d|j +tjj|j d|j,tjj|jdyy)Ngư>)stdg{Gz?)rrinitnormal_rrFrrrs r|rzNaFlexEmbeds.reset_parameters s >> % GGOODNNO 5 >> % GGOODNNO 5 >> % GGOODNNO 4    ' GGOOD,,#O 6    ' GGOOD,,#O 6 (r{cDt|j|jS)zGet feature information for feature extraction. Args: location: Feature extraction location identifier Returns: Dictionary containing feature channel count and reduction factor )num_chs reduction)dictr1r/)rlocations r| feature_infozNaFlexEmbeds.feature_infosDNNdooFFr{ as_scalarcH|rt|jS|jS)zGet the feature reduction ratio (stride) of the patch embedding. Args: as_scalar: Whether to return the maximum dimension as a scalar Returns: Feature reduction ratio as scalar or tuple )maxr/)rrs r| feat_ratiozNaFlexEmbeds.feat_ratio!s  t' '?? "r{img_sizec|jrPtj|d|jdz tj|d|jdz fS|d|jdz|d|jdzfS)aCalculate grid (feature) size for given image size. Takes into account dynamic padding when enabled. Args: img_size: Input image size as (height, width) Returns: Grid size as (grid_height, grid_width) rr)rVmathceilr/)rrs r|dynamic_feat_sizezNaFlexEmbeds.dynamic_feat_size/s   99Xa[4??1+==> (ST+X\XgXghiXjJj@kk kA;$//!"44hqkT__UVEW6WW Wr{r patch_coordc z t|}jjdd\ jjddddj fd}i}t |D]&\}}|j |gj|(|jD]\}}||} tjd| jd} ddd| fjdtj|j| ddd| fjt|ddy) a_Apply learned position embeddings to NaFlex batch in-place. Interpolates learned 2D position embeddings for each sample in the batch based on their individual grid sizes. Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values rrrrc|dk(r)|dk(r!jjdzd}nzjrtt |n|}t j |jddddddd|dd|dfjdjdd}|jj S) a Return a flattened positional-embedding grid at an arbitrary spatial resolution. Converts the learned 2-D table stored in NCHW format (pos_embed_nchw) into a (1, H*W, C) sequence that matches the requested size. rrFTsizemode align_cornersrNrr) rFrrJrrr interpolaterIr transposetor)r$pos_embed_flat _interp_sizeorig_horig_wpos_embed_nchwrrs r| _interp2dz?NaFlexEmbeds._apply_learned_naflex_pos_embed.._interp2dSsQ6!Q6(9!%!7!76F?B!O7;7S7SyT3Y] !""%33"'" " Qa(47(* ", -4GAJyyA "$$177$3 3r{Nrr")rrFrrrw enumerate setdefaultappendrmin index_add_r as_tensorrexpandlen)rrrnaflex_grid_sizesr0rrrrr+rr-r.r/s`` @@@r|_apply_learned_naflex_pos_embedz,NaFlexEmbeds._apply_learned_naflex_pos_embed?s. 8 D--a2//1a;AAC 4 4*=?01 9EB  & &q" - 4 4R 8 9!0 5 5 7  A}'q\N!''!*n&:&:1&=>G a'kN % % ahh?q(7({+223}3Er2N  r{c|j}|j\}}}|jdjdz}|jr.|j d}|j } | x} } | |z x} } n-|j d\} } | |dddfz } | |dddfz } t j|dd|t j}| |ddddf<| |ddddf<| dz |ddddf<| dz |ddddf<tj|||| | fd }tj|jjddddj|d d d j||j dd j#|j$ }t j&||t j(j+d}|||dd|d|dfz }y)aApply learned position embeddings to NaFlex batch using grid_sample. Uses F.grid_sample for efficient interpolation of learned 2D position embeddings based on patch coordinates. Based on proposal by https://github.com/stas-sl Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values rrrNrrrFr&r"borderr%r& padding_moder'.r.r)rrrvaluesrJrrrfloat32r affine_grid grid_samplerFrr8rwrIr*rarangelong unsqueeze)rrrrrNrshapesL_iL_global grid_size_y grid_size_xscale_xscale_ythetagridrFrs r|+_apply_learned_naflex_pos_embed_grid_samplez8NaFlexEmbeds._apply_learned_naflex_pos_embed_grid_samplexs''1aQ'..2  ' '++!+$CxxzH(0 0K+ (3 .Gg'-{{q{'9 $K!F1a4L0G!F1a4L0G Aq!F%--H aAg aAg 1aAg 1aAg}}UQ; $DTYZMM NN " "1aA . 5 5aR D J J L ++!  "177"  \\!F%** = G G J Yr1k&1;v3FF GGr{rc8|jjdd\}}|d|k(r)|d|k(r!|jjd||zd}n|jrt |}||f}n|}t j |jjddddj||jddd d d d d |dd |dfjdjdd}|j|j }|j|y ) a0Apply learned position embeddings to standard 2D batch in-place. Interpolates learned 2D position embeddings to match the specified grid size. Args: x: Input tensor to add position embeddings to [B, H*W, C] grid_size: Target grid size as [height, width] rrrr"rFTr#Nr')rFrrrJrrr(rrwrIrr)r*radd_)rrrr-r.r+Lr,s r|_apply_learned_pos_embedz%NaFlexEmbeds._apply_learned_pos_embeds--a2 Q<6 !ilf&<!^^33AvKN++ N !t ( ]]&&q!Q288:!//#  MYq\M=IaL=0 2 3:'!*YYq!_ (***9 ~r{c t|}t|jdk(sJ|jjd|j jd}}i}t |D]&\}}|j|gj|(dtjdtdtdtjffd } |jD]\}} |\} } |jrt| | x} }n| | }} | |j| |ddd| f}| |j ||ddd| f}|jd |jdz}|j!dd }t#jd|jd}ddd|fj%dtj&| j( |ddd|fj+t| d d  y) acApply factorized position embeddings to NaFlex batch in-place. Uses separate Y and X position embedding tables that are interpolated and combined for each sample's grid size. Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values rrtable new_length orig_lengthr~c ||k(r|jjStj|j dddj |ddj dddjjS)z Resample a 1-D positional-embedding table to specified length and return it in (1, L, C) layout, dtype matching x. r'rrrrbFr$r%r&r*rrr(rrwrZr[r\rs r| _interp1dzBNaFlexEmbeds._apply_factorized_naflex_pos_embed.._interp1dst [(xxaggx..== aA&,,.#  gaArrr0  1r{Nrr1r")rr9r$rrrr2r3r4rTensorrurrJrrIrr5r6r7rr8)rrrr:r-r.rrrrartarget_htarget_wlen_ylen_xpe_ype_xposrs ` r|"_apply_factorized_naflex_pos_embedz/NaFlexEmbeds._apply_factorized_naflex_pos_embeds 8 D$%222))//2D4D4D4J4J14M=?01 9EB  & &q" - 4 4R 8 9 1U\\ 1s 1 1QVQ]Q] 1!0 5 5 7  A}!" Hh++ #Hh 77'uT--uf=a(lKDT--uf=a(lKD..#dnnQ&77C++a#C!''!*ciil3G a'kN % % ahh?AxxK ''M(:BC  r{cj}j\}|jddz}|jr.|jd}|j}|x}} ||z x} } n,|jd\}} | |dddfz } ||dddfz } dtj dtj dtj dtj ffd } | |j | | } | |j| | }t j|tj jd}| |ddd|d f||ddd|d fzz y)aApply factorized position embeddings to NaFlex batch using grid_sample. Uses F.grid_sample for efficient interpolation of separate Y and X position embedding tables based on patch coordinates. Based on proposal by https://github.com/stas-sl Args: x: Input tensor to add position embeddings to [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values rrrNrZscale out_lengthr~c|jdddjdjdddj}t j ddj }||ddddf<|dz |ddddf<d|ddddf<tj|d|fd}tj||d dd }|jjS) Nrrrr"rr1Fr=bilinearr>r?) rrIr8rwrrrrrErFr*r) rZrlrmperRrSrrrs r|razNNaFlexEmbeds._apply_factorized_naflex_pos_embed_grid_sample.._interp1d'sq!Q'11!4;;Ar2rJPPRBKK1a9E"E!Q'N"QYE!Q'NE!Q'N==Aq*(=USDr4j\deB55> !r{)rlrmrrBrA) rrrrJrrbrrrGrHrI)rrrrrrKrLrMrNrOrPrQrarhrgrrrs ` @@r|._apply_factorized_naflex_pos_embed_grid_samplez;NaFlexEmbeds._apply_factorized_naflex_pos_embed_grid_samples]''1a!!a!(1,  ' '++!+$CxxzH(0 0K+ (3 .Gg(.{{1~ $K!F1a4L0G!F1a4L0G "U\\ "%,, "ELL "]b]i]i "))[Q))[Q \\!F%** = G G J T"aK// 04Aq+fBU8U3V VVr{c|jjd|jjd}}|\}}|jrt ||x}}n||}}dt j dtdtdt j ffd } | |j||ddd|f} | |j||ddd|f} | jd| jdz} | jdd} j| y) a]Apply factorized position embeddings to standard 2D batch in-place. Uses separate Y and X position embedding tables that are interpolated and combined for the specified grid size. Args: x: Input tensor to add position embeddings to [B, H*W, C] grid_size: Target grid size as [height, width] rrZr[r\r~c ||k(r|jjStj|j dddj |ddj dddjjS)Nr'rrrrbFr^r_r`s r|raz;NaFlexEmbeds._apply_factorized_pos_embed.._interp1dNsr[(xxaggx..== aA&,,.#  gaArrr0  1r{Nr) rrrrJrrrbrurIrrV)rrrr-r.rcrdrerfrargrhrFr+s ` r|_apply_factorized_pos_embedz(NaFlexEmbeds._apply_factorized_pos_embed8s ))//2D4D4D4J4J14M&(  ' '(3 3EE#X5E 1U\\ 1s 1 1QVQ]Q] 1))5&9!YhY,G))5&9!YhY,GNN1%q(99 "**1a0 ~r{ patch_validcd}|jd}|jr]|Jt|jdk(dt ||j |j |j\}}n*t|jdk(xs|jdk(d|jr|jdk(rt|jdud t|jr|jd dn|jdd}|j||jj|jj|d }nC|jd }|j|j|}|j|}nt|jdk(d |j r|jdd\}}|j d||j dzz |j dz} |j d||j dzz |j dz} t!j"|d| d| f}|j|}|jdd}|jr!|jd j%dd }|j'|}|j(dk(rJ||j+||n|j,r|j/||nl|j1||nX|j(dk(rI||j3||n3|j,r|j5||n|j7||g} |j8,| j;|j8j=|dd|j>,| j;|j>j=|dd| rtAjB| |gzd}|jE|}||fS)aForward pass for patch embedding with position encoding. Args: x: Input tensor. Supported formats: - [B, C, H, W] for conv mode - [B, N, P*P*C] for pre-patchified linear mode (normal) - [B, N, Ph, Pw, C] for pre-patchified linear mode (variable patch size) patch_coord: Optional patch coordinates [B, N, 2] for NaFlex mode. patch_valid: Optional validity mask for patches [B, N] for NaFlex mode. Returns: Tuple of (embedded_tensor, grid_size) where: - embedded_tensor: [B, num_prefix_tokens + N, embed_dim] - grid_size: (H, W) tuple for standard mode, None for NaFlex mode Nrrz-Expecting 2D image input with input ndim == 4)rrrrz/Expecting patchified input with ndim == 3 or 5.z,input norm not supported with patch resizingrT)r/rzConvolutional input must be 4DrrE)r)rrr"r)#rrrndimrr/rVrrprrrrweightrrrrr)rrrXrKrTr;rtrqrjrr4r8rrcatr) rrrrurrpatch_hwrrrrto_cats r|forwardzNaFlexEmbeds.forwardbsr*04 GGAJ >>"! %TU-tD,@,@PTPbPb d 9 ! 2qvv{4ef--!&&A+4/1_`!1C1C1QRSTV++II$$IINN'" ,IIaL??.*AIIaL AFFaK!A B##wwrs|1+a$//!2D.DDXYHZZ+a$//!2D.DDXYHZZEE!a512 ! A I||IIaL**1a0 IIaL   ) +$--a9-E11DDQT_D`88 8T  L 0$00i0H11GGWbGc;;A;;W >> % MM$..//2r: ; >> % MM$..//2r: ;  &A3,A.A MM! )|r{)r.rr0NTTrFNrE)r~rHFFNNNr:FTNNr~NTNN)!rqrrrsrtrrur r ryrxr rModulerwrrrrrrrrrrbr;rTr rXrjrqrtr} __classcell__r s@r|rrEs&T79 '+" $$)FJ&=E)2,1.3:>FJ48#%.3"&/Y c5c?23Y Y  Y  } Y  Y Y Y "Y 'uS%S/-A'BCY Y "*%S/!:Y $'Y &*Y (,Y 'tBII7!Y "#4$ryy/)B#BC#Y $!bii1%Y &!'Y ((,)Y * +Y 0 1Y v 7 GS#X G #D #E#uS#X:N4O #X%S/XeCHoX 6||66  66p,H||,H,H  ,H,H\!||!Cy!  !F;||;;  ;;z-W||-W-W  -W-W^(||(Cy(  (Z3726 j||j"%,,/j"%,,/ j u||XeCHo66 7 jr{rrur symmetricq_lenrc |y|j}|j\}}|}|dkDrA|j||ftj}tj||gd}||z }|r5|j d|j dz} | j d} n%|xs|}|ddddddfj |d||} tj| |} | j| tj|j| S)a3Creates an attention mask from patch validity information. Supports two modes controlled by `symmetric`: 1. `symmetric=True` (default): Creates a symmetric mask of shape [B, 1, seq_len, seq_len]. An attention pair (i, j) is allowed only if both token i and token j are valid. Suitable for standard self-attention. 2. `symmetric=False`: Creates a potentially non-square mask of shape [B, 1, q_len, kv_len]. An attention pair (q, k) is allowed only if the key/value token k is valid. Query token validity is not checked in the mask itself. Useful for cross-attention or specific self-attention implementations `q_len` can be specified. Used for NaFlex mode to handle variable token counts and padding tokens. Args: patch_valid: Tensor of shape [B, N] with True for valid patches, False for padding. num_prefix_tokens: Number of prefix tokens (class token, register tokens) to prepend, which are always considered valid. symmetric: If True, create a symmetric mask. If False, create an expanded mask based only on key/value validity. q_len: Query sequence length override. Only used when `symmetric` is False. Defaults to the key/value sequence length (`kv_len`) if None. dtype: Dtype of the output attention mask (e.g., torch.float32). Returns: Attention mask tensor. Additive mask (-inf for masked, 0 for unmasked). Shape is [B, 1, seq_len, seq_len] if symmetric=True, or [B, 1, q_len, kv_len] if symmetric=False. Nrr'rrr") rxrnew_onesrrzrIr8 zeros_like masked_fill_finfor5) rurrrrrrJkv_len prefix_valid mask_bool mask_floats r|create_attention_maskrsJ""$K   DAq F1"++Q0A,B%**+U ii{ ;C ##))"- 0E0Ea0HH ''* 4q 0188AufM !!)59JYJ E(:(>(>? r{ pool_typereduce_include_prefixc||dvrt||||}|S|dkDrF|r9|j|jd|}tj||gd}n |dd|df}|j |j }|dk(rN||jdzjd}|jdd jd }||z } | S|d k(r||jdzjd}|jdd jd }||z } |j} tj| j j| |<| jd} d | | zzS|dk(rO|j} tj| j j| |<| jdSJ)aPGlobal pooling with NaFlex support for masked tokens. Applies global pooling while respecting patch validity masks to exclude padding tokens from pooling operations. Args: x: Input tensor with shape [B, N, C] patch_valid: Optional validity mask for patches [B, N-num_prefix_tokens] pool_type: Type of pooling ('token', 'avg', 'avgmax', 'max') num_prefix_tokens: Number of prefix tokens (class/register) reduce_include_prefix: Whether to include prefix tokens in pooling reduction Returns: Pooled tensor with shape [B, C] N)avgavgmaxrrrrrrrrr"T)rkeepdim)r5rg?r)r*rrrrzr*rrIsumclampclonerr5r) rrurrrrpatch_valid_float masked_sums valid_countspooled masked_avgmasked_x masked_maxs r|global_pool_naflexrs.i/GG  /"7   1 '// d |j@ d |j d|j0d|d|jBd|j d|jDd|jFd|jHd|jJd| d|jLdtO|dddtO|dd||_(|j@r| |j(fi|ntjR|_*d|_+d|_,|jZr|jZdk7rddl.m/}m0}m1}m2}|jZdk(rO||j(f|jf|jh|jjd|jld ||_+d|_,nS|jZd!k(r\||j(|jhzf|jjdd|jn|jp|jld"||_+d|_,n|jZd#k(rP||j(|jhzf|jjd|jl|jrd$||_+d|_,n|jZd%k(rb|jtJd&||j(|jhzf|jt|jj|jld'||_+d|_,ntwd(|jZ|jxdkDr't{|jx|j.)|_>nd|_>t|j|jf}tjt|jfDcgc]v}| d2|j(|jh|j|j|j|j|j|j|j||| | | |d*|xc}|_J|jPjd+}t|jfDcgc]}td,||j(|-c}|_M|jr|js| |j(fi|ntjR|_P|jd.k(rQt|j(f|jxs |jh|jxs |j| | d/||_Tnd|_T|j}||jd0k(}|jr|r| |j(fi|ntjR|_Otj|j|_W|dkDr!tj|j(|fi|ntjR|_Y|j|_[|j|_\|j|jd1ycc}wcc}w)3aInitialize NaFlexVit model. Args: cfg: Model configuration. If None, uses default NaFlexVitCfg. in_chans: Number of input image channels. num_classes: Number of classification classes. img_size: Input image size (for backwards compatibility with classic vit). **kwargs: Additional config parameters to override cfg values. rrLrrrtokenr[r)rLrrErrrFr/rr1rr9rCrDrrVrFrGrIrJrKrr?rprrWTNr)RotaryEmbeddingCatRotaryEmbeddingDinoV3RotaryEmbeddingMixedRotaryEmbeddingMRopemixed)r3r4 temperature feat_shape grid_indexingaxial)r in_pixelsrref_feat_shape grid_offsetrdinov3)rrrrrz1rope_type='mrope' requires cfg.rope_mrope_section) mrope_sectionrrzUnknown rope_type: )r)rr4r5r7r8r9r= proj_drop attn_drop drop_pathrfrgrir3)rzblocks.)modulerrr[)r4r5rfrgr needs_resetrz)^rrr+rr\rCrFrrfrrerrgrGELUrrirrrr1 num_featureshead_hidden_sizerrDrrr]grad_checkpointingrr/rcrXrVrGrIrJrKr?getattrembedsrnorm_prer rope_is_mixedrMtimm.layers.pos_embed_sincosrrrrr3r4rOrSrPrQrTrUrr@r patch_droprrB Sequentialranger5r7r8r9r=rAr;blocksrrrrYrZrrr^r_ attn_poolrr> head_droprheadr`weight_init_modera init_weights)rr}rrrrrrrrfrergrhrirrrrdpripatch_reductionrZr s r|rzNaFlexVit.__init__js &  /#\^ !#00C"NNNN#//W"<<<}} EEEE$CNN3@y )#*>*>?!#--0;BGG $MM(S '  ??EH]]RRD1DN&)oo1 #..0!nn"#&#:#: "'# ~~  mm ))  ,,&    ~~ &  // mm !$ 7 7 #&";"; %($?$? '*&C&C - ++! "'.c3NPU&V# $"#'?F'  *.rescale1s9 7 499S9_56 7 7 7s (AAattnrmlp attn_out_proj mlp_out_proj) rrbrur2rhasattrrrryrfc2rr)rrlayer_idlayers r|fix_init_weightzNaFlexVit.fix_init_weight/s 75<< 7C 7D 7 )5 AOHeuf% ..1 =ue$ ,,hl;uo.++22HqLAun-**118a<@ Ar{r%rc|xs |j}|dvsJd|vr tj|j nd}t t |||||j r|jyy)afInitialize model weights according to specified scheme. Args: mode: Initialization mode ('jax', 'jax_nlhb', 'moco', or '') needs_reset: If True, call reset_parameters() on modules (default for after to_empty()). If False, skip reset_parameters() (for __init__ where modules already self-initialized). )jaxjax_nlhbmocorLnlhbr:rN)rrlogrr%get_init_weights_vitrar)rr%r head_biass r|rzNaFlexVit.init_weights@so,t,,666639T>TXXd..//r (ykRTXY ==  " r{checkpoint_pathprefixcBddlmddlmdfd }||||y)Nr) _load_weights) _torch_loadc|dd}t|tr d|vr|d}t|jD]}|j dr|j ||d|z<,|j dr|j ||d|z<U|j dr|j ||d|z<~|j d s|j ||d|d d z<|||S) z8Adapter function to handle the different model structurecpuT) map_location weights_only state_dictrembeds.rrF patch_embedr2N) isinstancerlistr startswithpop)modelrrrr_orig_load_weightsrs r|_load_weights_adapterz8NaFlexVit.load_pretrained.._load_weights_adapterWs$_5W[\J*d+ 0J' 5 *//+, G<< ,0:q0AJy1}-\\+.0:q0AJy1}-\\+.0:q0AJy1}-\\-05?^^A5FJy1RS612 G&eZ@ @r{rL)vision_transformerr_helpersr)rrrrrrs @@r|load_pretrainedzNaFlexVit.load_pretrainedQs  L) A& dOVembeds.cls_tokenembeds.pos_embedembeds.reg_tokenno_weight_decay)rrupdater)r skip_lists r|rzNaFlexVit.no_weight_decayls@Q 99,=>   TYY668 9r{c.|jjS)zHReturn the 2-tuple patch size. For NaFlex dataloader / transform wiring.)rr/rs r|get_patch_sizezNaFlexVit.get_patch_sizexs{{%%%r{coarsec tdddgS)zGet parameter group matcher for optimizer parameter grouping. Args: coarse: Whether to use coarse-grained grouping Returns: Dictionary mapping group names to regex patterns z^embeds)z^blocks\.(\d+)N)z^norm)i)stemr)r)rr s r| group_matcherzNaFlexVit.group_matcher}s-/CD  r{enablec||_t|jdrGt|jjdr&|jjj |yyy)zEnable or disable gradient checkpointing for memory efficiency. Args: enable: Whether to enable gradient checkpointing rset_grad_checkpointingN)rrrrr)rrs r|rz NaFlexVit.set_grad_checkpointingsO#) 4;; .74;;;R;RTl3m KK # # : :6 B4n .r{c|jS)zeGet the classification head module. Returns: Classification head module )rrs r|get_classifierzNaFlexVit.get_classifiersyyr{rrc t|}i}g}t|D]3\}}||vrg||<|j|||j|5|j\}} } | |jz } |j r/t |j|||| |j|jStj|| |jjdz|j|j} t|jdr]|jj|} t|| |j!D]#\}}}|\}}||z}|D]}|d|| |d|f<%nJ|j#D]7\}}|jj%|}|\}}||z}|d|| |d|f<9| j'd} | S)aGenerate ROPE position embeddings for NaFlex batch with variable grid sizes. Args: x: Input tensor [B, N, C] patch_coord: Patch coordinates [B, N, 2] with (y, x) values Returns: ROPE embeddings: - Axial mode: Tensor of shape [B, 1, N, dim*2] - Mixed mode: List of tensors, each of shape [B, num_heads, N, dim], one per depth layer - Mixed mode with iterator: Iterator yielding tensors per depth rrget_batch_embedsNrr)rr2r4rrrrrrrrrrrrrrCrrrI)rrrr:rrrrrrJrr rope_embeds unique_embedsrrrrrrs r|_generate_rope_naflexzNaFlexVit._generate_rope_naflexs&8 D &'89 2MB /-/ *##I. I & - -b 1  2 ''1ad,,,   %  kk!Wdiimma.?qwwWXW_W_` 4990 1 II66|DM36|]TcTjTjTl3m F/ 5- 1U 'FB38*3EKKZK0F F"1!6!6!8 G 2!YY00y0A  1U /9+:/F B  O,  G "++A. r{r\c||_|=|dvsJ|dk(r|jJd|dk7r|jd|_||_|dkDr&tj|j ||_ytj |_y)zReset the classification head with new number of classes and pooling. Args: num_classes: Number of classes for new classification head global_pool: Optional new global pooling type Nrr[z=Cannot currently add attention pooling in reset_classifier().r)rrr\rrr1rr)rrr\s r|reset_classifierzNaFlexVit.reset_classifiers'  ""NN NNe#(>]]]u%$..*D!%*D >IAoBIIdnnk: SUS^S^S` r{c(|du}|j|||\}}d}|j;||j||}n&||jj|}nJdd}|jrd|j X|j |\}}||j d|}|.|js"t||||}|s|jd}|"t||j|j}|j|}|||||dS) zR Forward pass through patch / abs pos / rope pos embeds and patch dropout N)rrurz4Expected one of patch_coord or grid_size to be validrpos_embed_has_batch)rr)rrur attn_mask keep_indices)rrrrtrainingrgatherrrrIrrrr) rrrrur naflex_moderrrs r|_forward_embedszNaFlexVit._forward_embedssD"- {{ ### 9 99 &"88KH &"ii11 1B TTTu04 ==T__8"ooa0OA|&)00LA &t/A/A5Q \grs ""-"7"7":K  -"&"8"8ggI MM! &&"(   r{indicesreturn_prefix_tokensr stop_early output_fmtintermediates_only output_dictrurc |dvsJd|dk(} g} tt|j|\}}t|trd| r t d|d} |j d| } |j d| } |d}d x}}|sW|jrK|j?t d |}|jd d \}}|jj||f\}}|j|| | | }|d}|j d d }|j dd }|j dd } |j d| } tjjs|s |j}n|jd |dz}|j xrtjj }|j"r|t%t'|j|D]\}\}}|jr-|j!|t)||||j dd}|rt+|||| }n |||| }||vsd| j-|r|j/|n|nwt%|D]i\}}||rt+|||| }n'|||| }n|rt+||| }n ||| }||vsF| j-|r|j/|n|k|j0rE| Dcgc]}|d d d|j0f}}| Dcgc]}|d d |j0d f} }nd }| rN| Dcgc]C}|j3|jd||dj5ddddj7E} }|r1i}| |d<||r||d<| | |d<|s|j/|} | |d<|Stjjs|r|t9t'| |} |r| S|j/|}|| fScc}wcc}wcc}w)a Forward features that returns intermediates. Args: x: Input image tensor indices: Take last n blocks if int, all if None, select matching indices if sequence return_prefix_tokens: Return both prefix and spatial intermediate tokens norm: Apply norm layer to all intermediates stop_early: Stop iterating over blocks when last desired intermediate hit output_fmt: Shape of intermediate feature outputs intermediates_only: Only return intermediate features output_dict: Return outputs as a dictionary with 'image_features' and 'image_intermediates' keys patch_coord: Optional patch coordinates [B, N, 2] for NaFlex mode patch_valid: Optional patch type indicators (1=patch, 0=padding) for NaFlex attn_mask: Optional attention mask for masked attention Returns: A tuple with (final_features, intermediates), a list of intermediate features, or a dictionary containing 'image_features' and 'image_intermediates' (and optionally 'image_intermediates_prefix'). NaFlex (dict / pre-patchified) inputs: NLC output only (per-sample grids are variable, a single spatial reshape is undefined); with ``output_dict=True`` the result also carries 'patch_valid' aligned with the spatial intermediates so consumers can mask padding or scatter via patch_coord. )NCHWNLCz)Output format must be one of NCHW or NLC.r*zoutput_fmt="NCHW" is not supported for NaFlex (dict) inputs, use "NLC". Per-sample grids vary; reconstruct spatial maps downstream via patch_coord.rrurrNztNaFlex forward_intermediates with active patch dropout requires output_dict=True to return the gathered patch_valid.rwrrurrrrr!Frrrrrr"rrimage_intermediatesimage_intermediates_prefiximage_features)r!r9rrrrgetrrrrrr"rjit is_scriptingrrr2rrr$r4rrrr contiguousr)!rrr#r$rr%r&r'r(rrurr intermediates take_indices max_indexrrrheightwidthrrrrdo_checkpointingrblkry prefix_tokens result_dictx_finals! r|forward_intermediateszNaFlexVit.forward_intermediates4sdJ_,Y.YY,& "6s4;;7G"Q i a  bccM*K%% {;Kk95I lGLA4==T__5P!KLLGGGBCLMFE;;00&%ADAq%% ## &  9 jj5 zz.$7 JJ{D1 jj <  99 ! ! #:[[F[[)a-0F22S599;Q;Q;S7S   +"9(1#dkk;2O(P F$$C==T__%@\E]!7"$,2JJ}e,L "J $"3 iPAAJ)DA $!((11E! F$$F+ F3*'&sAK9U yI'&sACY7 $!((11E F"  ! !ERSQq!D$:$:"::;SMSDQRqQq$"8"8"99:RMR M ' !''!*aB/771aCNNPM K1>K- .(-A  r{rcV|jht||jr |jnddd|j}|js|dd|jdf}|j||}|S| |j n|}t ||||j|j}|S)NrFr)rrrrr.r)rrr]rrr\r)rrrrurs r|_poolzNaFlexVit._pools >> %-<@{$'> I,eeM;? eeM;? EE+y9  Kt+-U V Kt+-U V,,''# -H"!!-H-A%%a(A!!!$Ar{)NrNNNr)rLTrFr)N) NFFFr*FFNNN)NNNr)FN)(rqrrrsrtr r+rurr rrryrxrrr3ignorerr rr rr rrrrrrbr rrrr"rArCrErHr}rrs@r|r,r,]s +/#>BC>,'C>C> C> uS%S/%9:; C> C>JA"####" YY=s=C===4 YY    YY&c3h&& YY  D  T     YYCTCTCC YY B||BB u||T%,,/4 5 BBHaCahsmaW[a"< c5<< < B8<).$$', %262604g U\\4U\\(9#::;g eCcN34g #' g  g  g g !%g g "%,,/g "%,,/g   -g  tELL!5tELL7I)I#JDQTVYQYNZ [g X372604 >\\>"%,,/>"%,,/ >   - > u||T#u||"344 5 >F(,26 || }"%,,/   B %26 1\\ 1 1"%,,/ 1  1372604 0U\\4U\\(9#::;0"%,,/0"%,,/ 0   - 0  0r{c|d}|d}|d}tt|D]}||||}||dddfjdzj}||dddfjdzj}|j ||dddj ddd dd}|j d|dz|dz}dd lm}||d |d d y)Nrrurrrr.rrr) save_imagepatch_z.jpgT) normalize)rr9rrrrtorchvision.utilsrP) rrrurrpatchrrrPs r|_debug_dump_patchesrUysM"KM"K lG 3w< < ;q>* Aq ! % % '! + 1 1 3 Aq ! % % '! + 1 1 3 aBA.66q!Q1E a2qt,05F1#T*d;A$OO"OP[}\dfgjkfkdmn`!a ,-H' ( + +,H' ( + +,H' ( \\. )rsVF&IIaAq)11!4&(G !HW HQKCAF Or{urlc 2|dddddttdddd |S) NrL)rrmg?rHz embeds.projrz apache-2.0) rkr input_size pool_sizecrop_pctrmeanr  first_conv classifierlicenser)rkrs r|_cfgrus7#"'%#    r{ztimm/) hf_hub_id)rvr)z)naflexvit_base_patch16_gap.e300_s576_in1kz-naflexvit_base_patch16_par_gap.e300_s576_in1kz0naflexvit_base_patch16_parfac_gap.e300_s576_in1kz$naflexvit_base_patch16_map.untrainedz,naflexvit_so150m2_patch16_reg1_gap.untrainedz,naflexvit_so150m2_patch16_reg1_map.untrainedz&naflexvit_base_patch16_siglip.v2_webliz(naflexvit_so400m_patch16_siglip.v2_weblivariant pretrainedc ~|jdd}|jdt}ttDchc]}|j}}t |Dcic]}||vs||j|}}|r t |fi|}t t||ft|t|dd|} | Scc}wcc}w)N out_indicesrr}getter)rz feature_cls)pretrained_filter_fnr} feature_cfg) rr+rnamerrr r,rjr) rwrxrrzr}fcfg_field_namesr cfg_updatesrs r|_create_naflexvitr s**]A.K **ULN +C'-l';"EY6 Y , , 0 FJJ'95A vzz*:DA VZZ U3 fjju5!" VZZ T2#$!**[%8!',>!F'( )M. Wj BM BBr{c Htdddddddd}td ||d |}|S) zCViT-Base with NaFlex functionality and global average pooling. r.r0r2h㈵>rrT)r/r1r3r4r=r\rDrZrxr})naflexvit_base_patch16_gapr+rrxrr}rs r|rrsC  C  ezWZ e^d eE Lr{c Jtddddddddd }td ||d |}|S) z]ViT-Base with NaFlex functionality, aspect preserving pos embed, global average pooling. r.r0r2rTrr) r/r1r3r4r=rJr\rDrZr)naflexvit_base_patch16_par_gaprrs r|rrsF  $ C  i:[^ ibh iE Lr{c Ltdddddddddd }td ||d |}|S) zjViT-Base with NaFlex functionality, aspect preserving & factorized pos embed, global average pooling. r.r0r2rTrrr) r/r1r3r4r=rJrFr\rDrZr)!naflexvit_base_patch16_parfac_gaprrs r|rrsI  $ C  lj^a lek lE Lr{c Ftddddddd}td ||d|}|S) zBViT-Base with NaFlex functionality and MAP attention pooling. r.r0r2rr[r)r/r1r3r4r=r\rDr)naflexvit_base_patch16_maprrs r|rrs@  C  ezWZ e^d eE Lr{c Ltddddddddd d }td ||d |}|S)/ViT-SO150M2 with NaFlex functionality for variable aspect ratios and resolutions. This model supports: 1. Variable aspect ratios and resolutions via patch coordinates 2. Position embedding interpolation for arbitrary grid sizes 3. Explicit patch coordinates and valid token masking r.@ NN@rFrrT) r/r1r3r4r5r=r7rDr\rZr)"naflexvit_so150m2_patch16_reg1_gaprrs r|rrsI  C  mz_b mfl mE Lr{c Jtddddddddd }td ||d |}|S) rr.rrrrrFrr[) r/r1r3r4r5r=r7rDr\r)"naflexvit_so150m2_patch16_reg1_maprrs r|rrsF  C  mz_b mfl mE Lr{c Dtdddddd}td||d|}|S) zGViT-Base with NaFlex functionality and SigLIP-style configuration. r.r0r2 gelu_tanhr[)r/r1r3r4rgr\r)naflexvit_base_patch16_sigliprrs r|rr s=   C  h*Z] hag hE Lr{c Ftddddddd}td ||d|}|S) zUViT-SO400M with NaFlex functionality for variable aspect ratios and resolutions. r.igZӼ @rr[)r/r1r3r4r5rgr\r)naflexvit_so400m_patch16_sigliprrs r|rr s@  C  jJ\_ jci jE Lr{)TT)NrrF)rr:TrrM)artloggingr dataclassesrrr functoolsrtypingrrr r r r r rrrtorch.nnrtorch.nn.functional functionalr timm.datarr timm.layersrrrrrrrrrrrr_builderr _featuresr! _features_fxr"r# _manipulater$r% _registryr&r'rr(rr)r*__all__ getLoggerrqr^r+rrbrurxrrrrrrrDrrryrr,rUrwrrjru default_cfgsrrrrrrrrrrrrzr{r|rs$ 22OOO E    ++L0<6 ; ' '  H % X,X, X,v  L " * <<*#s(O** *  5<<sCx () *ZJ J==@,l,x,^F 299F F R"##"]] A\\AAA} A {{ A ell AAH/3 !"&+ E <<Eell+EE E $ E  \\ EEPY  Y x <GsGuGQUGaiG"HT#s(^HIH$sTWx.HVcT#s(^"%1526:69=9-1F48F48F/3/151%& 0s9*!(A(A(A (AZ!ECECEC ECP4i"t)$$Y& 4 i   4i04i. d     9  r{