^jZ dZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z ddl Z ddlmZddlmcmZddlmZmZddlmZmZmZmZmZddlmZdd lmZdd l m!Z!m"Z"dd l#m$Z$m%Z%dd l&m'Z'd dgZ(GddejRZ*de jVde jVfdZ, dRde jVde jVde jVde-de jVf dZ. dSde jVde jVde jVde-de-de jVf dZ/GddejRZ0GddejRZ1GddejRZ2Gd d!ejRZ3Gd"d#ejRZ4Gd$d%ejRZ5Gd&d ejRZ6Gd'dejRZ7dTd)ejRd*e8d+e9ddfd,Z:dTd-e8d+e9defd.Z;d/ee8e jVfd0e6dee8e jVffd1ZdUd4e8d5e9de7fd7Z?dVd8e8dee8effd9Z@e$e@d:d;d<=e@d>d;d<=e@d?d<@e@dAd<@dBZAeBdCdDdEdFdGd3d(HZCeBdIdJdDdKdLd(MZDe%dUd5e9de7fdNZEe%dUd5e9de6fdOZFe%dUd5e9de7fdPZGe%dUd5e9de6fdQZHy)WaMGemma4 Vision Transformer Vision encoder from Google's Gemma 4 multimodal model. Custom ViT with 2D RoPE, Gated MLP, QKV normalization, and 4-norm sandwich blocks. Paper: https://ai.google.dev/gemma/docs/core/model_card_4 Reference impl: https://github.com/huggingface/transformers (Gemma4VisionModel) Copyright 2025 Yonghye Kwon N)partial)AnyCallableDictListOptionalSetTupleUnion)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)RmsNormDropPath to_2tupletrunc_normal_tf_use_fused_attn)build_model_with_cfg)feature_take_indices) checkpoint named_apply)generate_default_cfgsregister_model)batch_patchifyGemma4VitEncoderGemma4VitClassifierc |eZdZdZ d dedededdffd Zd dZd ejdejfd Z xZ S) Gemma4ClippableLinearzLinear layer with optional input/output clamping. Used in Gemma4 E4B variant where clamp values are finite and affect output. When use_clipped=False, behaves as a standard nn.Linear (no buffers registered). N in_features out_features use_clippedreturnc||d}t|||_tj||fddi||_|r|j dtjdi||j dtjdi||j dtjdi||j dtjdi||jyy) NdevicedtypebiasF input_min input_max output_min output_max)) super__init__r!nnLinearlinearregister_buffertorchemptyreset_parameters)selfrr r!r%r&dd __class__s a/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/models/gemma4_vit.pyr.zGemma4ClippableLinear.__init__'s / &ii \LLL   ekk.C.C D  ekk.C.C D  u{{/D/D E  u{{/D/D E  ! ! # cB|jr|jjtd |jjtd|j jtd |j jtdyy)Ninf)r!r(fill_floatr)r*r+r6s r9r5z&Gemma4ClippableLinear.reset_parameters;sn    NN %, / NN u . OO ! !5<- 0 OO ! !%, / r:xc|jr+tj||j|j}|j |}|jr+tj||j |j}|SN)r!r3clampr(r)r1r*r+r6r@s r9forwardzGemma4ClippableLinear.forwardDsZ    At~~t~~>A KKN    At@Ar:)FNNr"N) __name__ __module__ __qualname____doc__intboolr.r5r3TensorrE __classcell__r8s@r9rr s^!& $$$ $ $(0%,,r:rr@r"c|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Ndim)shaper3cat)r@x1x2s r9 rotate_halfrYMsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r:cossin unsqueeze_dimcn|j|}|j|}||zt||zzS)z0Apply rotary position embedding to input tensor.) unsqueezerY)r@rZr[r\s r9apply_rotary_pos_embr_Ts8 -- &C -- &C G A, --r:ndimc f|jd}d|d|zzz}|g|z}tj||d}tj||d} tj||d} t|D cgc]} t || | | | | |} } tj | dScc} w)zApply multidimensional RoPE to input tensor. Splits input along head_dim into ndim parts, applies RoPE to each, then concatenates back. rQrRrS)r\)rUr3splitranger_rV) r@rZr[r`r\num_input_channelsnum_rotated_channels_per_dim split_sizesx_parts cos_parts sin_partsky_partss r9apply_multidimensional_roperl`s#$(:q4x(H#I /047Kkk![b1G C"5I C"5IlqqukvfgWQZ1y|S`aG 99W" %%s6B.ceZdZdZ d dededdffd ZddZddZdd Z d e jd e jde e je jffd Z xZS)Gemma4RotaryEmbedding2Dz2D Rotary Position Embedding for Gemma4 vision encoder. Computes RoPE independently for each spatial dimension (x, y), using theta=100.0 and the partitioned head_dim. Nhead_dim rope_thetar"ct|||_||_|dzdz}|j dt j ||t jd|jy)NrRinv_freqr$F) persistent) r-r.rorpr2r3r4r> _init_buffers)r6rorpr%r& num_freqsr8s r9r.z Gemma4RotaryEmbedding2D.__init__si   $]q(    KK & D  r:c |jdz}d|jtjd|dtj|j j |z zz }|j j|y)z.Compute and fill non-persistent buffer values.rR?rr&r%N)rorpr3aranger>rrr%copy_)r6 spatial_dimrrs r9rtz%Gemma4RotaryEmbedding2D._init_buffersscmmq(  OO Q Qekk$--J^J^_bmm o  H%r:c$|jyrBrtr?s r9r5z(Gemma4RotaryEmbedding2D.reset_parameterss r:c$|jy)z"Initialize non-persistent buffers.Nr}r?s r9init_non_persistent_buffersz3Gemma4RotaryEmbedding2D.init_non_persistent_bufferss r:r@ position_idsc\tj5|jddddfjj |j dddj |j}g}g}tdD]}|dddd|f}|dddddfj}|j|jzjdd} tj| | fd} |j| j|j| jtj|dj |j} tj|dj |j} ddd| | fS#1swY  fSxYw)z Args: x: Hidden states tensor, used only for dtype/device. position_ids: (B, N, 2) Gemma4-internal ``(x, y)`` patch coords. Returns: cos, sin: (B, N, head_dim) tensors for RoPE application. NrrQrrRrS)r&)r3no_gradrrr>expandrUtor%rc transposerVappendrZr[r&) r6r@rinv_freq_expandedall_cosall_sinidim_posdim_pos_expandedfreqsembrZr[s r9rEzGemma4RotaryEmbedding2D.forwardsq ]]_ ? $ dAtm < B B D K KLL^L^_`Lacegh i l lmnmumu v *,G*,G1X *&q!Qw/#*1dA:#6#<#<#> *0025E5K5K5MMXXYZ\]^iiB7swwy)swwy) *))G,//agg/>C))G,//agg/>C ? Cx! ? Cxs E>FF+)Y@NNrF)rGrHrIrJrKr>r.rtr5rr3rMr rErNrOs@r9rnrnzs~!&   ,& ||  ,,  u||U\\) * r:rnc eZdZdZ ddeeeeeffdedededdf fd Zdd Zd ed ed ed e jdee je jff dZ de jde jde jfdZ ddee jeee jffdee jdee jdee je je jffdZxZS)Gemma4PatchEmbeduLinear patch embedding with a 2D position-embedding table. Unlike the standard ViT PatchEmbed (Conv2d), Gemma4 uses a Linear projection on flattened patches plus a separate 2D position-embedding table applied via one-hot lookup. Inputs may be supplied as: * ``(B, C, H, W)`` raw images — patchified inline in P-P-C flat order. * ``(B, N, P*P*C)`` pre-patchified P-P-C patches (NaFlex loader output). * ``(B, N, Ph, Pw, C)`` pre-patchified unflattened patches. * ``dict`` with keys ``patches`` / ``patch_coord`` / ``patch_valid``. All input-dispatch logic is owned by this module; the parent model simply calls ``self.patch_embed(x, patch_coord, patch_valid)`` and unpacks the returned tensors. N patch_sizein_chans embed_dimposition_embedding_sizer"cN||d}t |t||_||_||_|j\}} t j||z| z|fddi||_t jtjd||fi||_ |jy)Nr$r'FrR)r-r.rrrrr/r0 input_proj Parameterr3r4position_embedding_tabler5) r6rrrrr%r&r7phpwr8s r9r.zGemma4PatchEmbed.__init__s / #J/"'>$B))HrMB$6 TTQST(* KK2I D D) % r:c2t|jdy)N{Gz?std)rrr?s r9r5z!Gemma4PatchEmbed.reset_parameterss66DAr: batch_sizepHpWr%ctj||}tj||}tj||d\}}tj|j |j gd} | j dj |dd} tj|||ztj|} | | fS)z:Row-major grid coords in external NaFlex ``(y, x)`` order.)r%ij)indexingrQrSrrx) r3rymeshgridstackflattenr^ronesrL) r6rrrr%ysxsgrid_ygrid_x patch_coord patch_valids r9_default_patch_coordz%Gemma4PatchEmbed._default_patch_coords\\"V , \\"V ,B>kk6>>#3V^^5E"FBO !++A.55j"bI jjR"WEJJvV K''r:rpadding_positionscT|jd}tj||j}|j ddddj |j }||j z}|jd}tj|jdd |}|S) zCompute position embeddings via one-hot matmul against the 2D table. ``position_ids`` follows the Gemma4-internal ``(x, y)`` convention. rmin num_classesrRrrSrQ) rCFone_hotrpermuterrsumr3wherer^)r6rrclamped_positionsrposition_embeddingss r9_position_embeddingsz%Gemma4PatchEmbed._position_embeddingss)..1.5))-4;W;WX//!Q1-001N1NO%(E(EE155!5<#kk  ' ' +S2E #"r:r@rrct|tr)|jd|}|jd|}|d}|j\}}|jdk(rO|j \}}}} |'|j |||z| |z|j\}}t|||fdd\}}n|jdk(rA|jd d dd d j|j d |j d d }nz|jd k(rR|j \}} } | ||zz} |j|| ||| jd d dd d j|| | }ntd|jd| td|f|d k(jd } | jr| }n=tj |j dd tj"|j}d |dz z}|j%|j'|j$j(j*}|j-d}|}||j/||z}|||fS)aNormalize inputs, patchify / project / position-embed. Accepts external NaFlex ``patch_coord`` (y, x) / ``patch_valid`` inputs and returns the internal (x, y) form used by the rest of the model. Returns: embeddings: (B, N, embed_dim) patch embeddings with position info added. position_ids: (B, N, 2) Gemma4-internal (x, y) coords. padding_positions: (B, N) True for padding tokens. rrpatchesNF)pad channels_lastrrrRrrQz&Expected input ndim in (3, 4, 5); got .z1patch_coord is required for pre-patchified input.rSrx?)rQ)dims) isinstancedictgetrr`rUrr%rrreshapeview ValueErrorallanyr3rrLrrweightr&flipr)r6r@rrrrB_HWNPPCCsentinelrrs r9rEzGemma4PatchEmbed.forwardsS a %% {;K%% {;K) AB 66Q;JAq!Q"+/+D+DQRQRVXQXZ[ZbZb+c( [!!b"X5NDAq VVq[ !Q1a(00QWWQZLA VVq[IAq#R Aq!RQ'//1aA>FFq!SQA8B   PQ Q  #r)..2.6H||~'i #jj%%bq)KDVDV SM OOADD!7!7!=!=> ?#''U'3 (L )),8IJ J, 111r:)r(NNrFNN)rGrHrIrJr rKr r.r5r3r%rMrrrstrrrErNrOs@r9rrs_$79 +0 c5c?23    &)    .B((( ( LL ( u||U\\) * ( #,,# %||#  #.3726 @2U\\4U\\(9#::;@2"%,,/@2"%,,/ @2 u||U\\5<<7 8 @2r:rc&eZdZUdZej j eed< dde de de de e de d e d e d ed dffd Z ddejdejdejde ejd ejf dZxZS)Gemma4AttentionaGemma4 Vision Attention with QKV normalization and 2D RoPE. Key features: - Separate Q, K, V projections (not fused) - RMSNorm on Q, K (with scale) and V (without scale) - 2D RoPE applied after normalization - Attention scale = 1.0 (since QK are normalized) fused_attnNrT num_headsro num_kv_heads attn_drop proj_dropnorm_epsuse_clipped_linearsr"c d| | d} t |||_||_|xs||_||jz|_t |_t|||zfd|i| |_ t||j|zfd|i| |_ t||j|zfd|i| |_ t||z|fd|i| |_ t|f|dd| |_t|f|dd| |_t|f|dd| |_t#j$||_t#j$||_y)Nr$r!TepsaffineF)r-r.rror num_kv_groupsrrrq_projk_projv_projo_projrq_normk_normv_normr/Dropoutrr) r6rTrrorrrrrr%r&r7r8s r9r.zGemma4Attention.__init__hsB / "  (5I&$*;*;;(*+CX1EmSfmjlm +C1B1BX1Mu[nurtu +C1B1BX1Mu[nurtu +I,@#mSfmjlm hHHTHRH hHHTHRH hIHUIbI I.I.r:r@rope_cosrope_sin attn_maskc|j\}}}|j|j|||j|j}|j |j|||j |j} |j|j|||j |j} |j|}|j| } |j| } t|||dd}t| ||dd} |jdd}| jdd} | jdd} |jdkDr:| j|jd} | j|jd} |jr>t!j"|| | ||j$r|j&j(ndd}nD|| jdd z} || |z} | j+d } |j'| } | | z}|jddj-||d }|j/|}|j1|}|S) NrR)r`r\rrSrrw)r dropout_pscalerQ)rUrrrrorrrrrrrlrrrepeat_interleaverrscaled_dot_product_attentiontrainingrpsoftmaxrrr) r6r@rrrrrrqrjvattns r9rEzGemma4Attention.forwards''1a KKN  1dnndmm D KKN  1d&7&7 G KKN  1d&7&7 G KKN KKN KKN (8XAUV W '8XAUV W KK1  KK1  KK1     !##D$6$6A#>A##D$6$6A#>A ??..#.2mm$..** Aq{{2r**D$i'<>$'DqA KK1  % %aB / KKN NN1 r:) @Nrrư>FNNrB)rGrHrIrJr3jitFinalrL__annotations__rKrr>r.rMrErNrOs@r9rr\s %%  *."""(- / / / / #3- /  / / /"& /  /N15 5||5ll5ll 5   - 5  5r:rc eZdZdZ d dededeedededdf fd Z d e jde jfd Z xZ S) Gemma4GatedMlpzGated MLP for Gemma4 Vision Encoder. Uses GELUTanh activation: output = down_proj(gelu_tanh(gate_proj(x)) * up_proj(x)) Nrhidden_features act_layerdroprr"c ||d}t |t||fd|i||_t||fd|i||_t||fd|i||_||nt jd|_t j||_ y)Nr$r!tanh approximate) r-r.r gate_projup_proj down_projr/GELUactrr) r6rrrrrr%r&r7r8s r9r.zGemma4GatedMlp.__init__s / .{OsYlsprs,[/qWjqnpq . sYlsprs"+"79;RWWQW=XJJt$ r:r@c |j|j|j|j||j |zSrB)rrrrrrDs r9rEzGemma4GatedMlp.forwards:yy1B(CdllSTo(UVWWr:)NrFNN)rGrHrIrJrKrrr>rLr.r3rMrErNrOs@r9r r s{-1(-%%!% ) %  % "& % %$XX%,,Xr:r ceZdZdZ ddededededeeded ed ed ed eed eddffd Z dde jde jde jdee jde jf dZ xZ S) Gemma4Blocka Gemma4 Vision Encoder Block with 4-norm sandwich pattern. Unlike standard ViT (pre-norm with 2 norms), Gemma4 uses: - input_layernorm (norm1) + post_attention_layernorm (norm2) - pre_feedforward_layernorm (norm3) + post_feedforward_layernorm (norm4) NrTrrointermediate_sizerrrr drop_pathrrr"c ~| | d}t|t|fd|i||_t d|||||||| d||_t|fd|i||_t|fd|i||_td||| | d||_ t|fd|i||_ | dkDrt| |_ytj|_y)Nr$r)rTrrorrrrr)rrrrrr,)r-r.rnorm1rrnorm2norm3r mlpnorm4rr/Identityr)r6rTrrorrrrrrrrr%r&r7r8s r9r.zGemma4Block.__init__s  / S5h5"5 #  % 3     S5h5"5 S5h5"5 ! - 3    S5h5"5 09C),R[[]r:r@rrrc2|}|j|}|j||||}|j|}||j|z}|}|j |}|j |}|j |}||j|z}|S)Nr)r rr!rr"r#r$)r6r@rrrresiduals r9rEzGemma4Block.forwards JJqM IIa8yI A JJqM t~~a( ( JJqM HHQK JJqM t~~a( (r:) NrrrrNFNNrB)rGrHrIrJrKrr>rrLr.r3rMrErNrOs@r9rrs +/"""",0(-(S(S(S (S # (S #3- (S(S(S(S(S )(S"&(S (S^15 ||llll    -   r:rc "eZdZdZd dedeffd ZdejdejdeejejffdZ dejdejd ejdeejejffd Z xZ S) Gemma4VisionPooleruSpatial pooling for Gemma4 vision encoder output. Pools patches by averaging within k×k grid cells based on position coordinates. Output is scaled by sqrt(hidden_size). hidden_sizepooling_kernel_sizecRt|||_|dz|_||_y)Nr)r-r.r+root_hidden_sizer,)r6r+r,r8s r9r.zGemma4VisionPooler.__init__.s+ & +S 0#6 r: hidden_statesrr"c n|jd}|j}||z}||zdk7rtd|d|d|d||z}|jd}|dj d d ddz}t j ||d } | d||z| dzz} tj| j|j|z } | jdd|jz} t j| dk(jd} | j|j| fS)u@2D spatial pooling on a ``k × k`` grid (k = ``self.pooling_kernel_size``). ``N`` patches are binned into ``k^2``-sized cells, so the pool requires ``N % k^2 == 0`` (caller ensures both grid dims divide by k upstream). ``position_ids`` follows the Gemma4-internal ``(x, y)`` convention. rrz Cannot pool z tokens with k=z: N must be divisible by k^2=z.. Both grid dimensions must be divisible by k.r.rrQTrTkeepdimfloor) rounding_mode.rrRrS)rUr,rrCmaxr3divrrlongr>r logical_notrrr&) r6r/rrrj k_squared output_lengthrmax_x kernel_idxsweightsoutputmasks r9_avg_pool_by_positionsz)Gemma4VisionPooler._avg_pool_by_positions4sY    "  $ $E y=A qc3PQZP[\?@ Y (..1.5!&)--"d-CAFJii 11GL !&)UaZ;v;N,NN ))K,,. >DDFR""1a(=+>+>+@@  'Q,!3!3!3!:;yy,,-t33r:rc|j|jdd}|j||\}}||jz}||fS)uSpatial pool with ``pooling_kernel_size × pooling_kernel_size`` cells. Args: hidden_states: (B, N, D) encoder output. position_ids: (B, N, 2) Gemma4-internal ``(x, y)`` coords. padding_positions: (B, N) True for padding tokens. Returns: pooled hidden states (B, N // k^2, D) and validity mask (B, N // k^2). rQr) masked_fillr^rBr.)r6r/rr pooler_masks r9rEzGemma4VisionPooler.forwardSsV"&112C2M2Mb2QSVW %)%@%@P\%]" {%(=(== k))r:)r) rGrHrIrJrKr.r3rMr rBrErNrOs@r9r*r*'s 7C7c7 4 <<4 ,,4 u||U\\) * 4>* <<* ,,* %|| * u||U\\) * *r:r*c-&eZdZdZ dde deeeffd Ze jjd?d!e ddfd"Ze jjd?d#e ddfd$Zd%e j2ddfd&Z d@d%e j2d'e j2d(e j2d)ee ee j2gdfd*eede j2f d+Z d@d%ee j2eee j2ffd,ee j2d-ee j2de j2fd.Zd>d%e j2d/e de j2fd0Z d@d%ee j2eee j2ffd,ee j2d-ee j2de j2fd1Z dAd%ee j2eee j2ffd,ee j2d-ee j2d2eeeeefd3e d4e d5ed6e d7e deee j2ee j2ee j2feeefffd8Z dBd2eeeefd9e d:e deefd;Z!xZ"S)Cru#Gemma4 Vision Encoder. The pure encoder from Google's Gemma 4 multimodal model. Custom ViT with 2D RoPE, gated MLP, QKV normalization, and 4-norm sandwich blocks. When ``standardize=True`` (31B variant), ``std_bias/std_scale`` are applied after the soft-token pooler (original contract). Other pool modes don't apply it. Output shape depends on ``global_pool``: - ``'soft'`` (default): spatial ``k×k`` pooler + ``√D`` scale + optional standardization. Output ``(B, num_soft_tokens, embed_dim)`` - ``'avg'``: masked mean over patch tokens; skips the pooler's ``√D`` scale and standardization. Output ``(B, embed_dim)``. - ``'none'`` / ``''``: no pool — returns raw patch tokens ``(B, N, embed_dim)``. Useful for building custom pool heads. Nimg_sizerr global_poolrdepthrrorrrrprr, standardizerproj_drop_rateattn_drop_ratedrop_path_rater weight_initr"cX||d}t||dvs Jd|||_|x|_x|_|_d|_d|_d|_d|_ t||_ ||_ ||_ |xsttj d}t#d|j||| d ||_t'd|| d ||_t+j,d||Dcgc]}|j/}}tj0t3|Dcgc]}t5d|||| | | ||||||d |c}|_t9|| |_|rM|j=d t+j>|fi||j=dt+j>|fi|nd|_ d|_!tE|j}t3|Dcgc]}tG||d|c}|_$|dk(rdn||_%|dk7r|jMdyycc}wcc}wcc}w)Nr$)softavgnonez>global_pool must be one of 'soft', 'avg', 'none' (or ''); got rNLCFrr)rrrr)rorp) rTrrorrrrrrrr)r+r,std_bias std_scaleblocks.)num_chs reductionmoduleskipreset needs_resetr,)'r-r.rH num_featureshead_hidden_sizerr output_fmtnum_prefix_tokensgrad_checkpointingrrr,rrr/rr patch_embedrn rotary_embr3linspaceitem ModuleListrcrblocksr*poolerr2r4rUrVr7r feature_infoweight_init_mode init_weights)r6rGrrrHrrIrrorrrrprr,rJrrKrLrMrrNr%r&r7r@dprr_redr8s r9r.zGemma4VitEncoder.__init__|so4 / 99 ]L[O \ ]9&ENNND1DN !""'$J/#6 #6 Ef!E , $;    2 !  "'>5!IJAqvvxJJmm5\%   #!)"3!((a&#$7  %   &)! 3     U[[-Ib-I J  ekk).Jr.J K DM!DN 4??#dijodpq_`T)tgVWUXMZq+6&+@k &   %  0 !WK% Lrs#H!H"H'moder^c2|xs |j}|dvsJ|j)tjj |j|j )tjj |j tt|||y)aInitialize model weights. Args: mode: Init mode. '' applies trunc-normal-TF Linear init; 'reset' only calls ``reset_parameters`` on each sub-module. needs_reset: If True, call ``reset_parameters`` on modules that have one (for post-``to_empty()`` reinit). Set to False during ``__init__`` since modules already self-initialize there. )rSr\Nr]) rlrUr/initzeros_rVones_rget_init_weights_gemma4_vitr6rpr^s r9rmzGemma4VitEncoder.init_weightsss,t,,}$$$ == $ GGNN4== ) >> % GGMM$.. )/+NPTUr:cdhS)Nz$patch_embed.position_embedding_tabler,r?s r9no_weight_decayz Gemma4VitEncoder.no_weight_decays 677r:c|jSzHReturn the 2-tuple patch size. For NaFlex dataloader / transform wiring.)rr?s r9get_patch_sizezGemma4VitEncoder.get_patch_sizesr:coarsec tdddgS)Nz^patch_embed|^rotary_emb)z^blocks\.(\d+)N)z ^pooler|^std_)istemrirr6r|s r9 group_matcherzGemma4VitEncoder.group_matcher s,-/KL  r:enablec||_yrB)rcr6rs r9set_grad_checkpointingz'Gemma4VitEncoder.set_grad_checkpointings "(r:enabledc^|jD]}t|ts||_y)aToggle the ``Gemma4ClippableLinear`` clamp ops. Gemma4's pretrained E4B checkpoint ships finite clamp buffers on every projection, which can saturate and stall gradient flow during classifier fine-tuning. Setting ``enabled=False`` skips both the input and output clamps in the forward pass (buffers are left untouched so the call is reversible and checkpoint-safe). N)modulesrrr!)r6rmods r9set_clamp_enabledz"Gemma4VitEncoder.set_clamp_enableds+<<> *C#45") *r:r@c |jdk7s|jdk7ry|jdd\}}|j\}}|j}||z||z}}||zdk7s||zdk7rt d|d|d|d|d |d|d y) zWhen using the soft-token pooler, raw-image H/W must divide by ``patch_size * pooling_kernel_size`` so the pool cell grid is integral. Pre-patchified / NaFlex inputs are assumed to be conformant already. rrPNrrz Image size (z, z=) must be divisible by (patch_size * pooling_kernel_size) = (z3) when global_pool='soft'. Resize to multiples of (z#), or use global_pool='avg'/'none'.)r`rHrUrr,r) r6r@rrrrrjcell_hcell_ws r9_assert_raw_img_conformantz+Gemma4VitEncoder._assert_raw_img_conformant$s 66Q;$**f4 wwrs|1B  $ $aa v:?a&jAoqcA3'99?6(K++1("VH<_a .r:rrblock_callbackmax_block_indexc.|jdd\}}|j||\}} d} |jrUtj|dd||j |j } | j|ddddddftd| |j} n|jd|dz} t| D]W\} } |jr.tjjst| ||| | }n | ||| | }|O|| |Y|S)z?RoPE + transformer-block pipeline over already-embedded tokens.NrRrr$z-infr')rUrerr3zerosr%r& masked_fill_r>ri enumeratercr  is_scriptingr)r6r@rrrrrrrrrrirblks r9_encodezGemma4VitEncoder._encode6swwr{1!__Q =(,0  " Aq!QqxxqwwOI  " "#4QdA5E#Ff V  "[[F[[!5/A"56F' %FAs&&uyy/E/E/GsAx9E8XC)q!$  %r:rrc|jt|ts|n|d|j|||\}}}|j |||S)z9Raw patch tokens pre-pool. Returns ``(B, N, embed_dim)``.r)rrrrdr)r6r@rrrrs r9forward_featuresz!Gemma4VitEncoder.forward_featuresWsQ ''At1D!I,W-1-=-=ak-Z*<*||A|->??r: pre_logitsctd)Nz;Gemma4VitEncoder does not support classification use cases.)NotImplementedError)r6r@rs r9 forward_headzGemma4VitEncoder.forward_headbs!"_``r:cZ|jt|ts|n|d|j|||\}}}|j |||}|j dk(r@|j |||\}}|j||jz |jz}|S|j dk(r||jrZ|j|jdd}|jd|jdd jd z }|S|jd}|S) uEncode + apply the configured pool. Output shape depends on ``self.global_pool``: ``'soft'`` → ``(B, num_soft_tokens, D)`` ``'avg'`` → ``(B, D)`` ``'none'`` → ``(B, N, D)`` (raw patch tokens, identical to forward_features) rrPrQrQrrrSTr2r)rrrrdrrHrjrUrVrrDr^rrCmean)r6r@rrrrrs r9rEzGemma4VitEncoder.forwardes+ ''At1D!I,W-1-=-=ak-Z*<* LLL*; <   v %;;q,0ABDAq}}(&$..8   & $$&MM"3"="=b"A3GEEaEL%6$6#;#;4#;#P#V#V[\#V#]] FFqFMr:indicesnorm stop_earlyraintermediates_only output_dictc j|dvsJ|dk(} tt|j|\} t|ts|n|d} | j } |j | |j|||\}}}gdtdtjddffd }d}|r tjjs| }|j||||| }| r| d k7r td |jd }t|d j!j#dz}t|dj!j#dz}Dcgc]6}|j%|||dj'd dddj)8c}| rdi}|s||d<||d<|S|rS|fScc}w)aForward features returning intermediates. Args: x: Input tensor ``(B, C, H, W)`` or NaFlex pre-patchified tensor/dict. patch_coord: ``(B, N, 2)`` patch coords in NaFlex external (y, x) convention. patch_valid: ``(B, N)`` boolean mask, True for valid tokens. indices: Block indices to return intermediates for. norm: Not used (no final norm in Gemma4 encoder). stop_early: Stop iterating after last needed intermediate. output_fmt: Output format ('NCHW' or 'NLC'). NCHW requires a fixed full grid. intermediates_only: Only return intermediate features. output_dict: Return a dict matching the NaFlexVit intermediates contract: 'image_intermediates' (+ 'image_features' unless ``intermediates_only``) and 'patch_valid' aligned with the token sequence (all-True for dense image input; gemma4 has no patch dropout, so it always equals the input validity). )NCHWrTrrryr"Nc2|vrj|yyrB)r)rr intermediates take_indicess r9_cbz3Gemma4VitEncoder.forward_intermediates.._cbsL $$Q'!r:)rrrz:output_fmt='NCHW' requires a raw image (B, C, H, W) input.rr1rr6rQrrRimage_intermediatesimage_featuresr)rlenrirrr`rrdrKr3rMr rrrrUr7rgrr contiguous)r6r@rrrrrrarrr max_indexrawraw_input_ndimrrrrrrrr result_dictrrs @@r9forward_intermediatesz&Gemma4VitEncoder.forward_intermediatess8_,,,&"6s4;;7G"Q i!!T*a)  '',-1-=-=ak-Z*<*,.  (3 (5<< (D ( eii446'O LL  +   " !]^^""1%A\&)--/4467!;B\&)--/4467!;B`mn[\QYYq"b"5==aAqITTVnM +@-*PK%01 ,-*;);K &   -os;F0 prune_norm prune_headcxtt|j|\}}|jd|dz|_|S)aPrune layers not required for specified intermediates. ``prune_head`` is accepted for API compatibility; the encoder has no classifier head so it's a no-op here. (The wrapping ``Gemma4VitClassifier`` handles its own head pruning.) Nr)rrri)r6rrrrrs r9prune_intermediate_layersz*Gemma4VitEncoder.prune_intermediate_layerss:#7s4;;7G"Q ikk.9q=1 r:)rrrrPrrrrN rrrrFFrrrNrSNNrSTFTr)NNNFFrFFrFT)#rGrHrIrJr rKr rrr>rLrr.r3r ignorermr rxr{rrrrrrMrrrrrErrrrNrOs@r9rrjsI&58 % *.%)" %+0'( %(-$'$'$',0!1m1CsCx01m1m1 m1  m1  m1m1m1m1#3-m1 #m1m1m1&)m1"%m1 !m1""&#m1$"%m1&"'m1(")m1* )+m1,-m12 3m1^ YYVVVVV, YY8S88 YYc3h YY D T#s(^   YY)T)T)) YY * * * *ELLT.MQ-1 || ,, %||  %XsELL.A4.G%HI  &c]  H3726 @U\\4U\\(9#::; @"%,,/ @"%,,/ @  @aellaaa 3726 U\\4U\\(9#::; "%,,/ "%,,/   J37267;$$', %J U\\4U\\(9#::;J "%,,/J "%,,/ J eCcN34 J  J J J !%J J  tELL!5tELL7I)I#JDQTVYQYNZ [J \./$# 3S >*  c r:c5eZdZdZ d;deeeeeffdededededed ed ed ed ed eedede de dedede de de de de de de dee deddf4fd Z e jjdd$e ddfd%Ze jjd>d&e ddfd'Ze jjdeeeffd(Ze jjdej4fd)Zd?dedeeddfd*Z d@d+ee j:eee j:ffd,ee j:d-ee j:de j:fd.Z dAd+e j:d-ee j:d/e de j:fd0Z d@d+ee j:eee j:ffd,ee j:d-ee j:de j:fd1Z dBd+ee j:eee j:ffd,ee j:d-ee j:d2eeee!efd3e d4e d5ed6e dee!e j:ee j:e!e j:fffd7Z" dCd2eee!efd8e d9e de!efd:Z#xZ$S)DruClassification wrapper around ``Gemma4VitEncoder``. Holds: - ``encoder``: a ``Gemma4VitEncoder``. By default constructed with ``global_pool=''`` so the classifier does its own masked-mean pool over raw patch tokens in ``forward_head``. Pass ``encoder_pool='soft'`` to turn on the native VLM spatial ``k×k`` soft-token pool inside the encoder; ``forward_features`` then returns ``(B, num_soft_tokens, D)`` already pooled (plus the ``√D`` scale and, for the 31B variant, ``std_bias/std_scale``). - ``norm``: optional ``RmsNorm`` after pooling. - ``head``: linear classifier. Input/output contract matches timm convention: ``forward_features`` returns pre-head features (pre-pool ``(B, N, D)`` patch tokens by default, or ``(B, num_soft_tokens, D)`` post-soft-pool when ``encoder_pool='soft'``); ``forward_head`` does classifier pool + norm + head; ``forward`` = ``forward_features`` + ``forward_head``. NrGrrrrH encoder_poolrrIrrorrrrprr,rJr final_norm drop_raterKrLrMrrNr"c(||d}t||dvs Jd|d|dvs Jd|d||_||_||_t d!id|d |d |d |d |d |d| d| d| d| d| d|d|d|d|d|d|d|d|d|d|||_|rt|f| dd|ntj|_ tj||_ |dkDrtj||fi|ntj|_|j jx|_|_|j j"|_|j j$|_|j j&|_|j j(|_|d k7rpt+|jtjrK|jj,4tj.j1|jj,yyyy)"Nr$rQrRrSBGemma4VitClassifier global_pool must be 'avg', 'none', or '' (got z>); use Gemma4VitEncoder directly for 'soft' VLM-style pooling.)rSrRrPzDGemma4VitClassifier encoder_pool must be '', 'none', or 'soft' (got z).rGrrrHrrIrrorrrrprr,rJrrKrLrMrrNFrrr[r,)r-r.rrHrrencoderrr/r%rr head_dropr0headr_r`rrrkrlrr'rrrs)r6rGrrrrHrrrIrrorrrrprr,rJrrrrKrLrMrrNr%r&r7r8s r9r.zGemma4VitClassifier.__init__sz< / 11 KPQ\P_`J K K133 fRS_Rbbd e f3&&((  !  %         & 0  " %< !4 $ !4! "*# $*% &*' ( ) *$-  0MWGIH8EHRH\^\g\g\i I.?JQBIIi;;TVT_T_Ta 59LL4M4MMD1//,,11 LL55 $ = = & $))RYY/DIINN4Ntyy~~.5O/ !r:rpr^c|jj||t|jtj rK|jj 4tjj|jj yyy)z>Init encoder + classifier-specific pieces (head bias to zero).rpr^N) rrmrrr/r0r'rrrsrvs r9rmz Gemma4VitClassifier.init_weightsLs[ !!t!E dii + 0J GGNN499>> *1K +r:cb|jjDchc]}d| c}Scc}w)Nencoder.)rrx)r6rjs r9rxz#Gemma4VitClassifier.no_weight_decaySs)(, (D(D(FG1(1#GGGs ,r|ctddgS)Nz*^encoder\.patch_embed|^encoder\.rotary_emb)z^encoder\.blocks\.(\d+)Nr~rrs r9rz!Gemma4VitClassifier.group_matcherWs>67  r:rc:|jj|yrB)rrrs r9rz*Gemma4VitClassifier.set_grad_checkpointing_s ++F3r:rc:|jj|yrB)rr)r6rs r9rz%Gemma4VitClassifier.set_clamp_enabledcs &&w/r:c6|jjSrz)rr{r?s r9r{z"Gemma4VitClassifier.get_patch_sizegs||**,,r:c|jSrB)rr?s r9get_classifierz"Gemma4VitClassifier.get_classifierls yyr:c||_||dvs Jd|d||_|dkDr tj|j|ntj |_t|j tjrK|j j4tjj|j jyyy)Nrr)r) rrHr/r0rr%rrr'rrrs)r6rrHs r9reset_classifierz$Gemma4VitClassifier.reset_classifierps&  ""55 fTU`Tccde f5*D >IAoBIIdnnk:SUS^S^S` dii + 0J GGNN499>> *1K +r:r@rrc|jdk(r|j|||S|jj|||S)u]Pre-head features. Shape depends on ``encoder_pool``: ``''`` / ``'none'``: raw patch tokens ``(B, N, embed_dim)``. ``'soft'``: spatial ``k×k`` soft-token pool output ``(B, num_soft_tokens, embed_dim)`` (with ``√D`` scale + optional ``std_bias/std_scale`` baked in by the encoder). rPrr)rrr)r6r@rrs r9rz$Gemma4VitClassifier.forward_featureszsD    &<<{ <T T||,,QKU`,aar:rc|jdk(r||jdk(s||jd}nX|j|j dd}|j d|j ddj d z }|j|}|j|}|r|S|j|S) uPool (if configured) → norm → head_drop → head. Args: x: pre-head features from ``forward_features``. patch_valid: ``(B, N)`` valid mask for masked mean over raw patch tokens. Ignored when ``encoder_pool='soft'`` (the soft pooler has already collapsed padding tokens, and the new token count no longer aligns with ``patch_valid``). pre_logits: If True, return pre-classifier features. rQrPrrSrQrTr2r) rHrrrDr^rrCrrr)r6r@rrs r9rz Gemma4VitClassifier.forward_heads   u $  F*k.AFFqFMMMK<":":2">DEEaEL;??q$?#G#M#MRS#M#TT IIaL NN1 q0DIIaL0r:ct|tr)|jd|}|jd|}|d}|*|(|dk(jd}|j r|}|j |||}|j ||S)NrrrrQrSr)r)rrrrrrr)r6r@rrrfeatss r9rEzGemma4VitClassifier.forwards a %% {;K%% {;K) A  ;#:#r)..2.6H||~'i %%a[k%Z  K @@r:rrrrarc H|jj||||||||S)N)rrrrrrar)rr) r6r@rrrrrrars r9rz)Gemma4VitClassifier.forward_intermediatess8||11 ##!!12  r:rrch|jj||d}|r|jd|S)NF)rrr)rrr)r6rrrrs r9rz-Gemma4VitClassifier.prune_intermediate_layerss8 ||==gR\in=o   ! !! $r:)rrrirQrSrrrrNrrrrrFFTrrrrNrSNNrrrrBr)NF)NNNFFrFr)%rGrHrIrJr rKr rrr>rLrr.r3r rrmr rxrrrrrr{r/ModulerrrMrrrErrrrNrOs@r9rrs6,58 #$ " *.%)" %+0'( %(-#"$'$'$',0!9Q/CsCx01Q/Q/ Q/  Q/  Q/Q/Q/Q/Q/Q/#3-Q/ #Q/Q/Q/ &)!Q/""%#Q/$%Q/&"&'Q/()Q/*+Q/,"-Q/."/Q/0"1Q/2 )3Q/45Q/: ;Q/f YY+++++  YYHSHH YY D T#s(^   YY4T4T44 YY0000 YY-c3h-- YY +C+hsm+W[+3726 bU\\4U\\(9#::;b"%,,/b"%,,/ b  b*37$ 1||1"%,,/1 1  1<3726 AU\\4U\\(9#::;A"%,,/A"%,,/ A  A837267;$$', U\\4U\\(9#::; "%,,/ "%,,/  eCcN34      !%  tELL!5tELL7I)I#JJ K 0./$# 3S >*    c r:TrZnamer^cNt|tjrNt|jd|j *tj j|j yyt|dr|jy|rt|dr|jyyy)aPer-module init for Gemma4Vit (trunc-normal-TF for Linear weights). Args: module: Module to initialize. name: Dotted module name (from ``named_apply``). needs_reset: If True, call ``reset_parameters`` on modules that define one. rrNrmr5) rr/r0rrr'rrrshasattrrmr5)rZrr^s r9init_weights_gemma4_vitrsy&"))$D1 ;; " GGNN6;; ' #  ( );<!=r:rpc$tt|S)Nr])rrrs r9rurus * DDr: state_dictmodelci}d}|jD]\}}d}|D]}|j|s|}n||jdr|||<=|t|d}d|vrP|jdd}|jdd}|jd d }|jd d }|jd d}|jdd}|jdd}|||<|S)uConvert HuggingFace Gemma4 vision encoder weights → ``Gemma4VitEncoder`` keys. Pure key remapping — no value transforms. The encoder consumes patches in HF's native C-P-P flat layout (via ``batch_patchify(channels_last=False)`` in ``Gemma4PatchEmbed``), so ``input_proj.weight`` passes through unchanged. ``Gemma4ClippableLinear`` preserves HF's ``.linear.weight`` + clamp-buffer structure, so those pass through too. )zmodel.vision_tower.zmodel.vision_model.z vision_model.z vision_tower.N) patch_embed.rWstd_zpooler.z rotary_emb.rezpatch_embedder.rzencoder.layers.rWz.input_layernorm.z.norm1.z.post_attention_layernorm.z.norm2.z.pre_feedforward_layernorm.z.norm3.z.post_feedforward_layernorm.z.norm4.z .self_attn.z.attn.)items startswithrreplace) rrout_dict hf_prefixesrjrmatched_prefixprefixnew_ks r9checkpoint_filter_fn_encoderrsHbK  "1! F||F#!'   !||YZ #n%&' 5  /@ /; 19= :IF ;YG #&"!  .' ( ) r:z"timm/gemma4_vit_167m.gemma4_e4b_itzencoder.patch_embed.input_projz apache-2.0) hf_hub_idrlicensez"timm/gemma4_vit_570m.gemma4_31b_itz&timm/gemma4_vit_167m_enc.gemma4_e4b_it)rrz&timm/gemma4_vit_570m_enc.gemma4_31b_it)zgemma4_vit_167m.gemma4_e4b_itzgemma4_vit_570m.gemma4_31b_itz!gemma4_vit_167m_enc.gemma4_e4b_itz!gemma4_vit_570m_enc.gemma4_31b_itrrrrr)rrIrrorrJriHi)rrIrrorrJc Rttd}tdd|it|fi|S)zGemma4 ~167M (E2B/E4B vision tower) classifier. Masked mean pool over patch tokens + norm + linear classifier. Output: ``(B, num_classes)``. For the native VLM encoder interface (soft-token output), use ``gemma4_vit_167m_enc``. Trr)gemma4_vit_167m)r _167M_ARCHr rr model_argss r9rrs0jT2J ( pz pUYZdUohnUo ppr:c Rttd}tdd|it|fi|S)u Gemma4 ~167M (E2B/E4B vision tower) — native VLM encoder. ``global_pool='soft'`` applies the spatial ``k×k`` soft-token pool + √D scale; output: ``(B, num_soft_tokens, embed_dim)``. Bit-perfect with HF ``Gemma4VisionModel`` on matching weights. rPrHr)gemma4_vit_167m_enc)rrrrs r9r#r#0jf5J % q qVZ[eVpioVp qqr:c Rttd}tdd|it|fi|S)agGemma4 ~570M (26B/31B vision tower) classifier. Masked mean pool over patch tokens + norm + linear classifier. Output: ``(B, num_classes)``. The classifier uses ``encoder.forward_features`` and does its own pool + norm; the encoder's ``std_bias/std_scale`` (which only applies in ``'soft'`` pool mode) is not used on the classifier path. Trr)gemma4_vit_570m)r _570M_ARCHr rs r9r&r&s0jT2J ( pz pUYZdUohnUo ppr:c Rttd}tdd|it|fi|S)uGemma4 ~570M (26B/31B vision tower) — native VLM encoder. ``global_pool='soft'`` output ``(B, num_soft_tokens, embed_dim)`` with ``std_bias/std_scale`` standardization applied post-pool (HF-native ordering — bit-perfect with ``Gemma4VisionModel``). rPr"r)gemma4_vit_570m_enc)rr'rrs r9r)r)r$r:)rR)rRrRrr)rS)IrJmath functoolsrtypingrrrrrr r r r3torch.nnr/torch.nn.functional functionalr timm.datar r timm.layersrrrrr_builderr _featuresr _manipulaterr _registryrr naflexvitr__all__rrrMrYrKr_rlrnrrr rr*rrrrLrrurrrr r default_cfgsrrr'rr#r&r)r,r:r9r9s  III AVV*+0<% 4 5*BII*Z(5<<(ELL( . << . \\ .\\ . .  \\ .  & <<& \\&\\& &  &  \\ &4MbiiM`O2ryyO2dcbiicLXRYYX8F"))FR@*@*Fwryywt {")){|"BII"S"D"\`"$EcETEXE ,ell*+,, #u|| ,^ell*+$ #u|| 0   Td  3 D Wj cT#s(^2%%)63& &*63& *.:**.:*#& 0       qq;NqqrDr?Orr q q;N q qrDr?Orrr: