^j dZddlZddlZddlmZmZmZmZm Z m Z ddl Z ddl m Z ddl mZmZmZddlmZmZddlmZddlmZdd lmZmZmZmZdd lmZdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z&e$jNe(Z)d ee*de*de*dee*de ee*e jVff dZ,deee-dee ee*e e*dffdee-de ee-ee*ffdZ.deee-dee ee*e e*dffdee-de ee-ee*ffdZ/dZ0GddeeZ1Gdde jdZ3Gdde jdZ4Gdd e jdZ5Gd!d"e5Z6Gd#d$e jdZ7Gd%d&e jdZ8Gd'd(e8Z9Gd)d*e jdZ:dId+e jvd,eGd/d0e jdZ?Gd1d2e jdZ@Gd3d4e jdZAe8e9d5ZBGd6d7e jdZCGd8d9e jdZDGd:d;eZEdeFGd?d@eEZHdAZIe"dBeFGdCdDeEZJe"dEeFGdFdGeEeZKgdHZLy)JaDINOv2-with-Registers backbone with windowed self-attention. This module is a local copy of the HuggingFace Transformers DINOv2-with-Registers implementation, extended with windowed attention support for RF-DETR. It targets the transformers v5 API (``transformers>=5.0.0``). Transformers v5 API changes vs v4 ---------------------------------- ``head_mask`` removed: The ``head_mask`` parameter that appeared on every ``forward()`` in v4 has been dropped in v5. It defaulted to ``None`` throughout the call chain and callers universally passed ``None``, so removing it produces **identical numerics**. Permanent head pruning is still available via ``model._prune_heads()``. ``BackboneMixin._init_transformers_backbone`` signature: In v4 this method accepted ``(self, config)``. In v5 it accepts only ``(self)``; the config is accessed via ``self.config`` internally. Helper functions copied locally: ``get_aligned_output_features_output_indices`` and ``find_pruneable_heads_and_indices`` were removed from the transformers v5 public API. Private copies (``_get_aligned_output_features_output_indices`` and ``_find_pruneable_heads_and_indices``) are kept in this module. N)DictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)BackboneConfigMixin BackboneMixin)ACT2FN)PretrainedConfig)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)PreTrainedModel)prune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings torch_intheadsn_heads head_sizealready_pruned_headsreturncTtj||}t||z }|D]tfd|Dzd|< |j dj j d}tjt||j}||fS)alReturn the set of pruneable heads and their index mask for weight pruning. Copied from transformers.pytorch_utils.find_pruneable_heads_and_indices (removed from public API in transformers v5.0). Source: https://github.com/huggingface/transformers/blob/v4.49.0/src/transformers/pytorch_utils.py#L127 MAINTENANCE: if this function is moved to another module or deleted, update the "Copyright 2022 The HuggingFace Team" line in the file header accordingly. Args: heads: Indices of heads to prune. n_heads: Total number of heads in the layer. head_size: Size of each attention head. already_pruned_heads: Heads that have already been pruned. Returns: A tuple of (heads, index) where heads is the adjusted set of head indices and index is a LongTensor boolean mask selecting the remaining weights. c30K|] }|krdndyw)rN).0hheads {/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/rfdetr/models/backbone/dinov2_with_windowed_attn.py z4_find_pruneable_heads_and_indices..WsGQTAq(Gsrr#) torchonessetsumview contiguouseqarangelenlong)rrrrmaskindexr's @r(!_find_pruneable_heads_and_indicesr7?s* ::gy )D J- -E G2FGGGT  99R= # # % ( ( +D LLT #D ) . . 0E %< out_features out_indices. stage_namesc||t|dz g}|dg}||fS|$|"|Dcgc]}|j|}}||fS|||Dcgc]}|| }}||fScc}wcc}w)Nr#r*)r3r6)r9r:r;layeridxs r(%_align_output_features_output_indicesr?^s |3;'!+, #B(  $$  !9=IJE{((/J J  $$  +"94?@S C(@ @  $$K@s A A#cJ| t|nd}t|||\}}||fS)aAlign out_features and out_indices against stage_names, filling in defaults when either is None. Copied from transformers.utils.backbone_utils.get_aligned_output_features_output_indices (removed from public API in transformers v5.0). Source: https://github.com/huggingface/transformers/blob/v4.49.0/src/transformers/utils/backbone_utils.py#L30 MAINTENANCE: if this function is moved to another module or deleted, update the "Copyright 2023 The HuggingFace Inc. team" line in the file header accordingly. Args: out_features: Names of the backbone stages to return features from, or None to derive from out_indices. out_indices: Integer indices of the stages to return features from, or None to derive from out_features. stage_names: Ordered list of all stage names defined by the backbone config. Returns: A tuple of (out_features, out_indices) with both fields populated consistently. Nr9r:r;)listr?rAs r(+_get_aligned_output_features_output_indicesrCms8*(3'>${#DK E!{ !L+  $$r8!WindowedDinov2WithRegistersConfigcXeZdZdZdZ dfd ZxZS)rDaS This is the configuration class to store the configuration of a [`Dinov2WithRegistersModel`]. It is used to instantiate a Dinov2WithRegisters model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the DINOv2 with Registers [facebook/dinov2-with-registers-base](https://huggingface.co/facebook/dinov2-with-registers-base) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the encoder layers and the pooler layer. num_hidden_layers (`int`, *optional*, defaults to 12): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer encoder. mlp_ratio (`int`, *optional*, defaults to 4): Ratio of the hidden size of the MLPs relative to the `hidden_size`. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported. hidden_dropout_prob (`float`, *optional*, defaults to 0.0): The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. layer_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the layer normalization layers. image_size (`int`, *optional*, defaults to 224): The size (resolution) of each image. patch_size (`int`, *optional*, defaults to 16): The size (resolution) of each patch. num_channels (`int`, *optional*, defaults to 3): The number of input channels. qkv_bias (`bool`, *optional*, defaults to `True`): Whether to add a bias to the queries, keys and values. layerscale_value (`float`, *optional*, defaults to 1.0): Initial value to use for layer scale. drop_path_rate (`float`, *optional*, defaults to 0.0): Stochastic depth rate per sample (when applied in the main path of residual layers). use_swiglu_ffn (`bool`, *optional*, defaults to `False`): Whether to use the SwiGLU feedforward neural network. num_register_tokens (`int`, *optional*, defaults to 4): Number of register tokens to use. out_features (`List[str]`, *optional*): If used as backbone, list of features to output. Can be any of `"stem"`, `"stage1"`, `"stage2"`, etc. (depending on how many stages the model has). If unset and `out_indices` is set, will default to the corresponding stages. If unset and `out_indices` is unset, will default to the last stage. Must be in the same order as defined in the `stage_names` attribute. out_indices (`List[int]`, *optional*): If used as backbone, list of indices of features to output. Can be any of 0, 1, 2, etc. (depending on how many stages the model has). If unset and `out_features` is set, will default to the corresponding stages. If unset and `out_features` is unset, will default to the last stage. Must be in the same order as defined in the `stage_names` attribute. apply_layernorm (`bool`, *optional*, defaults to `True`): Whether to apply layer normalization to the feature maps in case the model is used as backbone. reshape_hidden_states (`bool`, *optional*, defaults to `True`): Whether to reshape the feature maps to 4D tensors of shape `(batch_size, hidden_size, height, width)` in case the model is used as backbone. If `False`, the feature maps will be 3D tensors of shape `(batch_size, seq_len, hidden_size)`. Example: >>> from rfdetr.models.backbone.dinov2_with_windowed_attn import WindowedDinov2WithRegistersConfig >>> # Initializing a tiny configuration suitable for doctests >>> configuration = WindowedDinov2WithRegistersConfig( ... image_size=32, ... patch_size=16, ... hidden_size=32, ... num_hidden_layers=2, ... num_attention_heads=4, ... num_register_tokens=2, ... ) >>> configuration.hidden_size 32 dinov2_with_registersc *t|di|||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ | |_||_||_||_||_dgt'd|dzDcgc]}d| c}z|_t+|||j(\|_|_||_||_||_|t7t'|n||_||_ycc}w)Nstemr#stagerAr$)super__init__ hidden_sizenum_hidden_layersnum_attention_heads mlp_ratio hidden_acthidden_dropout_probattention_probs_dropout_probinitializer_rangelayer_norm_eps image_size patch_size num_channelsqkv_biaslayerscale_valuedrop_path_rateuse_swiglu_ffnnum_register_tokensranger;rC _out_features _out_indicesapply_layernormreshape_hidden_states num_windowsrBwindow_block_indexesgradient_checkpointing)selfrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r9r:r`rarbrcrdkwargsr> __class__s r(rKz*WindowedDinov2WithRegistersConfig.__init__s08 "6"&!2#6 "$#6 ,H)!2,$$(  0,,#6 "8aIZ]^I^@_&`se}&``0[%;DL\L\1 -D- /%:"&.B.JD() *Pd !'=#'as D)i rhgelurkg{Gz?gư>T?rkFriNNTTr#NF)__name__ __module__ __qualname____doc__ model_typerK __classcell__rgs@r(rDrDsePd)J%("!$39=9=r8cZeZdZdZfdZdej dej fdZxZS)"Dinov2WithRegistersPatchEmbeddingszThis class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer.ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nr#r) kernel_sizestride)rJrKrUrVrWrL isinstance collectionsabcIterable num_patchesr Conv2d projection)reconfigrUrVrWrLrrgs r(rKz+Dinov2WithRegistersPatchEmbeddings.__init__#s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hir8 pixel_valuesr c|jd}||jk7rtd|jd|d|j|j dj dd}|S)Nr#zoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .)shaperW ValueErrorrflatten transpose)rerrW embeddingss r(forwardz*Dinov2WithRegistersPatchEmbeddings.forward2sz#))!, 4,, ,!../yaI __\2::1=GG1M r8) rprqrrrsrKr+Tensorrrurvs@r(rxrxs) jELLU\\r8rxceZdZdZdeddffd Zdejdededejfd Z d d ejd e ejdejfd Z xZ S)%WindowedDinov2WithRegistersEmbeddingszTConstruct the CLS token, mask token, register tokens, position and patch embeddings.rr Nct|tjt j dd|j |_tjt jd|j |_ |jdkDr>tjt jd|j|j nd|_ t||_ |jj}tjt j d|dz|j |_tj |j"|_|j&|_||_y)Nr#r)rJrKr Parameterr+randnrL cls_tokenzeros mask_tokenr\register_tokensrxpatch_embeddingsrposition_embeddingsDropoutrQdropoutrVr)rerrrgs r(rKz.WindowedDinov2WithRegistersEmbeddings.__init__@s ekk!Q8J8J&KL,,u{{1f6H6H'IJ))A- LLQ(B(BFDVDVW X  !C6 J++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++ r8rheightwidthcn|jddz }|jjddz }tjj s||k(r||k(r |jS|jdddf}|jddddf}|jd}||j j z}||j j z}t|dz} |jd| | |}|jdddd}|j} tjj|jtjt|t|fd d |j j"d k7 j| }tjj sAt%||jd k7st%||jdk7r t'd|jddddj)dd|}tj*|j-d|fdS)aThis method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This implementation supports torch.jit tracing while maintaining backwards compatibility with the original implementation. Adapted from: - https://github.com/facebookresearch/dino/blob/main/vision_transformer.py - https://github.com/facebookresearch/dinov2/blob/main/dinov2/models/vision_transformer.py r#Nrr*g?rnrdtypebicubicFmps)sizemode align_corners antialiaszHWidth or height does not match with the interpolated position embeddingsdim)rrr+jit is_tracingrrVrreshapepermuterr functional interpolatetofloat32devicetypeintrr/cat unsqueeze) rerrrr num_positionsclass_pos_embedpatch_pos_embedrsqrt_num_positions target_dtypes r(interpolate_pos_encodingz>WindowedDinov2WithRegistersEmbeddings.interpolate_pos_encodingQs!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221a48221ab59r"4;;111///'}c'9:)11!5GI[]`a)11!Q1=',, --33   U]]  3F#Yu%56%,,11U: 4 "<" yy##%6{o33B773u:I^I^_aIb;b !kll*11!Q1=BB1b#Nyy/33A6HaPPr8rbool_masked_posc N|j\}}}}|j|jjz}||zdk7s||zdk7r9t d|jd|jjd|d|d|d |j j jj}|j |j|} |Xtj|jd |jj| jjd| } |jj|d d } tj | | fd } | |j#| ||z} |jjd kDr||jjz} ||jjz} | d d d d f} | d d d d f}|j%|| | d }| |jjz}| |jjz}|jj}|j'||z|||d }|j)dd d dd}|j'||d zz||zd }| j+|d zd d }tj ||fd } |jj,dkDrTtj | d d d d f|j.j| jdd d | d d d d ffd n| } |j1| } | S)a"Compute windowed patch embeddings for the given pixel values. Args: pixel_values: Image tensor of shape ``(B, C, H, W)``. Both ``H`` and ``W`` must be divisible by ``patch_size * num_windows``. bool_masked_pos: Optional boolean mask of shape ``(B, num_patches)``. Masked positions are replaced with the learnable ``mask_token``. Returns: Patch embedding tensor. When ``num_windows > 1`` the batch dimension is expanded to ``B * num_windows ** 2`` and the sequence length corresponds to patches within a single window (plus CLS token and any register tokens). Raises: ValueError: If ``H`` or ``W`` is not divisible by ``patch_size * num_windows``. rzHInput spatial dimensions must be divisible by patch_size * num_windows (z * z = z), but got height=z, width=rrNr*r#rrrnri)rrVrrbrrrweightrrr+whererrrexpandrrr/rrrepeatr\rr)rerr batch_size_rrdivisorrr cls_tokens num_h_patches num_w_patchescls_token_with_pos_embedpixel_tokens_with_pos_embednum_w_patches_per_windownum_h_patches_per_windowrbwindowed_pixel_tokens!windowed_cls_token_with_pos_embeds r(rz-WindowedDinov2WithRegistersEmbeddings.forwardsA$(4'9'9$ Avu//DKK$;$;; G q EGOq$8OO$C (?(?'@G9M""(%;  ,,77>>DD ** YY J7Q?  $"?"? FTY"ZZ ;; " "Q &"dkk&<&<