^jdnjddlmZmZddlmZddlZddlmZddlmZ ddl m Z ddl m Z dd lmZdd lmZmZmZdd lmZmZdd lmZdd lmZmZmZmZddlmZm Z ddl!m"Z"ddl#m$Z$GddejJZ&GddejJZ' d9dejJdejPdejPdejPdejPdzde)dzde)deefdZ*Gdd ejJZ+Gd!d"ejJZ,Gd#d$eZ-eGd%d&eZ.Gd'd(ejJZ/eGd)d*e.Z0ed+,Gd-d.e.Z1ed/,Gd0d1e.Z2ed2,eGd3d4eZ3ed5,Gd6d7e.Z4gd8Z5y):)CallableIterable) dataclassN)nn)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputWithPoolingImageClassifierOutputMaskedImageModelingOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstring torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) DeiTConfigc`eZdZdZdeffd ZdejdejfdZxZ S)DeiTPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. configct||j}|j}t |t r|n||f}t |t r|n||f}|d|dz|d|dzz|_||_||_|j|_tj|j|j|||_ y)Nrr) kernel_sizestride) super__init__ image_size patch_size isinstancer num_patches num_channelsrConv2d hidden_size projection)selfrr#r$ __class__s q/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/deit/modeling_deit.pyr"zDeiTPatchEmbeddings.__init__/s && && #-j(#CZ*V`Ia #-j(#CZ*V`Ia &qMZ]:z!}PZ[\P]?]^$$"//))F$7$79K9KYclvw pixel_valuesreturnc|jd}||jk7rtd|jd|d|j|j dj ddS)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .)shaper' ValueErrorr*flatten transpose)r+r/r's r-forwardzDeiTPatchEmbeddings.forward<su#))!, 4,, ,!../yaI |,44Q7AA!QGGr.) __name__ __module__ __qualname____doc__rr"torchTensorr8 __classcell__r,s@r-rr(s4 xz xHELLHU\\Hr.rc eZdZdZddededdffd Zdejde d e dejfd Z dd ejd ejdzd edejfdZ xZ S)DeiTEmbeddingsa Construct the CLS token, distillation token, position and patch embeddings. Optionally, also the mask token. Differences from ViTEmbeddings: - Adds a distillation token (for distillation pre-training). - Position embeddings include +2 slots (CLS + distillation) instead of +1. - interpolate_pos_encoding handles 2 special tokens instead of 1. - forward concatenates distillation token and handles position encoding for both. ruse_mask_tokenr0Nct|tjt j dd|j |_|r4tjt j dd|j nd|_t||_ |jj}tjt j d|dz|j |_ tj|j|_|j |_|jj"|_tjt j dd|j |_y)Nrr3)r!r"r Parameterr=zerosr) cls_token mask_tokenrpatch_embeddingsr&position_embeddingsDropouthidden_dropout_probdropoutr$r#distillation_token)r+rrCr&r,s r-r"zDeiTEmbeddings.__init__Qs ekk!Q8J8J&KLQ_",,u{{1a9K9K'LMei 3F ;++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++//::"$,,u{{1aASAS/T"Ur. embeddingsheightwidthc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and 2 class embeddings. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rr3N?rrbicubicF)sizemode align_cornersdim)r4rJr=jit is_tracingr$rreshapepermuter functional interpolateviewcat) r+rOrPrQr& num_positionsclass_and_dist_pos_embedpatch_pos_embedrZ new_height new_widthsqrt_num_positionss r-interpolate_pos_encodingz'DeiTEmbeddings.interpolate_pos_encoding^sb!&&q)A- 0066q9A= yy##%+*F6UZ?++ +#'#;#;ArrE#B 221ab59r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy2OD!LLr.r/bool_masked_posric |j\}}}}|j|}|j\}} }|K|jj || d} |j dj | } |d| z z| | zz}|jj |dd} |jj |dd} tj| | |fd}|r||j|||z}ne||jdk7s||jdk7r2td|d|d|jdd|jdd ||jz}|j|}|S) NrSg?rrYrzInput image size (*z) doesn't match model (z).)r4rIrVrHexpand unsqueezetype_asrGrNr=rbrir#r5rJrM)r+r/rjri_rPrQrO batch_size seq_length mask_tokensmask cls_tokensdistillation_tokenss r-r8zDeiTEmbeddings.forwards +001fe**<8 $.OO$5! J  &//00ZLK",,R088ED#sTz2[45GGJ^^**:r2> "55<&>>J\\*- r.)F)NF)r9r:r;r<rboolr"r=r>intri BoolTensorr8r?r@s@r-rBrBFs Vz V4 VD V&M5<<&M&MUX&M]b]i]i&MV48). ll ))D0 #'   r.rBmodulequerykeyvalueattention_maskscalingrMkwargsc ||jddz}tj||jdd|z}|||z}tj j |dtjj|j}tj j|||j}tj||} | jddj} | |fS)NrSr3r)rZdtype)ptrainingr) rVr=matmulr7rr_softmaxfloat32torrMr contiguous) rzr{r|r}r~rrMr attn_weights attn_outputs r-eager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r.c eZdZdeffd Z d dej dej dzdeede ej ej ffdZ xZ S) DeiTAttentionrc(t|||_|j|_t |d|j |jz|_|j|_|j dz|_ d|_ tj|j |j|j z|j|_tj|j |j|j z|j|_tj|j |j|j z|j|_tj|j|j z|j d|_y)Nhead_dimrF)biasT)r!r"rnum_attention_headsgetattrr)rattention_probs_dropout_probattention_dropoutr is_causalrLinearqkv_biasq_projk_projv_projo_projr+rr,s r-r"zDeiTAttention.__init__s,  #)#=#=  F4F4F&JdJd4de !'!D!D}}d* ii 2 2F4N4NQUQ^Q^4^eketetu ii 2 2F4N4NQUQ^Q^4^eketetu ii 2 2F4N4NQUQ^Q^4^eketetu ii : :T]] JFL^L^eij r.N hidden_statesr~rr0c |jdd}g|d|j}|j|j|j dd}|j |j|j dd}|j |j|j dd}tj|jjt} | |||||f|jsdn |j|jd|\} } | jg|dj!} |j#| } | | fS)NrSrr3)rMr)r4rrrar7rrr get_interfacer_attn_implementationrrrrr]rr) r+rr~r input_shape hidden_shape query_states key_states value_statesattention_interfacerrs r-r8zDeiTAttention.forwardsK $))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r.N) r9r:r;rr"r=r>rrtupler8r?r@s@r-rrsf kz k"/3)||) t+)+, ) u||U\\) * )r.rc\eZdZdeffd Zdej dej fdZxZS)DeiTMLPrct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) r!r"rr hidden_act activation_fnrrr)intermediate_sizefc1fc2rs r-r"zDeiTMLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr.rr0cl|j|}|j|}|j|}|Sr)rrr)r+rs r-r8zDeiTMLP.forwards4/ **=9 / r. r9r:r;rr"r=r>r8r?r@s@r-rrs,KzKU\\ellr.rc eZdZdeffd Z d dej dej dzdeedej fdZ xZ S) DeiTLayerrcjt|t||_t j |j |j|_t j |j |j|_ t||_ t j|j|_y)Neps)r!r"r attentionr LayerNormr)layer_norm_epslayernorm_beforelayernorm_afterrmlprKrLrMrs r-r"zDeiTLayer.__init__sy &v. " V-?-?VEZEZ [!||F,>,>FDYDYZ6?zz&"<"<= r.Nrr~rr0c |}|j|}|j||fi|\}}|j|}||z}|}|j|}|j |}|j|}||z}|Sr)rrrMrr)r+rr~rresidualrps r-r8zDeiTLayer.forwards!--m< )4>>-R6R q ]3 %0 !,,]; /  ]3 %0 r.r) r9r:r;rr"r=r>rrr8r?r@s@r-rrsV>z>/3|| t++,   r.rceZdZUeed<dZdZdZdZddgZ dZ dZ dZ dZ dZeedZd Zej(d ej,ej.zej0zd d ffd ZxZS)DeiTPreTrainedModelrdeitr/)imageTrBr)r attentionsrIrzr0Nct||t|tr|j6t j |jd|jjt j |jd|jj|jt j|jt|trt j|jt j|jt j|j|j t j|jyyy)zInitialize the weightsNr)meanstd) r! _init_weightsr%rBrJinit trunc_normal_rinitializer_rangerGrHzeros_rN)r+rzr,s r-rz!DeiTPreTrainedModel._init_weights9s f% fn -))5""6#=#=CT[[MjMjk   v//ct{{?\?\ ]  , F--. fn - KK(( ) KK22 3 KK11 2  , F--.- .r.)r9r:r;r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backend_can_compile_fullgraphrr_can_record_outputs_input_embed_layerr=no_gradrrr(rrr?r@s@r-rr&s $O!&*#);7N"&!"#,U]]_/BII $9BLL$H/T//r.rc\eZdZdeffd Zdej dej fdZxZS) DeiTPoolerrct|tj|j|j |_t|j|_ yr) r!r"rrr)pooler_output_sizedenser pooler_act activationrs r-r"zDeiTPooler.__init__Ls> YYv1163L3LM  !2!23r.rr0c\|dddf}|j|}|j|}|S)Nr)rr)r+rfirst_token_tensor pooled_outputs r-r8zDeiTPooler.forwardQs6+1a40 #56  6 r.rr@s@r-rrKs*4z4 U\\ellr.rceZdZddededeffd Zeede dde jdzd e jdzd edzd e jdzd e e d ef dZxZS) DeiTModelFradd_pooling_layerrCct||||_t|||_t j t|jDcgc] }t|c}|_ t j|j|j|_|r t|nd|_|j#ycc}w)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )rCrN)r!r"rrBrOr ModuleListrangenum_hidden_layersrlayersrr)r layernormrpooler post_init)r+rrrCrpr,s r-r"zDeiTModel.__init__\s   (OmmfF^F^@_$`1Yv%6$`a f&8&8f>S>ST,=j(4   %asC)tie_last_hidden_statesNr/rjrir~rr0c |jjjjj}|j|k7r|j |}|j|||}t |j||}|}|jD] } | ||fi|}|j|} |j|j| nd} t| | S)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )rjri)r inputs_embedsr~N)last_hidden_state pooler_output) rOrIr*weightrrr rrrrr ) r+r/rjrir~rexpected_dtypeembedding_outputrlayersequence_outputrs r-r8zDeiTModel.forwardls 99DDKKQQ    /'??>:L?? /Tl+ 3;;*)  ) [[ KE!-J6JM K..78< 8O O4UY )O[hiir.)TFNNNN)r9r:r;rrwr"rrrr=r>ryrrr r8r?r@s@r-rrZszd[_  E2-13704.2 jllT) j))D0 j#'+ j  t+ j +, j $ j3  jr.rad DeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introceZdZdeffd Zee d dejdzdejdzde dejdzde e d e f d ZxZS) DeiTForMaskedImageModelingrcNt||t|dd|_t j t j |j|jdz|jzdt j|j|_ |jy)NFT)rrCr3r) in_channels out_channelsr) r!r"rrr Sequentialr(r)encoder_strider' PixelShuffledecoderrrs r-r"z#DeiTForMaskedImageModeling.__init__s  fdS }} II"..#22A58K8KK  OOF11 2    r.Nr/rjrir~rr0c h|j|f|||d|}|j}|ddddf}|j\}} } t| dzx} } |j dddj || | | }|j |} d}||jj|jjz}|j d||}|j|jjdj|jjdjdj}tjj|| d }||zj!|j!d zz |jj"z }t%|| |j&|j( S) a; bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```)rjrir~Nr3rTrrrSnone) reductiongh㈵>)lossreconstructionrr)rrr4rxr^r]rrr#r$repeat_interleavernrrr_l1_losssumr'rrr)r+r/rjrir~routputsrrqsequence_lengthr'rPrQreconstructed_pixel_valuesmasked_im_lossrVrtreconstruction_losss r-r8z"DeiTForMaskedImageModeling.forwardsL/8dii / +%=) /  / "33*!QR%04C4I4I1 O\_c122)11!Q:BB:|]cejk&*\\/%B"  &;;))T[[-C-CCD-55b$EO11$++2H2H!L""4;;#9#91=1  #%--"7"7 F`lr"7"s 1D8==?488:PTCTUX\XcXcXpXppN(5!//))   r.)NNFN)r9r:r;rr"rrr=r>ryrwrrrr8r?r@s@r-rrsz"-137)..2 J llT)J ))D0J #' J  t+ J +, J  #J J r.ra DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. Note that it's possible to fine-tune DeiT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ceZdZdeffd Zee d dejdzdejdzde dzdejdzde e d e f d Z xZS) DeiTForImageClassificationrc.t|||j|_t|d|_|jdkDr*t j |j|jnt j|_ |jyNF)rr) r!r" num_labelsrrrrr)Identity classifierrrs r-r"z#DeiTForImageClassification.__init__ss   ++f> OUN_N_bcNc"))F$6$68I8IJikititiv r.Nr/labelsrir~rr0c |j|f||d|}|j}|dddddf}|j|} d} ||j|| |jfi|} t | | |j |jS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rir~Nr)r logitsrr)rrr loss_functionrr rr) r+r/rrir~rrrrrr s r-r8z"DeiTForImageClassification.forwards"/8dii / %=)/  / "33'1a0 /  %4%%ffdkkLVLD$!//))   r.r)r9r:r;rr"rrr=r>rwrrr r8r?r@s@r-rrs z -1&*04.2 # llT)#  t## #'+ #  t+ # +, #  # # r.rzC Output type of [`DeiTForImageClassificationWithTeacher`]. ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e ej dzed<dZ e ej dzed<y)+DeiTForImageClassificationWithTeacherOutputaj logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). Nr cls_logitsdistillation_logitsrr) r9r:r;r<rr= FloatTensorrr#r$rrrr.r-r"r"Bs} (,FE   $++/J!!D(/48**T1859M5**+d2926Je''(4/6r.r"a DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet. .. warning:: This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported. c eZdZdeddffd Zee d dejdzde dejdzde e de f d Z xZS) %DeiTForImageClassificationWithTeacherrr0Nct|||j|_t|d|_|jdkDr*t j |j|jnt j|_ |jdkDr*t j |j|jnt j|_ |jyr) r!r"rrrrrr)rcls_classifierdistillation_classifierrrs r-r"z.DeiTForImageClassificationWithTeacher.__init__gs   ++f> AG@Q@QTU@UBIIf((&*;*; <[][f[f[h AG@Q@QTU@UBIIf((&*;*; <[][f[f[h $ r.r/rir~rc |j|f||d|}|j}|j|dddddf}|j|dddddf}||zdz } t | |||j |j S)Nrrrr3)rr#r$rr)rrr*r+r"rr) r+r/rir~rrrr#r$rs r-r8z-DeiTForImageClassificationWithTeacher.forwardxs/8dii / %=)/  / "33((Aq)AB "::?1aQR7;ST22a7:! 3!//))   r.)NFN)r9r:r;rr"rrr=r>rwrrr"r8r?r@s@r-r(r([szd"-1)..2  llT) #'  t+  +,  5   r.r()rr(rrr)Nr)6collections.abcrr dataclassesrr=rrr activationsr masking_utilsr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_deitrModulerrBr>floatrrrrrrrrrr"r(__all__r&r.r-r=s*/! &!69llF&OOI5*H"))H<`RYY`R!% II% <<% % << % LL4' % T\ %%'(%8.)BII.)b bii  *@!//!/!/H  4j#4j4jn  ^ !4^  ^ B  2 !42  2 j  7+7  7& 0 ,?0  0 f r.