^j\(dZddlZddlmZmZddlZddlmZddlmZ ddl m Z ddl m Z dd lmZdd lmZmZmZdd lmZmZdd lmZdd lmZmZmZmZddlmZm Z ddl!m"Z"ddl#m$Z$ejJe&Z'GddejPZ)GddejPZ* d3dejPdejVdejVdejVdejVdzde,dzde,deefdZ-Gdd ejPZ.Gd!d"ejPZ/Gd#d$eZ0Gd%d&ejPZ1eGd'd(eZ2eGd)d*e2Z3ed+,Gd-d.e2Z4ed/,Gd0d1e2Z5gd2Z6y)4zPyTorch ViT model.N)CallableIterable)nn)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputWithPoolingImageClassifierOutputMaskedImageModelingOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) ViTConfigc`eZdZdZdeffd ZdejdejfdZxZ S)ViTPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. configct||j}|j}t |t r|n||f}t |t r|n||f}|d|dz|d|dzz|_||_||_|j|_tj|j|j|||_ y)Nrr) kernel_sizestride) super__init__ image_size patch_size isinstancer num_patches num_channelsrConv2d hidden_size projection)selfrr"r# __class__s o/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/vit/modeling_vit.pyr!zViTPatchEmbeddings.__init__1s && && #-j(#CZ*V`Ia #-j(#CZ*V`Ia &qMZ]:z!}PZ[\P]?]^$$"//))F$7$79K9KYclvw pixel_valuesreturnc|jd}||jk7rtd|jd|d|j|j dj ddS)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .)shaper& ValueErrorr)flatten transpose)r*r.r&s r,forwardzViTPatchEmbeddings.forward>su#))!, 4,, ,!../yaI |,44Q7AA!QGGr-) __name__ __module__ __qualname____doc__rr!torchTensorr7 __classcell__r+s@r,rr*s4 xy xHELLHU\\Hr-rc eZdZdZddedeffd Zdejde de dejfd Z dd ejd ejd zd edejfdZ xZ S) ViTEmbeddingszb Construct the CLS token, position and patch embeddings. Optionally, also the mask token. ruse_mask_tokencrt|tjt j dd|j |_|r4tjt jdd|j nd|_ t||_ |jj}tjt j d|dz|j |_ tj|j|_|j"|_|jj$|_y)Nr)r r!r Parameterr<randnr( cls_tokenzeros mask_tokenrpatch_embeddingsr%position_embeddingsDropouthidden_dropout_probdropoutr#r")r*rrBr%r+s r,r!zViTEmbeddings.__init__Ms ekk!Q8J8J&KLQ_",,u{{1a9K9K'LMei 26 :++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++//::r- embeddingsheightwidthr/c|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rN?rrr2bicubicF)sizemode align_cornersdim)r3rJr<jit is_tracingr#rreshapepermuter functional interpolateviewcat) r*rNrOrPr% num_positionsclass_pos_embedpatch_pos_embedrY new_height new_widthsqrt_num_positionss r,interpolate_pos_encodingz&ViTEmbeddings.interpolate_pos_encodingYs`!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221bqb59221ab59r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr-Nr.bool_masked_posrhc |j\}}}}|j|}|Z|jd} |jj|| d} |j dj | } |d| z z| | zz}|j j|dd} tj| |fd}|r||j|||z}ne||jdk7s||jdk7r2td|d|d|jdd|jdd ||jz}|j|}|S) NrrRg?rXrzInput image size (*z) doesn't match model (z).)r3rIrHexpand unsqueezetype_asrFr<rarhr"r4rJrM) r*r.rirh batch_sizer&rOrPrN seq_length mask_tokensmask cls_tokenss r,r7zViTEmbeddings.forwardsc 3?2D2D/ L&%**<8  &#))!,J//00ZLK",,R088ED#sTz2[45GGJ^^**:r2> YY J7Q? ##d&C&CJPVX]&^^J++u8J/J (%9+,Adooa.@-AE$d&>&>>J\\*- r-)F)NF)r8r9r:r;rboolr!r<r=intrh BoolTensorr7r>r?s@r,rArAHs ;y ;$ ;&D5<<&D&DUX&D]b]i]i&DV48). ll ))D0 #'   r-rAmodulequerykeyvalueattention_maskscalingrMkwargsc ||jddz}tj||jdd|z}|||z}tj j |dtjj|j}tj j|||j}tj||} | jddj} | |fS)NrRr2r)rYdtype)ptrainingr) rUr<matmulr6rr^softmaxfloat32torrMr contiguous) rwrxryrzr{r|rMr} attn_weights attn_outputs r,eager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r-c eZdZdeffd Z d dej dej dzdeede ej ej ffdZ xZ S) ViTAttentionrc(t|||_|j|_t |d|j |jz|_|j|_|j dz|_ d|_ tj|j |j|j z|j|_tj|j |j|j z|j|_tj|j |j|j z|j|_tj|j|j z|j d|_y)Nhead_dimrF)biasT)r r!rnum_attention_headsgetattrr(rattention_probs_dropout_probattention_dropoutr| is_causalrLinearqkv_biasq_projk_projv_projo_projr*rr+s r,r!zViTAttention.__init__s,  #)#=#=  F4F4F&JdJd4de !'!D!D}}d* ii 2 2F4N4NQUQ^Q^4^eketetu ii 2 2F4N4NQUQ^Q^4^eketetu ii 2 2F4N4NQUQ^Q^4^eketetu ii : :T]] JFL^L^eij r-N hidden_statesr{r}r/c |jdd}g|d|j}|j|j|j dd}|j |j|j dd}|j |j|j dd}tj|jjt} | |||||f|jsdn |j|jd|\} } | jg|dj!} |j#| } | | fS)NrRrr2)rMr|)r3rrr`r6rrr get_interfacer_attn_implementationrrrr|r\rr) r*rr{r} input_shape hidden_shape query_states key_states value_statesattention_interfacerrs r,r7zViTAttention.forwardsK $))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r-N) r8r9r:rr!r<r=rrtupler7r>r?s@r,rrsf ky k"/3)||) t+)+, ) u||U\\) * )r-rc\eZdZdeffd Zdej dej fdZxZS)ViTMLPrct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) r r!rr hidden_act activation_fnrrr(intermediate_sizefc1fc2rs r,r!zViTMLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr-rr/cl|j|}|j|}|j|}|Sr)rrr)r*rs r,r7zViTMLP.forwards4/ **=9 / r- r8r9r:rr!r<r=r7r>r?s@r,rrs,KyKU\\ellr-rc eZdZdeffd Z d dej dej dzdeedej fdZ xZ S) ViTLayerrcjt|t||_t j |j |j|_t j |j |j|_ t||_ t j|j|_y)Neps)r r!r attentionr LayerNormr(layer_norm_epslayernorm_beforelayernorm_afterrmlprKrLrMrs r,r!zViTLayer.__init__sy %f- " V-?-?VEZEZ [!||F,>,>FDYDYZ&>zz&"<"<= r-Nrr{r}r/c |}|j|}|j||fi|\}}|j|}||z}|}|j|}|j |}|j|}||z}|Sr)rrrMrr)r*rr{r}residual_s r,r7zViTLayer.forward s!--m< )4>>-R6R q ]3 %0 !,,]; /  ]3 %0 r-r) r8r9r:rr!r<r=rrr7r>r?s@r,rrsV>y>/3|| t++,   r-rc\eZdZdeffd Zdej dej fdZxZS) ViTPoolerrct|tj|j|j |_t|j|_ yr) r r!rrr(pooler_output_sizedenser pooler_act activationrs r,r!zViTPooler.__init__"s> YYv1163L3LM  !2!23r-rr/c\|dddf}|j|}|j|}|S)Nr)rr)r*rfirst_token_tensor pooled_outputs r,r7zViTPooler.forward's6+1a40 #56  6 r-rr?s@r,rr!s*4y4 U\\ellr-rceZdZUeed<dZdZdZdZddgZ dZ dZ dZ dZ dZeedZd Zej(fd ZxZS) ViTPreTrainedModelrvitr.)imageTrAr)r attentionsrIct||t|tr|j6t j |jd|jjt j |jd|jj|j t j|jyyy)zInitialize the weightsNr)meanstd) r _init_weightsr$rArJinit trunc_normal_rinitializer_rangerFrHzeros_)r*rwr+s r,rz ViTPreTrainedModel._init_weightsCs f% fm ,))5""6#=#=CT[[MjMjk   v//ct{{?\?\ ]  , F--.- -r-)r8r9r:r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backend_can_compile_fullgraphrr_can_record_outputs_input_embed_layerr<no_gradrr>r?s@r,rr0sx $O!&*#(*5N"&!!",U]]_//r-rceZdZddededeffd Zeede dde jdzd e jdzd edzd e jdzd e e d ef dZxZS)ViTModelFradd_pooling_layerrBct||||_t|||_t j t|jDcgc] }t|c}|_ t j|j|j|_|r t|nd|_|j#ycc}w)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. )rBrN)r r!rrArNr ModuleListrangenum_hidden_layersrlayersrr(r layernormrpooler post_init)r*rrrBrr+s r,r!zViTModel.__init__Qs   '~NmmuVE]E]?^$_!Xf%5$_` f&8&8f>S>ST+:L?? /Tl+ 3;;*)  ) [[ KE!-J6JM K..78< 8O O4UY )O[hiir-)TFNNNN)r8r9r:rrtr!rrrr<r=rvrrr r7r>r?s@r,rrOsyTZ^  E2-13704.2 jllT) j))D0 j#'+ j  t+ j +, j $ j3  jr-rac ViT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://huggingface.co/papers/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). ) custom_introceZdZdeffd Zee d dejdzdejdzde dzdejdzde e d e f d ZxZS) ViTForMaskedImageModelingrcNt||t|dd|_t j t j |j|jdz|jzdt j|j|_ |jy)NFT)rrBr2r) in_channels out_channelsr) r r!rrr Sequentialr'r(encoder_strider& PixelShuffledecoderrrs r,r!z"ViTForMaskedImageModeling.__init__s  FeDQ}} II"..#22A58K8KK  OOF11 2    r-Nr.rirhr{r}r/c N|g|jj|jjk7r:td|jjd|jjd|j|f|||d|}|j }|ddddf}|j \}} } tj| dzx} } |jdd dj|| | | }|j|} d}||jj|jjz}|jd ||}|j|jjdj|jjd jdj}t j"j%|| d }||zj'|j'd zz |jj(z }t+|| |j,|j.S)a| bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, ViTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k") >>> model = ViTForMaskedImageModeling.from_pretrained("google/vit-base-patch16-224-in21k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```NzWhen `bool_masked_pos` is provided, `patch_size` must be equal to `encoder_stride` to ensure that the reconstructed image has the same dimensions as the input. Got `patch_size` = z and `encoder_stride` = r1)rirhr{rrSrr2rRnone) reductiongh㈵>)lossreconstructionrr)rr#rr4rrr3mathfloorr]r\rr"repeat_interleavermrrr^l1_losssumr&r rr)r*r.rirhr{r}outputsrrosequence_lengthr&rOrPreconstructed_pixel_valuesmasked_im_lossrUrrreconstruction_losss r,r7z!ViTForMaskedImageModeling.forwardsP  &DKK,B,BdkkF`F`,`&&*kk&<&<%==UVZVaVaVpVpUqqrt  /7dhh / +%=) /  / "33*!QR%04C4I4I1 O\OS$899)11!Q:BB:|]cejk&*\\/%B"  &;;))T[[-C-CCD-55b$EO11$++2H2H!L""4;;#9#91=1  #%--"7"7 F`lr"7"s 1D8==?488:PTCTUX\XcXcXpXppN(5!//))   r-r)r8r9r:rr!rrr<r=rvrtrrr r7r>r?s@r,rrsy"-13704.2 R llT)R ))D0R #'+ R  t+ R +, R  #R R r-ra ViT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ceZdZdeffd Zee d dejdzdejdzde dzdejdzde e d e f d Z xZS) ViTForImageClassificationrc.t|||j|_t|d|_|jdkDr*t j |j|jnt j|_ |jy)NF)rr) r r! num_labelsrrrrr(Identity classifierrrs r,r!z"ViTForImageClassification.__init__ ss   ++Fe<OUN_N_bcNc"))F$6$68I8IJikititiv r-Nr.labelsrhr{r}r/c |j|f||d|}|j}|dddddf}|j|} d} ||j|| |jfi|} t | | |j |jS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). )rhr{Nr)r logitsrr)rrr loss_functionrr rr) r*r.rrhr{r}rrrrr s r,r7z!ViTForImageClassification.forwards"/7dhh / %=)/  / "33'1a0 /  %4%%ffdkkLVLD$!//))   r-r)r8r9r:rr!rrr<r=rtrrr r7r>r?s@r,rrs y -1&*04.2 # llT)#  t## #'+ #  t+ # +, #  # # r-r)rrrr)Nr)7r;r collections.abcrrr<rrr activationsr masking_utilsr modeling_layersr modeling_outputsr r r modeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_vitr get_loggerr8loggerModulerrAr=floatrrrrrrrrr__all__r-r,r1s . &!69 G&KKI5(   H %HH