^jsFdZddlZddlZddlmZddlZddlmZddlm Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZddlm Z ejBe"Z#edeGddeZ$GddejJZ&GddejJZ'GddejJZ(GddejJZ)GddejJZ*GddejJZ+Gd d!ejJZ,Gd"d#ejJZ-Gd$d%eZ.Gd&d'ejJZ/eGd(d)eZ0eGd*d+e0Z1Gd,d-ejJZ2ed.Gd/d0e0Z3Gd1d2ejJZ4Gd3d4ejJZ5ed5Gd6d7e0Z6ed8Gd9d:e0Z7ed;Gd<d=e0Z8eGd>d?e0Z9gd@Z:y)AzPyTorch ViLT model.N) dataclass)nn)CrossEntropyLoss)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) ViltConfigzF Class for outputs of [`ViltForImagesAndTextClassification`]. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ e e ej dzed<dZ e e ej dzed<y)(ViltForImagesAndTextClassificationOutputa7 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`list[tuple(torch.FloatTensor)]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. Nlosslogits hidden_states attentions) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrlisttuplerq/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/vilt/modeling_vilt.pyrr,sq &*D%  d ")'+FE   $+;?M4e//01D8?8)'r@ pixel_values pixel_maskmax_image_length_phpwxx_maskx_hx_w batch_size num_channelsheightwidth patch_dim spatial_poshw pos_embed patch_indexeffective_resolution valid_idx non_valid_idx unique_rowsu valid_row_idxnon_valid_row_idxv valid_numsnon_valid_numspad_numsselectinvp valid_choice pad_choice cls_tokenss' r( visual_embedzViltEmbeddings.visual_embed\sW,,77>>DD 1b"  ! !, /AtQM*002**6QWWQZ8P*QVVXQTl1%ad+QTl1%ad+23''/ L&%KK**dkk.D.DD ..q!"ax8BB1aHMMaQ]_hjst II SM Aq !!MM--#V'&* .  1fqj1  %%a(221a8 IIaL " "1a (kk NN5<< R(895<< UWHX;Ydh ioq "FMM" " "$aA"56 !((a&,,q/2rSUV !))!Q/ " a #3#;:N^`cCd $'9 3779 #&9 "#7#;#;#=?OP NNEN2 V,,e,< 1o,,. BMNQ9QT?a#78N NNYZ]=A+>!+CDZZ)67AaffQi7 7->?!&&)??2<=Q$q(==&s:~x'PQ fMAz2qAv$00A1D1D1FHXY  mA.|<="..uzz"~/C/C/EqVZ[  eiiq)9;LQ;OPZ;[(\bcde  f6q) fQTlF1a4L( ) . .z2| Lq!t fQTl2388RH!&A,q!t "<=BB:rSTU fQTlF1a4L89>>z2|\ ^^**:r2> IIz1o1 -II  % %aAg .q$z : A A*bRT UW` agh   M LLOEJJv||A:==fEvNTUV&;888S POZ7?=s%?AZ Z5ZZ!/Z& Z+c (|j|||} |-|j|||jj\}} } n|j d} |d}| |j t j|t j| jz} ||j t j| |t j| jz}t j| |gd} t j|| gd} | | fS)N) input_idstoken_type_ids inputs_embeds)rsrdtyperOrE) r0rr?rsrar;r! zeros_likerXrO full_liker^)r@rattention_maskrrqrrr image_embedsimage_token_type_idx text_embeds image_masksr embeddingsmaskss r(forwardzViltEmbeddings.forwards**m+  595F5Fj4;;;W;W6G6 2L+{%,,Q/K '#$ !D$>$>   ^5::kFXFX Y%  $d&@&@ OOK)=UZZXcXjXj k'  YY \:B  >;7Q?5  r'))r)rrrr r.rr __classcell__rAs@r(r*r*Ds V9B'!r'r*c*eZdZdZfdZddZxZS)r/zGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j#dt%j*|j,j/t$j0dy) N) padding_idxeps position_idsrrKF) persistentrr)r-r.rr9 vocab_sizer3 pad_token_idword_embeddingsmax_position_embeddingsr8type_vocab_sizer; LayerNormlayer_norm_epsr<r=r>register_bufferr!rdrfr2rrDrXr@r?rAs r(r.zTextEmbeddings.__init__s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  r'c6||j}n|jdd}|d}||jddd|f}|st|dr-|jddd|f}|j |d|}|}n:t j |t j|jj}||j|}|j|} || z} |j|} | | z } |j| } |j| } | S)NrKrrrr)rDrhasattrrrfr!r2rXrOrr;r8rr>) r@rrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr;rr8s r(rzTextEmbeddings.forwards/  #..*K',,.s3K ^  ,,Q  ^ J"%:: "66|D)) ^^J/ \\*- r')NNNNrrrr r.rrrs@r(r/r/sQ r'r/c(eZdZdZfdZdZxZS)r5z# Image to Patch Embedding. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nrr) kernel_sizestride)r-r.rZr[r|r3rg collectionsabcIterabler7rConv2drR)r@r?rZr[r|r3r7rAs r(r.zViltPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hir'c|j\}}}}||jk7r td|jjj }|j|j |}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rTr| ValueErrorrRrSrre)r@rqr{r|r}r~ target_dtyperws r(rzViltPatchEmbeddings.forward(si2>2D2D/ L&% 4,, ,w --33 OOLOO,O? @r'rrs@r(r5r5s jr'r5c&eZdZfdZddZxZS)ViltSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|j|_ tj|j|j|j|_ tj|j|j|j|_tj|j |_y)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)bias)r-r.r3num_attention_headsrrrhattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer<attention_probs_dropout_probr>rs r(r.zViltSelfAttention.__init__4s.     : : :a ?PVXhHi"6#5#5"67334A7  $*#=#= #&v'9'9Fpermute contiguousrDr)r@rroutput_attentionsr hidden_shape query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputss r(rzViltSelfAttention.forwardFs#))#2. CCbC$*B*BC jj/44\BLLQPQR HH]+00>HHAN jj/44\BLLQPQR !<< Y5H5HR5PQ+dii8P8P.QQ  %/.@ -"**,-=>,,7 _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD 6G=/2O\M]r'NFrrrr.rrrs@r(rr3sG$r'rcxeZdZdZdeffd ZdejdejdejfdZxZ S)ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r?ct|tj|j|j|_tj |j|_yN) r-r.rrr3denser<r=r>rs r(r.zViltSelfOutput.__init__msB YYv1163E3EF zz&"<"<= r'r input_tensorreturncJ|j|}|j|}|Srrr>r@rrs r(rzViltSelfOutput.forwardrs$ =1  ]3 r') rrrr rr.r!Tensorrrrs@r(rrgs= >z> U\\RWR^R^r'rc&eZdZfdZddZxZS) ViltAttentioncbt|t||_t ||_yr)r-r.r attentionroutputrs r(r.zViltAttention.__init__ys& *62$V, r'ch|j|||}|j|d|}|f|ddz}|S)Nrr)rr)r@rrr self_outputsattention_outputrs r(rzViltAttention.forward~sC~~m^EVW ;;|A F#% QR(88r'rrrs@r(rrxs - r'rc\eZdZdeffd Zdej dej fdZxZS)ViltIntermediater?ct|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r-r.rrr3intermediate_sizerrg hidden_actstrrintermediate_act_fnrs r(r.zViltIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r'rrcJ|j|}|j|}|Sr)rr r@rs r(rzViltIntermediate.forwards& =1 00? r' rrrrr.r!rrrrs@r(rrs*9z9U\\ellr'rcteZdZdeffd Zdej dej dej fdZxZS) ViltOutputr?ct|tj|j|j |_tj|j|_ yr) r-r.rrrr3rr<r=r>rs r(r.zViltOutput.__init__sB YYv779K9KL zz&"<"<= r'rrrcT|j|}|j|}||z}|Srrrs r(rzViltOutput.forwards. =1  ]3 % 4 r'r rs@r(rrs8>z> U\\RWR^R^r'rc*eZdZdZfdZddZxZS) ViltLayerz?This corresponds to the Block class in the timm implementation.crt||j|_d|_t ||_t ||_t||_ tj|j|j|_tj|j|j|_y)Nrr)r-r.chunk_size_feed_forward seq_len_dimrrr intermediaterrrrr3rlayernorm_beforelayernorm_afterrs r(r.zViltLayer.__init__s '-'E'E$&v.,V4 ( " V-?-?VEZEZ [!||F,>,>FDYDYZr'c|j|j|||}|d}|dd}||j|jz}|j |}|j |}|j ||}|f|z}|S)N)rrr)rrrerOrrr)r@rrrself_attention_outputsrr layer_outputs r(rzViltLayer.forwards!%  ! !- 0 /"0"  2!4(,)=+;+;z&ViltEncoder.forward..smq_`_lms)last_hidden_staterr)rmr#r%r ) r@rrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsr layer_module layer_outputss r(rzViltEncoder.forwards#7BD$5b4(4 POA|#$58H$H!(HYZM)!,M &9]1=MeZdZUeed<dZdZdZddgZfdZ xZ S)ViltPreTrainedModelr?vilt)imagetextTr*rc6t||t|tryt j |j tj|j jdjdt j|jyy)NrKr) r- _init_weightsrgr/initcopy_rr!rdrTrfzeros_r)r@modulerAs r(r6z!ViltPreTrainedModel._init_weightssl f% fn - JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- . .r') rrrrr#base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modulesr6rrs@r(r1r1s1 (&*#)+>?//r'r1cteZdZdfd ZdZdZe ddejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd e dzd e dzde dzde dzde e ejzfdZxZS) ViltModelct||||_t||_t ||_tj|j|j|_ |r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r-r.r?r*rrencoderrrr3r layernorm ViltPoolerpooler post_init)r@r?add_pooling_layerrAs r(r.zViltModel.__init__sk   (0"6* f&8&8f>S>ST,=j(4  r'cB|jjjSrrr0rr@s r(get_input_embeddingszViltModel.get_input_embeddingss..>>>r'c:||jj_yrrI)r@rs r(set_input_embeddingszViltModel.set_input_embeddingss:?''7r'Nrrrrqrrrrrrr*r+rc <| | n|jj} | | n|jj} | | n|jj} | | t d|#|j |||j } n!||j dd} n t d| \}}| |jn |j}|tj||f|}| | t d| | t d||jdn|jd}||k7r t d |Btj||jj|jjf|}|j|||||||| \}}t|j|| }|j||| | | }|d}|j|}|j |j!|nd}| s ||f|d dzSt#|||j$|j&S)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerKz5You have to specify either input_ids or inputs_embedsrNzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)r?rr)rrr*r+r)r) pooler_outputrr)r?rr*r+r%warn_if_padding_and_no_attention_maskrDrOr!rorTrZrr rBrCrEr rr)r@rrrrqrrrrrrr*r+kwargsrtext_batch_sizerrOimage_batch_sizeembedding_outputencoder_outputssequence_output pooled_outputs r(rzViltModel.forwardslX2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U&1#%.%:!! @T@T  !"ZZ/:)FPVWN  # (@ef f  !l&:VW W4@4L<--a0R^RdRdefRg  .`a a  %5t{{7M7Mt{{OeOe$fouvJ+/??       !5,; , (.3;;*)  ,, )/!5# ' *!,..98< 8O O4UY #]3oab6II I)-')77&11   r')T NNNNNNNNNNN)rrrr.rKrMrr! LongTensorr"rhboolr r%rrrs@r(r@r@s;"?@.2372615.22615+/)-,0#'p ##d*p ))D0p ((4/ p ''$. p $$t+ p ((4/p ''$.p "Djp  $;p #Tkp D[p  $eE,=,=&> >p p r'r@c$eZdZfdZdZxZS)rDct|tj|j|j|_tj |_yr)r-r.rrr3rTanh activationrs r(r.zViltPooler.__init__s9 YYv1163E3EF '')r'c\|dddf}|j|}|j|}|S)Nr)rr^)r@rfirst_token_tensorrWs r(rzViltPooler.forwards6+1a40 #56  6 r'rrs@r(rDrDs $ r'rDzU ViLT Model with a language modeling head on top as done during pretraining. ceZdZddiZfdZdZdZe ddejdzdejdzd ejdzd ejdzd ejdzd ejdzd ejdzdejdzde dzde dzde dzde e ejzfdZxZS)ViltForMaskedLMzmlm_score.decoder.weightz6vilt.embeddings.text_embeddings.word_embeddings.weightct||t||_t ||_|j yr)r-r.r@r2 ViltMLMHead mlm_scorerFrs r(r.zViltForMaskedLM.__init__s4  f% $V, r'c.|jjSr)redecoderrJs r(get_output_embeddingsz%ViltForMaskedLM.get_output_embeddingss~~%%%r'c\||j_|j|j_yr)rergr)r@new_embeddingss r(set_output_embeddingsz%ViltForMaskedLM.set_output_embeddingss !/,11r'Nrrrrqrrrrlabelsrr*r+rc D| | n|jj} |j|||||||| | |  } | dd\}}||jdn|jd}|ddd|f|dd|df}}|j |}d}|at }|j |j}||jd|jj|jd}| s|f| ddz}||f|zS|St||| j| jS)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrqrrrrrr*r+rCrrKrrrr) r?r+r2rTrerrerOr]rr rr)r@rrrrqrrrrrlrr*r+rQrrVrW text_seq_len text_featuresrt mlm_logitsmasked_lm_lossloss_fctrs r(rzViltForMaskedLM.forwardsZV&1%S>STr'cl|j|}|j|}|j|}|Sr)rryrr s r(rz#ViltPredictionHeadTransform.forward6s4 =1 --m< }5 r'rrs@r(rwrw,sUr'rwc$eZdZfdZdZxZS)rdct|||_t||_t j |j|j|_ yr) r-r.r?rw transformrrr3rrgrs r(r.zViltMLMHead.__init__>s?  4V<yy!3!3V5F5FG r'cJ|j|}|j|}|Sr)r}rg)r@rws r(rzViltMLMHead.forwardDs" NN1  LLOr'rrs@r(rdrd=sH r'rdz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. czeZdZfdZe ddej dzdejdzdej dzdejdzdej dzdejdzd ejdzd ej dzd edzd edzd edzde e ejzfdZ xZ S)ViltForQuestionAnsweringc t|||j|_t||_t j t j|j|jdzt j|jdzt jt j|jdz|j|_ |jy)NrC) r-r. num_labelsr@r2r Sequentialrr3rGELU classifierrFrs r(r.z!ViltForQuestionAnswering.__init__Qs   ++f% -- IIf((&*<*>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrnrrCro)r?r+r2rOrrerOrrV binary_cross_entropy_with_logitsrTrrr)r@rrrrqrrrrrlrr*r+rQrrOrrrs r(rz ViltForQuestionAnswering.forwardbsf&1%