^jG)dZddlmZddlmZddlmZddlZddlmZddl m Z dd l m Z dd lmZmZdd lmZmZmZdd lmZmZdd lmZddlmZmZmZmZmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0ddl1m2Z2m3Z3m4Z4e*jje6Z7dZ8e)deGddeZ9e)deGddeZ:Gdd ejvZ<Gd!d"ejvZ=Gd#d$ejvZ>Gd%d&ejvZ?Gd'd(ejvZ@Gd)d*ejvZAGd+d,ejvZBGd-d.ejvZCGd/d0ejvZD ddd1ejvd2ejd3ejd4ejd5ejdzd6eFdzd7eFd8e$e(fd9ZGGd:d;ejvZHGd<d=ejvZIGd>d?ejvZJGd@dAejvZKGdBdCeZLGdDdEejvZMGdFdGejvZNe)GdHdIe"ZOGdJdKeOZPe)dLGdMdNeOZQe)dOGdPdQeOZRGdRdSejvZSGdTdUejvZTGdVdWejvZUe)dXGdYdZeOZVe)d[Gd\d]eOZWGd^d_ejvZXe)d`GdadbeOZYgdcZZy)ezPyTorch BridgeTower Model) OrderedDict)Callable) dataclassN)nn)CrossEntropyLoss)initialization)ACT2FNQuickGELUActivation)Cache DynamicCacheEncoderDecoderCache)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutput ModelOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringlogging torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs)BridgeTowerConfigBridgeTowerTextConfigBridgeTowerVisionConfigRobertaTokenizerz. Output type of [`BridgeTowerModel`]. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e ej dzed<dZ e ej dzed<y)BridgeTowerModelOutputa text_features (`torch.FloatTensor` of shape `(batch_size, text_sequence_length, hidden_size)`): Sequence of hidden-states at the text output of the last layer of the model. image_features (`torch.FloatTensor` of shape `(batch_size, image_sequence_length, hidden_size)`): Sequence of hidden-states at the image output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size x 2)`): Concatenation of last layer hidden-state of the first token of the text and image sequence (classification token), respectively, after further processing through layers used for auxiliary pretraining tasks. N text_featuresimage_features pooler_output hidden_states attentions) __name__ __module__ __qualname____doc__r*torch FloatTensor__annotations__r+r,r-tupler./var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/bridgetower/modeling_bridgetower.pyr)r)2s|/3M5$$t+2/3NE%%,3.2M5$$t+259M5**+d2926Je''(4/6r8r)z> Output type of ['BridgeTowerForContrastiveLearning'] cHeZdZUdZdZej dzed<dZej dzed<dZ e ej dzed<dZ e ej dzed<dZ e ej dzed<dZ e ej dzed<dZe ej dzed <y) BridgeTowerContrastiveOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Image-text contrastive loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). text_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. image_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. cross_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text-image cross-modal embeddings obtained by applying the projection layer to the pooler_output. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Nlosslogits text_embeds image_embeds cross_embedsr-r.)r/r0r1r2r<r3r4r5r=r>r6r?r@r-r.r7r8r9r;r;Js &*D%  d ")'+FE   $+37Ku(()D0748L%))*T1848L%))*T1859M5**+d2926Je''(4/6r8r;ceZdZfdZdej dej fdZddej dej dzfdZxZS)BridgeTowerResidualAttentionc ht|tj|j|jdz|_tj |j|j|_tjtdtj|j|jdzfdtfdtj|jdz|jfg|_ tj |j|j|_d|_y)N@epsc_fcgeluc_proj)super__init__rMultiheadAttention hidden_sizeattn LayerNormlayer_norm_epsln_1 ModuleDictrLinearr mlpln_2 attn_maskselfconfig __class__s r9rLz%BridgeTowerResidualAttention.__init__ks ))&*<*P>PTV>VW LL!3!39N9NO == RYYv'9'96;M;MPQ;QRS023ryy););a)?ASASTU  LL!3!39N9NO r8 hidden_stateattention_maskc,|+|jtj|j}|j1|jj|j |jnd|_|j |||d|j|dS)NdtypedeviceF) need_weightsrWkey_padding_maskr)tor3boolrarWr`rO)rYr\r]s r9 attentionz&BridgeTowerResidualAttention.attention|s  %+..UZZ H[H[.\N~~) NN  L$6$6|?R?R  S  yy   nn+     r8Nc||j|j||z}|j|}|jj D] }||} ||z}|SN)rfrRrVrUvalues)rYr\r]residual_statelayers r9forwardz$BridgeTowerResidualAttention.forwardsc%tyy7NP^(__yy0 XX__& /E .L /% 4 r8rh) r/r0r1rLr3Tensorrfrl __classcell__r[s@r9rBrBjsD" ell ELL "ELL%,,QUBUr8rBc^eZdZfdZddej dej dzfdZxZS)BridgeTowerTransformerct||j|_|j|_|jrHt j t|jdz Dcgc] }t|c}|_ nDt j t|jDcgc] }t|c}|_ |j|_ ycc}wcc}w)Nr") rKrLrNnum_hidden_layersremove_last_layerr ModuleListrangerB resblocks stop_gradientrYrZ_r[s r9rLzBridgeTowerTransformer.__init__s !--!'!9!9  # #]]?DTE[E[^_E_?`a!-f5aDN ]]?DTE[E[?\]!-f5]DN$11 b^s 'C,C!Nr\r]cg}|jD]H}|||}|jr |j|j8|j|J|Srh)rwrxappenddetach)rYr\r]r-blocks r9rlzBridgeTowerTransformer.forwards\ ^^ 3E ~>L!!$$\%8%8%:;$$\2  3 r8rhr/r0r1rLr3rmrlrnros@r9rqrqs) 2ELL%,,QUBUr8rqceZdZdeffd Zdej dededej fdZd dejdej fd Z xZ S) BridgeTowerVisionEmbeddingsrZct|||_|j|_|j |_|j |_tjtj|j|_ tj|j|j|j |j d|_|j |j zdz|_|jdz|_tj"|j |j|_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestridebiasr" position_idsr" persistent)rKrLrZrN embed_dim image_size patch_sizer Parameterr3randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferarangeexpandrXs r9rLz$BridgeTowerVisionEmbeddings.__init__s   ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr8 embeddingsheightwidthreturnc|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r"rNr?rrbicubicF)sizemode align_cornersdim)shaperweight unsqueezer3jit is_tracingrrrreshapepermuter functional interpolateviewcat) rYrrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r9interpolate_pos_encodingz4BridgeTowerVisionEmbeddings.interpolate_pos_encodingsv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr8 pixel_valuesc `|j\}}}}|sJ||jk7s||jk7r,td|d|d|jd|jd |jjj }|j|j |}|jdjdd}|jj|dd} tj| |gd } |r| |j| ||z} | S| |j|jz} | S) NzInput image size (*z) doesn't match model (z).r`rr"rr)rr ValueErrorrrr`rdflatten transposerrr3rrrr) rYrr batch_sizerzrr target_dtype patch_embeds class_embedsrs r9rlz#BridgeTowerVisionEmbeddings.forwards6'3'9'9$ Avu'Vt-F%SWSbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr8F) r/r0r1r%rLr3rmintrr4rlrnros@r9rrseq6q,'D5<<'D'DUX'D]b]i]i'DRE$5$5Z_ZfZfr8rceZdZfdZ ddej defdZ ddej defdZdej fdZ xZ S) BridgeTowerVisionTransformerc 0t|t||_t j |j |j|_t||_ t j |j |j|_ |j|_ |jset jt|jDcgc]-}t j |j |j/c}|_yycc}wNrE)rKrLrrrrPrNrQln_prerq transformerln_postshare_layernormrurvrs ln_separaterys r9rLz%BridgeTowerVisionTransformer.__init__s 5f=ll6#5#56;P;PQ 1&9||F$6$6F>-2K"KL L  ! !\ 1>>-$2D2D"DG`"`a a  ! !] 2>>-1tyy="AD]`d`i`iDi"ij j%(89M9M8NNa&bc cr8r/r0r1rLrlrnros@r9rr>s fdr8rcneZdZfdZdej dej dej fdZxZS)BridgeTowerSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yr) rKrLrrTrNdenserPrQDropouthidden_dropout_probdropoutrXs r9rLzBridgeTowerSelfOutput.__init__Ys` YYv1163E3EF f&8&8f>S>STzz&"<"<= r8r- input_tensorrcr|j|}|j|}|j||z}|SrhrrrPrYr-rs r9rlzBridgeTowerSelfOutput.forward_7 =1  ]3 }|'CD r8rros@r9rrX1> U\\RWR^R^r8rcVeZdZfdZdej dej fdZxZS)BridgeTowerIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yrh) rKrLrrTrNintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrXs r9rLz BridgeTowerIntermediate.__init__hs] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r8r-rcJ|j|}|j|}|Srh)rrrYr-s r9rlzBridgeTowerIntermediate.forwardps& =1 00? r8rros@r9rrgs#9U\\ellr8rcneZdZfdZdej dej dej fdZxZS)BridgeTowerOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) rKrLrrTrrNrrPrQrrrrXs r9rLzBridgeTowerOutput.__init__xs` YYv779K9KL f&8&8f>S>STzz&"<"<= r8r-rrcr|j|}|j|}|j||z}|Srhrrs r9rlzBridgeTowerOutput.forward~rr8rros@r9rrwrr8rcVeZdZfdZdej dej fdZxZS)BridgeTowerPoolerct|tj|j|j|_tj |_yrh)rKrLrrTrNrTanh activationrXs r9rLzBridgeTowerPooler.__init__s9 YYv1163E3EF '')r8r-rc\|dddf}|j|}|j|}|S)Nr)rr )rYr-first_token_tensor pooled_outputs r9rlzBridgeTowerPooler.forwards6+1a40 #56  6 r8rros@r9rrs#$ U\\ellr8rmodulequerykeyvaluer]scalingrkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)Nrrrr)ptrainingr") rr3matmulrrrsoftmaxrr contiguous) rrrrr]rrr attn_weights attn_outputs r9eager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r8c eZdZd fd Z d dej dej dzdedzdee de ej f dZ xZ S) BridgeTowerSelfAttentionNc@t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_|j&|_||_||_yNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r)rKrLrNnum_attention_headshasattrrrZrattention_head_size all_head_sizerrrTrrrrattention_probs_dropout_probr is_decoder is_causal layer_idxrYrZr*r+r[s r9rLz!BridgeTowerSelfAttention.__init__sP     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9F|} t|tr |j} | j|| |j\}} tj|jjt } | |||| |f|j"sdn|j$j&|j(d|\} } | j*g|dj-} | | fS)Nrr"rrr)rr&rrrrrrrself_attention_cacheupdater+r get_interfacerZ_attn_implementationrrrrrrr)rYr-r]r-r input_shape hidden_shape query_layer key_layer value_layercurrent_past_key_valuesattention_interfacerrs r9rlz BridgeTowerSelfAttention.forwards$))#2. CCbC$*B*BC 5djj/44lCMMaQRS 0DHH]+00,?II!QO 4djj/44lCMMaQRS  &&5 #/+>?*9*N*N'&=%C%CI{\`\j\j%k "I{(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((r8FN)NN r/r0r1rLr3rmr4r rrr6rlrnros@r9rrsg#648(, ')||')))D0') ') +, ') u||  ')r8rceZdZd fd Z d dej dej dzdej dzdedzdee de ej f d Z xZ S) BridgeTowerCrossAttentionNct||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_||_||_yr!)rKrLrNr$r%rrZrr&r'rrrTrrrrr(rr*r+r,s r9rLz"BridgeTowerCrossAttention.__init__sC     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9FB**4>>:(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((r8r<NNN) r/r0r1rLr3rmr4rrrr6rlrnros@r9r?r?s#4;?376: 1)||1) %00471)))D0 1) -t3 1) +, 1) u|| 1)r8r?ceZdZd fd Z d dej dej dzdej dzdej dzdedzdee d e ej fd Z xZ S) BridgeTowerAttentionNct|||_|rtnt}|||||_t ||_y)Nr*r+)rKrLis_cross_attentionr?rrYroutput)rYrZr*r+rNattention_classr[s r9rLzBridgeTowerAttention.__init__Es= "47I3Og#Fi9U +F3 r8r-r]rAencoder_attention_maskr-rrc |js|n|}|j|f|||d|\}}|j||}||fS)N)rAr]r-)rNrYrO) rYr-r]rArQr-rattention_outputrs r9rlzBridgeTowerAttention.forwardLsd04/F/FLb)2 * "7)+ *  * &, ;;'7G--r8)FNFNNNNr=ros@r9rKrKDs448:>;?(, .||.))D0. %0047 . !& 1 1D 8 .  .+,. u|| .r8rKc@eZdZdfd Z ddeefdZdZxZS)BridgeTowerBertCrossLayerc$t||j|_d|_t |d||_|j |_|j|_t |d|d|_t||_ t||_ y)Nr"TrMFr*r+rN) rKrLchunk_size_feed_forward seq_len_dimrKrfr)add_cross_attentioncrossattentionr intermediaterrOrYrZr+r[s r9rLz"BridgeTowerBertCrossLayer.__init__bs '-'E'E$-fPYZ ++#)#=#= 2 #   4F;'/ r8rc |j|f|dd|\}}|} |j| f||||d|\} } | } t|j|j|j | } | || fS)N)r]r-)r]rArQr-)rfr\rfeed_forward_chunkrYrZ) rYr-rAr]rQr-rself_attention_outputself_attn_weightsrScross_attention_outputcross_attn_weights layer_outputs r9rlz!BridgeTowerBertCrossLayer.forwardrs4B4>> 4 ) 4  4 00 15HT5H5H 6 )"7#9+ 6  6 2 220  # #T%A%A4CSCSUe      r8cL|j|}|j||}|Srhr]rOrYrSintermediate_outputres r9r`z,BridgeTowerBertCrossLayer.feed_forward_chunk,"//0@A{{#68HI r8rhrI) r/r0r1rLrrrlr`rnros@r9rVrVas.0(# " +," Hr8rVceZdZd fd Z d dej dej dzdej dzdej dzdedzdee d ej fd Z d Z xZ S)BridgeTowerTextLayerNct||j|_d|_t ||j ||_|j |_|j|_|jr.|j st|dt |d|d|_ t||_ t||_ y)Nr"rMz> should be used as a decoder model if cross attention is addedFTrX)rKrLrYrZrKr)rfr[rr\rr]rrOr^s r9rLzBridgeTowerTextLayer.__init__s '-'E'E$-f@Q@Q]fg ++#)#=#=  # #?? D6)g!hii"6##' #D  4F;'/ r8r-r]rArQr-rrc |j||fd|i|\}}|} |jr:|8t|dstd|d|j|d||fd|i|\} }| } t |j |j|j| } | S)Nr-r\z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) rfr)r%rr\rr`rYrZ) rYr-r]rArQr-rrarzrSrcres r9rlzBridgeTowerTextLayer.forwards$24>>  $ ,$  $  q 1 ??4@4!12 =dVDDD )<(;(;%%& ) !0 )  ) % "A 6 0  # #T%A%A4CSCSUe r8cL|j|}|j||}|Srhrgrhs r9r`z'BridgeTowerTextLayer.feed_forward_chunkrjr8rhrT) r/r0r1rLr3rmr4r rrrlr`rnros@r9rlrls0.48:>;?(, %||%))D0% %0047 % !& 1 1D 8 %  %+,% %Nr8rlceZdZfdZ d dej dej dzdej dzdej dzdedzdedzd e e d e fd Z xZ S) BridgeTowerTextEncoderc t|||_tjt |j Dcgc]}t||c}|_ycc}w)Nr+) rKrLrZrrurvrsrlrk)rYrZir[s r9rLzBridgeTowerTextEncoder.__init__sI  ]]@EfF^F^@_ `1 !&A 6 `  `sANr-r]rArQr- use_cacherrc l|jD]}||||f||d|}t||r|SdS)N)rQr-)last_hidden_stater-)rkr) rYr-r]rArQr-rur layer_modules r9rlzBridgeTowerTextEncoder.forwardse!JJ L(%(> /   M 9+/8O  >B  r8)NNNNN)r/r0r1rLr3rmr4r rerrrrlrnros@r9rqrqs 48:>;?(,!% || ))D0  %0047  !& 1 1D 8    $; +,  3 r8rqc eZdZdZfdZ d dej dzdej dzdej dzdejdzded ejf d Z e d Z e dd Z xZS)BridgeTowerTextEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cTt|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_|jdt!j"|j$j'dd|jdt!j(|j*j-t j.d|j |_tj|j$|j |j0|_y) N) padding_idxrErrFrtoken_type_idsr)rKrLrr vocab_sizerN pad_token_idword_embeddingstype_vocab_sizetoken_type_embeddingsrPrQrrrrr3rmax_position_embeddingsrzerosrrlongr|position_embeddingsrXs r9rLz"BridgeTowerTextEmbeddings.__init__s4 !||F,=,=v?Q?Q_e_r_rs%'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  "..#%<<  * *F,>,>DL\L\$  r8N input_idsr}r inputs_embedspast_key_values_lengthrc|<||j||j|}n|j||j}||j}n|jdd}|\}}|t |drm|j j |jj|jdd} tj| d|} | j||}n:tj|tj|jj}||j|}|j!|} || z} |j#|} | | z} |j%| } |j'| } | S)Nrr}rr")rindexr_)"create_position_ids_from_input_idsr|&create_position_ids_from_inputs_embedsrr%r}rdrarrr3gatherrrrrrrrPr) rYrr}rrrr5r seq_lengthbuffered_token_type_idsrrrs r9rlz!BridgeTowerTextEmbeddings.forwards  $#FFt//1G   $JJ=Z^ZjZjk  #..*K',,.s3K!, J  !t-.*.*=*=*@*@ATAT*U*\*\]i]o]opq]rtv*w'*/,,7NTU]i*j'!8!?!? J!W!&[ SWSdSdSkSk!l   00;M $ : :> J"%:: "66|D"55 ^^J/ \\*- r8c|jdd}|d}tj|dz||zdztj|j}|j dj |S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nrr"r_r)rr3rrrarr)rr|r5sequence_lengthrs r9rz@BridgeTowerTextEmbeddings.create_position_ids_from_inputs_embedsIsp$((*3B/ %a.|| !O_{:Q>ejjYfYmYm %%a(// <nt |t@rMtjB|jDtGjH|jJjMdnt |tNrxtjB|jDtGjH|jDjPdjMdtj|jRt |t0j2tTfr-|jV tj|jVyyy) Nrr)stdr/g?)meanrrr),rK _init_weightsrZinitializer_factorrrrNrsrrwinitnormal_rOin_proj_weightzeros_ in_proj_biasout_projrrUrGrJrrrrrTrr!BridgeTowerForContrastiveLearning constant_ logit_scalelogit_scale_init_valuercopy_rr3rrrrzrr}BridgeTowerMLMHeadr)rYrrproj_stdattn_stdfc_stdr~r[s r9rz(BridgeTowerPreTrainedModel._init_weightszs f%kk,, f: ; //51t{{?\?\;\ae:efH{{..4H$++111d:F++55 J UZZ66HsNK EJJ334 UZZ0077X^L UYY^^22 E UYY--44(S.I  J LL**::3 O LL**==DD(UX. Y BIIr|| D E LLSdSj A  A B NN6--t{{/Q/Q R  ; < JJv**ELL9M9M,N,U,UV],^ _  9 : JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- . fryy*<= >6;;CZ KK $D[ >r8)r/r0r1r#r5base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placementrlrr?_can_record_outputsr3no_gradrModulerrnros@r9rrlsh %(&+#35ST#4"5-.5 U]]_%BII%%r8rcFeZdZUeed<dZfdZedZddZ xZ S)BridgeTowerVisionModelrZ)rcdt||t||_|j yrh)rKrLrvisual post_initrXs r9rLzBridgeTowerVisionModel.__init__s&  26:  r8cj|jjjjjSrh)rrrrr`rYs r9r`zBridgeTowerVisionModel.dtypes${{%%55<<BBBr8c Z|j|j|j||Srh)rtyper`)rYr image_maskrrs r9rlzBridgeTowerVisionModel.forwards#{{5::djj1:?WXXr8)NF) r/r0r1r%r5rrLpropertyr`rlrnros@r9rrs0 ##! CCYr8ra0 The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. .. _*Attention is all you need*: https://huggingface.co/papers/1706.03762 cpeZdZUeed<dZdfd ZdZdZe e e dde jdzde jdzd e jdzd e jdzd e jdzd e jdzd e jdzdedzdedzdeedefdZdZxZS)BridgeTowerTextModelrZ)rct||||_d|_t ||_t ||_|r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) rKrLrZgradient_checkpointingrzrrqencoderrpoolerr)rYrZadd_pooling_layerr[s r9rLzBridgeTowerTextModel.__init__sV   &+#3F;-f5 3D'/$  r8c.|jjSrhrrrs r9get_input_embeddingsz)BridgeTowerTextModel.get_input_embeddingss...r8c&||j_yrhrrYrs r9set_input_embeddingsz)BridgeTowerTextModel.set_input_embeddingss*/'r8Nrr]r}rrrArQr-rurrc |du|duz r td|jjsd} | r6|4tt |jt |j}||j nd} |j ||||| } |j||| ||\}}|j| f||||| |d| } | d}|j|j|nd}t||| jS) Nz:You must specify exactly one of input_ids or inputs_embedsF)rZr)rrr}rr)r]rQembedding_outputrAr-)r]rArQr-rur)rwr,r-) rrZr)rr get_seq_lengthr_create_attention_masksrrrr-)rYrr]r}rrrArQr-rurrrencoder_outputssequence_outputr s r9rlzBridgeTowerTextModel.forwards=&  -t"; <YZ Z{{%%I 01,dkk2RT`hlhshsTtuOETE`!?!?!Afg??%)'#9 + 261M1M)#9-"7+ 2N2 ..'$,,   )"7#9+%    *!,8< 8O O4UY ;-'+;;  r8c|jjrt|j|||}nt|j||}|t|j|||}||fS)N)rZrr]r-rZrr])rZrr]rA)rZr)rr)rYr]rQrrAr-s r9rz,BridgeTowerTextModel._create_attention_maskssu ;; ! !/{{.- / N7{{.-N " -%>{{.5&; & "555r8)T) NNNNNNNNN)r/r0r1r$r5rrLrrr r!rr3rmr rerrrrlrrnros@r9rrs  "! "/0  *..2.2,0-1596:(,!%9 <<$&9  t+9  t+ 9 llT) 9 ||d* 9  %||d29 !& t 39 9 $;9 +,9  69  9 x6r8rzv The bare BridgeTower Model transformer outputting BridgeTowerModelOutput object without any specific head on ceZdZfdZdZdZdejdedejfdZ dejdedejfdZ e e dd ejd zd ejd zd ejd zd ejd zdejd zdejd zdejd zded zdejd zdedeedeejezfdZdZxZS)BridgeTowerModelc t||||_|j}|j}|j r_t j|j|j|_ t j|j|j|_ nt jt|jDcgc],}t j|j|j.c}|_ t jt|jDcgc],}t j|j|j.c}|_ t jd|j|_t!||_t%||_|j(s|j*r|j"j,j.D]}|j"j,j0j2j4|j2_|j"j,j0j6j4|j6_t jt|jDcgc] }t9|c}|_t jt|jDcgc] }t9|c}|_t?||_ t?||_!t jD|j|jF|_$t jD|j|jF|_%|jLr!tO||_(tO||_)nt jt|jdz Dcgc] }tO|c}|_(t jt|jdz Dcgc] }tO|c}|_)|jUycc}wcc}wcc}wcc}wcc}wcc}w)NrrEr")+rKrLrZ vision_config text_config$share_cross_modal_transformer_layersrrTrNcross_modal_text_transformcross_modal_image_transformrurvrsrrr vision_modelr text_modelr"init_layernorm_from_vision_encoderrcross_modal_ln_separaterrdatarrVcross_modal_image_layerscross_modal_text_layersrcross_modal_image_poolercross_modal_text_poolerrPrQcross_modal_text_layernormcross_modal_image_layernormshare_link_tower_layersrcross_modal_text_link_towercross_modal_image_link_towerr)rYrZrrrzrr[s r9rLzBridgeTowerModel.__init__<s9   ,, ((  6 6.0ii 8O8OQWQcQc.dD +/1yy9R9RTZTfTf/gD ,.0mmQVW]WoWoQpqA;22F4F4FGq/D +02}}SXY_YqYqSrsa=44f6H6HIs0D ,&(\\!V5G5G%H"2=A.{;,,1Z1Z''..FF J!%!2!2!9!9!A!A!H!H!M!M #0077??DDII  J)+ =B6C[C[=\ ] &{ 3 ]) %(*}}=B6C[C[=\ ] &{ 3 ]( $ ):&(A%'8'@$+-,,v7I7IvOdOd*e'+-<<8J8JPVPePe+f(  ) )/CF/KD ,0DV0LD -/1}}7>> from transformers import BridgeTowerProcessor, BridgeTowerModel >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = "hello world" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base") >>> model = BridgeTowerModel.from_pretrained("BridgeTower/bridgetower-base") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> outputs.keys() odict_keys(['text_features', 'image_features', 'pooler_output']) ```NzYBridgeTowerModel does not use `inputs_embeds`. Make sure to pass in `input_ids` instead.r")rr_rrrrrsr")r]rQ)r*r+r,r-r.)(rrrrr|r3onesrrarrZlenrrkrsrrrrr`rrrwrrrr expand_asrrfullrrrrvrrget_cls_featuresr)r6)*rYrr]r}rrrr?rrrrall_hidden_states_textall_hidden_states_imageall_hidden_states_crossall_self_attentionsr5r>extend_text_masks split_indexrkr~image_embeds_with_lncross_modal_texttext_token_type_embeddingsimage_token_type_embeddingscross_modal_imageextend_image_maskslayer_outputs_textcross_text_featureslayer_outputs_imagecross_image_featureslink_layer_indexrttext_link_towerimage_link_towertransformed_text_embedscross_text_features_cross_image_features_r*r+ cls_featuress* r9rlzBridgeTowerModel.forwardsb\"$"$"$   $):%k  48qnn& oo0090E %%k2  !"ZZ 5::iN^N^_N5;;%a1ai0) $//117784;;;X;XX[\\ __,,22?K " ) )+ 6 7  ,,33??!!$"3"3"9"9:Um@L (//1a8L&&|4&&--99CCL[Q 9E .L # * *< 8 9 $0077DD\EVEVW[WhWhWnWnEop 55kQ5O%)%?%? KKI4D4D E& )$ % # ::;KNh;hi#::;O[\:]&*&@&@ JJt1IL\L\ ]' )( ) $ 46QQ <<=QRZZ  # #A &(9(>(>q(A B**## 7;;+%  =T99!<  ,#5   13>d;;A>  -#4   315&&(;=Q'RS""$6q$9;Nq;Q#RS{C(?(?(E(E$FG- XA:$//1177:;HYZKL4,,33??II!L\Z__!!''L++D,=,=,D,D,Q,QR^,_aqtuauv-. ! #>>?OPO#@@AQR '+&@&@N^abNb&c ##2'*DD#!$ %55IK_as$t !"T!=!=>NQR>R!S$%0'9 "  #5Q"7 "U$"?"?@PST@T"U%$1'8 #  $7q#9  !  " ) )+ 6 # * *< 8 # * *,?AU+V W  & &(:1(=?RST?U'V W[- X`)<=Q~ ,,]NK %')&,--.-. 01  r8cx|j|}|j|}tj||gdS)Nrr)rrr3r)rYr*r+cls_features_textcls_features_images r9rz!BridgeTowerModel.get_cls_featuresVs< 88G!::>Jyy+-?@bIIr8) NNNNNNNNNF)r/r0r1rLrrr3rmrrrrrrr4rerrr6r)rlrrnros@r9rr6s 6p64I5<<ICITYT`T`I JELLJSJUZUaUaJ .2372615.22615+/*.).N ##d*N ))D0N ((4/ N ''$. N $$t+ N ((4/N ''$.N "DjN   4'N #'N +,N  u|| 5 5N N `Jr8rc$eZdZfdZdZxZS)"BridgeTowerPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) rKrLrrTrNrrrrr transform_act_fnrPrQrXs r9rLz+BridgeTowerPredictionHeadTransform.__init__^s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr8cl|j|}|j|}|j|}|Srh)rr!rPrs r9rlz*BridgeTowerPredictionHeadTransform.forwardgs4 =1 --m< }5 r8rros@r9rr]sUr8rc&eZdZdfd ZdZxZS)rcpt|||_t||_t j |j|jjd|_ t jtj|jj|_|||j_yy)NF)r)rKrLrZr transformrrTrNrr~decoderrr3rrr)rYrZrr[s r9rLzBridgeTowerMLMHead.__init__os  ;FCyy!3!3V5G5G5R5RY^_ LLV-?-?-J-J!KL  "(DLL  r8cd|j|}|j||jz}|Srh)r%r&r)rYx mlm_scores r9rlzBridgeTowerMLMHead.forwardxs-NN1% LL+dii7 r8rhrros@r9rrns )r8rc$eZdZfdZdZxZS)BridgeTowerITMHeadcXt|tj|d|_yNrrKrLrrTfc)rYrNr[s r9rLzBridgeTowerITMHead.__init__s  ))K+r8c(|j|}|Srhr/)rYr( itm_scores r9rlzBridgeTowerITMHead.forwardsGGAJ r8rros@r9r+r+~s ,r8r+z\ BridgeTower Model with a language modeling head on top as done during pretraining. c^eZdZddiZfdZdZdZee dde jdzde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde e defdZxZS)BridgeTowerForMaskedLMzmlm_score.decoder.weightz8bridgetower.text_model.embeddings.word_embeddings.weightct||t||_t ||_|j yrh)rKrLrrrr)rrXs r9rLzBridgeTowerForMaskedLM.__init__s5  +F3+F3 r8c.|jjSrhr)r&rs r9get_output_embeddingsz,BridgeTowerForMaskedLM.get_output_embeddingss~~%%%r8c&||j_yrhr7)rYnew_embeddingss r9set_output_embeddingsz,BridgeTowerForMaskedLM.set_output_embeddingss!/r8Nrr]r}rrrr?rrrc |jd|||||||d| } |j| j} d} |kt} |j | j }| | j d|jjj|j d} t| | | j| jS)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000360943.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())).convert("RGB") >>> text = "a looking out of the window" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist()) >>> print(results) .a cat looking out of the window. ```rr]r}rrrr?Nrr<r=r-r.r7) rr)r*rrdrarrZrr~rr-r.)rYrr]r}rrrr?rroutputs mlm_logitsmasked_lm_lossloss_fcts r9rlzBridgeTowerForMaskedLM.forwardsd#$""  ))%!'%    ^^G$9$9:   ')HYYz001F%joob$++:Q:Q:\:\&]_e_j_jkm_noN!//))   r8NNNNNNNN)r/r0r1_tied_weights_keysrLr8r;rrr3rr4rrrrlrnros@r9r4r4s" 56pq&0.2372615.22615*.H ##d*H ))D0H ((4/ H ''$. H $$t+ H ((4/H ''$.H   4'H +,H  H H r8r4z BridgeTower Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text matching. cJeZdZfdZee ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd e e d e fd Z xZ S)#BridgeTowerForImageAndTextRetrievalct||t||_t |j dz|_|jyr-)rKrLrrr+rNr2rrXs r9rLz,BridgeTowerForImageAndTextRetrieval.__init__s@  +F3+F,>,>,BC r8Nrr]r}rrrr?rrrc |jd|||||||d| } | j} |j| } d} |.t}|j | j }|| |} t | | | j| jS)a^ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, 1].item() ```r=Nr>r7) rr,r2rrdrarr-r.)rYrr]r}rrrr?rrr?r,r=itm_lossrBs r9rlz+BridgeTowerForImageAndTextRetrieval.forwards\#$""  ))%!'%     --  .  ')HYYv}}-F/H'!//))   r8rC)r/r0r1rLrrr3rr4rrrrlrnros@r9rFrFs .2372615.22615*.G ##d*G ))D0G ((4/ G ''$. G $$t+ G ((4/G ''$.G   4'G +,G  "G G r8rFc$eZdZfdZdZxZS)BridgeTowerContrastiveHeadcXt|tj|||_yrhr.)rYrN embed_sizer[s r9rLz#BridgeTowerContrastiveHead.__init__Js  ))K4r8c(|j|}|Srhr1)rYr(s r9rlz"BridgeTowerContrastiveHead.forwardNs GGAJr8rros@r9rKrKIs 5r8rKzl BridgeTower Model with a image-text contrastive head on top computing image-text contrastive loss. c6eZdZfdZee ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd e dzd e e d e fd Z xZS)rct||t||_t |j |j |_t |j |j |_t |j dz|j |_ tjtj|jj|_|j#yr-)rKrLrrrKrNcontrastive_hidden_size itc_text_headitc_image_headitc_cross_modal_headrrr3rrZrrrrXs r9rLz*BridgeTowerForContrastiveLearning.__init__Ys  +F378J8JFLjLjk89K9KVMkMkl$>v?Q?QTU?UW]WuWu$v!<< T[[5W5W(XY r8Nrr]r}rrrr? return_lossrrc | jdd|jd|||||||d| } | j} | j\} } }| d}| d}|jjj j |}|jjtjddtj|jjjjj|}|jj||z}tj j#|j%|ddd ddfdd }tj j#|j'|ddd ddfdd j)|j }tj j#|j+| dd j)|j }tj,|||gd }|j.j1j)|j }tj2||j5|z}tj2||j5|z}tj2||j5|z}d}|rtj6t9||j }tj j;||}tj j;||}tj j;||}||z|zdz }t=|||||| j| j>S)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import torch >>> image_urls = [ ... "https://farm4.staticflickr.com/3395/3428278415_81c3e27f15_z.jpg", ... "http://images.cocodataset.org/val2017/000000039769.jpg", ... ] >>> texts = ["two dogs in a car", "two cats sleeping on a couch"] >>> with httpx.stream("GET", urls[0]) as response: ... image1 = Image.open(BytesIO(response.read())) >>> with httpx.stream("GET", urls[1]) as response: ... image2 = Image.open(BytesIO(response.read())) >>> images = [image1, image2] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> inputs = processor(images, texts, padding=True, return_tensors="pt") >>> loss = model(**inputs, return_loss=True).loss >>> inputs = processor(images, texts[::-1], padding=True, return_tensors="pt") >>> loss_swapped = model(**inputs, return_loss=True).loss >>> print("Loss", round(loss.item(), 4)) Loss 0.0019 >>> print("Loss with swapped images", round(loss_swapped.item(), 4)) Loss with swapped images 2.126 ```output_hidden_statesTr=rrr"r_Nrr)rr)rarg@)r<r=r>r?r@r-r.r7) setdefaultrr,r-rrrrr3rrrrarrrr normalizerRrSrdrTrrexprtrr cross_entropyr;r.)rYrr]r}rrrr?rUrr?r,hidden_states_txthidden_states_imghidden_states_cross_modalr>r r r@r=rlogits_text_to_imagelogits_text_to_crosslogits_image_to_crossitc_lossrtext_to_image_losstext_to_cross_lossimage_to_cross_losss r9rlz)BridgeTowerForContrastiveLearning.forwardfs+v 0$7"$""  ))%!'%     -- JQJ_J_G,.G'+ (, #//<<CCPPQ]^&*&6&6&L&L JJtQejj9I9I9_9_9f9f9m9m n' )( ) $''CCDXY\ww mm--d.@.@QPQSTWAU.V\^bc-d }}..t/B/B>;K L{ Z \\#f+fmmDF!#!SU["\ *-??BUUY\\H+#%%!//))  r8rC)r/r0r1rLrrr3rr4rerrr;rlrnros@r9rrSs .2372615.22615#'s ##d*s ))D0s ((4/ s ''$. s $$t+ s ((4/s ''$.s D[s +,s  &s s r8r)rrFr4rr)Nr/)[r2 collectionsrcollections.abcr dataclassesrr3rtorch.nnrr r activationsr r cache_utilsr r r masking_utilsrrmodeling_layersrmodeling_outputsrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrr utils.genericrr utils.output_capturingr!configuration_bridgetowerr#r$r% get_loggerr/logger_TOKENIZER_FOR_DOCr)r;rrBrqrrrrrrrrmfloatrrr?rKrVrlrqrzrrrrrrr+r4rFrKr__all__r7r8r9r~s #$! %&6CCJ9G&6KKI5hh   H %'  7[7  7$  7;7  74)299)XRYY6P"))Pf7"2997"td299d4 BII  bii     ,!% II% <<% % << % LL4' % T\ %%'(%:@)ryy@)HI) I)Z.299.:8 8v?5?F RYY Fg8 g8T)%)%)%XY7Y"  {65{6 {6| ^J1^J  ^JD "    \ 7\  \ ~ T *DT  T n C (BC  C L r8