^jyHddlZddlZddlmZddlmZddlmZddlm Z ddl m Z m Z m Z ddlmZdd lmZmZdd lmZdd lmZmZmZdd lmZdd lmZddlmZddlm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'ddl(m)Z)e"jTe+Z,GddejZZ.GddejZZ/GddejZZ0GddejZZ1GddejZZ2GddejZZ3Gd d!eZ4Gd"d#ejZZ5Gd$d%ejZZ6Gd&d'ejZZ7Gd(d)ejZZ8Gd*d+ejZZ9Gd,d-eZ:Gd.d/e:Z;Gd0d1e:eZS>STzz&"<"<=   ELL)G)GHOOPWXej   N input_idsr# inputs_embedspast_key_values_lengthreturnc*||j}n|jdd}|d}||jdd|||zf}||j|}|}|j|}||z }|j |}|j |}|S)Nr%r)sizer#r-r/r0r4) r;r?r#r@rA input_shape seq_length embeddingsr/s r=forwardzBlipTextEmbeddings.forward@s  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL   00;M" "66|D)) ^^J/ \\*- r>)NNNr) __name__ __module__ __qualname____doc__r(r6 LongTensor FloatTensorintTensorrH __classcell__r<s@r=rr.ssE ".20426&' ##d*&&-((4/  !$   r>rceZdZdfd ZdZdZdZdZ ddejdejdzd ejdzd ejdzd e dzd e e d eejejffdZxZS)BlipTextSelfAttentionNct|||_|j|jzdk7r0t |ds$t d|j|jfz|j|_t|j|jz |_|j|jz|_ ||_ tj|j|j|_ |r_tj|j|j|_tj|j|j|_n^tj|j|j|_tj|j|j|_tj"|j$|_y)Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d))r'r(r9r+num_attention_headshasattr ValueErrorrOattention_head_size all_head_size layer_idxrLinearqueryencoder_hidden_sizekeyvaluer2attention_probs_dropout_probr4r;r9is_cross_attentionr\r<s r=r(zBlipTextSelfAttention.__init__`s`     : : :a ?PVXhHi^%%v'A'ABC  $*#=#= #&v'9'9Fc||_yNattn_gradients)r;rhs r=save_attn_gradientsz)BlipTextSelfAttention.save_attn_gradientsxs ,r>c|jSrfrgr;s r=get_attn_gradientsz(BlipTextSelfAttention.get_attn_gradients{s"""r>c||_yrf attention_map)r;ros r=save_attention_mapz(BlipTextSelfAttention.save_attention_map~s *r>c|jSrfrnrks r=get_attention_mapz'BlipTextSelfAttention.get_attention_maps!!!r> hidden_statesattention_maskencoder_hidden_statesencoder_attention_maskpast_key_valueskwargsrBc |jdd}g|d|j}|j|j|j dd} |du} | r|n|}d} |St |t rA|jj|j} | r |j} n|j} n|} | r|n|} | rK|I| rG j|jj}| j|jj}ng| jddd|j}|j| j|j dd}|j!| j|j dd}|K j#|||j\}}| r)t |t rd|j|j<t%j&| |j dd}|t)j*|jz }|||j-|j.z}t1j2d|}|j5|}t%j&||}|j7dddd j9}|j;dd|j<fz}|j|}||fS) Nr%rFT)dimrr)shaperZr^view transpose isinstancer is_updatedgetr\cross_attention_cacheself_attention_cachelayerskeysvaluesr`raupdater6matmulmathsqrttodevicerSoftmaxr4permute contiguousrDr[)r;rsrtrurvrwrxrE hidden_shape query_layerrdrcurr_past_key_valuescurrent_states key_layer value_layerkv_shapeattention_scoresattention_probsattention_probs_dropped context_layernew_context_layer_shapes r=rHzBlipTextSelfAttention.forwards$))#2. CCbC$*B*BC jj/44\BLLQPQR 3$>3E/>  &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$2D.- /"=*,33DNNCHHI.55dnnELLKQ--cr2QBQ8P8PQH055h?II!QOI**^499(CMMaQRSK*)=)D)DYP[]a]k]k)l& ;%*_FY*ZAEO..t~~>!<< Y5H5HR5PQ+dii8P8P.QQ  %/.2C2CDTD[D[2\\ -"**,-=>#',,"? %S"S* **,CD o--r>rfNNNN)rIrJrKr(rirlrprrr6rPrNrrrtuplerHrQrRs@r=rTrT_sG0-#+" 48:>;?(, D.||D.))D0D. %0047 D. !& 1 1D 8 D.  D.+,D. u||U\\) *D.r>rTcneZdZfdZdej dej dej fdZxZS)BlipTextSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr!) r'r(rr]r+denser0r1r2r3r4r:s r=r(zBlipTextSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r>rs input_tensorrBcr|j|}|j|}|j||z}|Srfrr4r0r;rsrs r=rHzBlipTextSelfOutput.forward7 =1  ]3 }|'CD r>rIrJrKr(r6rPrHrQrRs@r=rr1> U\\RWR^R^r>rceZdZd fd Z d dej dej dzdej dzdedzdee de ej ej ff d Z xZ S) BlipTextAttentionNcht|t||||_t ||_y)Nr\)r'r(rTr;routputrcs r=r(zBlipTextAttention.__init__s, )&2DPYZ (0 r>rsrtrurwrxrBc ^|j||||\}}|j||}||fS)Nrtrurw)r;r) r;rsrtrurwrxrrattention_outputs r=rHzBlipTextAttention.forwardsF*. )"7+ *3* &   ;;}mD00r>)FN)NNN) rIrJrKr(r6rPrNrrrrrHrQrRs@r=rrs148:>(, 1||1))D01 %0047 1  1 +, 1 u||U\\) *1r>rcVeZdZfdZdej dej fdZxZS)BlipTextIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yrf) r'r(rr]r+intermediate_sizerr hidden_actstrrintermediate_act_fnr:s r=r(zBlipTextIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r>rsrBcJ|j|}|j|}|Srf)rrr;rss r=rHzBlipTextIntermediate.forwards& =1 00? r>rrRs@r=rrs#9U\\ellr>rcneZdZfdZdej dej dej fdZxZS)BlipTextOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r'r(rr]rr+rr0r1r2r3r4r:s r=r(zBlipTextOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r>rsrrBcr|j|}|j|}|j||z}|Srfrrs r=rHzBlipTextOutput.forward rr>rrRs@r=rrrr>rceZdZfdZ d dej dej dzdej dzdej dzdedzdee d ej fd Z d Z xZ S) BlipTextLayercLt|||_|j|_d|_t |||_||_|jjr't ||jj||_ t||_ t||_ y)Nrr)rdr\)r'r(r9chunk_size_feed_forward seq_len_dimr attention layer_num is_decodercrossattentionr intermediaterr)r;r9rr<s r=r(zBlipTextLayer.__init__s  '-'E'E$*6YG" ;; ! !"34;;+A+AY#D 18$V, r>NrsrurtrvrwrxrBc |j|||\}}||j||||\}}t|j|j|j |} | S)N)rtrwr)rrrfeed_forward_chunkrr) r;rsrurtrvrwrxr_ layer_outputs r=rHzBlipTextLayer.forward!s#nn )+- ! ! ,"&"5"5 5&; / #6#  a 1  # #T%A%A4CSCSUe r>cL|j|}|j||}|Srf)rr)r;rintermediate_outputrs r=rz BlipTextLayer.feed_forward_chunk<s,"//0@A{{#68HI r>r) rIrJrKr(r6rPrNrrrrHrrQrRs@r=rrs -";?37;?(, || %0047))D0  !& 1 1D 8   +, 6r>rceZdZfdZ d dej dej dzdej dzdej dzdedzdedzd e e d e fd Z xZ S) BlipTextEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}w)NF) r'r(r9r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r;r9ir<s r=r(zBlipTextEncoder.__init__DsP  ]]eFLdLdFe#fM&!$<#fg &+#$gsA$Nrsrtrurvrw use_cacherxrBc |jr%|jr|rtjdd}|rgt |t r!t |t |j}n6|4t t |jt |j}|jD]}|||f|||d|}t||S)NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r9)rtrvrw)last_hidden_staterw) rtrainingloggerwarningrr r r9rr) r;rsrtrurvrwrrx layer_modules r=rHzBlipTextEncoder.forwardJs  & &4==p" /<8"5o|[_[f[fGg"h ("5  4l$++6V#!JJ L(% .'= /   M 9++  r>)NNNNN)rIrJrKr(r6rPrNrboolrrrrHrQrRs@r=rrCs,48:>;?(,!%( ||( ))D0(  %0047 ( !& 1 1D 8 (  ( $;( +,(  3( r>rcVeZdZfdZdej dej fdZxZS)BlipTextPoolerct|tj|j|j|_tj |_yrf)r'r(rr]r+rTanh activationr:s r=r(zBlipTextPooler.__init__ws9 YYv1163E3EF '')r>rsrBc\|dddf}|j|}|j|}|S)Nr)rr)r;rsfirst_token_tensor pooled_outputs r=rHzBlipTextPooler.forward|s6+1a40 #56  6 r>rrRs@r=rrvs#$ U\\ellr>rcVeZdZfdZdej dej fdZxZS)BlipTextPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r'r(rr]r+rrrrrtransform_act_fnr0r1r:s r=r(z(BlipTextPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr>rsrBcl|j|}|j|}|j|}|Srf)rrr0rs r=rHz'BlipTextPredictionHeadTransform.forwards4 =1 --m< }5 r>rrRs@r=rrs$UU\\ellr>rc$eZdZfdZdZxZS)BlipTextLMPredictionHeadct|t||_t j |j |jd|_t jtj|j|_ y)NT)bias) r'r(r transformrr]r+r*decoder Parameterr6zerosrr:s r=r(z!BlipTextLMPredictionHead.__init__s[ 8@yy!3!3V5F5FTR LLV->->!?@ r>cJ|j|}|j|}|Srf)rrrs r=rHz BlipTextLMPredictionHead.forwards$}5  ]3 r>)rIrJrKr(rHrQrRs@r=rrsAr>rcVeZdZfdZdej dej fdZxZS)BlipTextOnlyMLMHeadcBt|t||_yrf)r'r(r predictionsr:s r=r(zBlipTextOnlyMLMHead.__init__s 3F;r>sequence_outputrBc(|j|}|Srf)r)r;rprediction_scoress r=rHzBlipTextOnlyMLMHead.forwards ,,_=  r>rrRs@r=rrs#<!u||! !r>rcjeZdZUdZeed<dZgZee e ddge e ddgdZ fd Z xZ S) BlipTextPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r9bertrz .attention.)index layer_namez.crossattention.)rs attentionscross_attentionsct||t|trZt j |j tj|j jdjdyy)Nr%r$) r' _init_weightsrrinitcopy_r#r6r7r}r8)r;moduler<s r=rz%BlipTextPreTrainedModel._init_weightss[ f% f0 1 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h 2r>)rIrJrKrLr__annotations__base_model_prefix_no_split_modulesrrrT_can_record_outputsrrQrRs@r=rrsZ & 0m T  0FX Y iir>rc`eZdZdZdfd ZdZdZee dde jdzde jdzde jdzd e jdzd e jdzd e jdzd e jdzd e dzde dzde dzde edefdZxZS) BlipTextModela& The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. argument and `is_decoder` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ct||||_t||_t ||_|r t|nd|_|jyrf) r'r(r9rrGrencoderrpooler post_init)r;r9add_pooling_layerr<s r=r(zBlipTextModel.__init__sI   ,V4&v. 0AnV,t  r>c.|jjSrfrGr-rks r=get_input_embeddingsz"BlipTextModel.get_input_embeddingss...r>c&||j_yrfr)r;ras r=set_input_embeddingsz"BlipTextModel.set_input_embeddingss*/'r>Nr?rtr#r@encoder_embedsrurvrwrrrxrBc | sd} | | td||jnd} ||j|||| } n|} | rt|j| ||}nt |j| |}|t |j| ||}|j | f||||| d | }|j}|j|j|nd}t|||j|j|j|j S) a encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`Cache`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). FNzDYou cannot specify both input_ids and inputs_embeds at the same timer)r?r#r@rA)r9r@rtrw)r9r@rt)r9r@rtru)rtrurvrwr)r pooler_outputrwrsrr)rYget_seq_lengthrGr r9r rrrrrwrsrr)r;r?rtr#r@rrurvrwrrrxrAembedding_outputencoder_outputsrrs r=rHzBlipTextModel.forwardsXDI  ]%>cd dETE`!?!?!Afg  !##)+'= /   .  /{{.- / N7{{.-N " -%>{{.5&; & "FRT\\ F )"7#9+ F F *;;8< 8O O4UY ;-'+;;)77&11,==   r>)T) NNNNNNNNNF)rIrJrKrLr(rrrrr6rPrrrrrrHrQrRs@r=rrs(/0 *..2,0-1.2596:(,!%"'[ <<$&[  t+[ llT) [ ||d* [  t+ [  %||d2[ !& t 3[ [ $;[ 4K[ +,[  6[  [ r>rc eZdZdddZfdZdZdZdZdZe dd e jd zd e jd zd e jd zd e jd zde jd zde jd zde jd zde d zde d zde d zde d zded zdee jzdeedefdZdfd ZxZS)BlipTextLMHeadModelzcls.predictions.biasz&bert.embeddings.word_embeddings.weight)zcls.predictions.decoder.biaszcls.predictions.decoder.weightct||t|d|_t ||_|j |_|jy)NF)r)r'r(rrrclslabel_smoothingrr:s r=r(zBlipTextLMHeadModel.__init__NsB  !&EB &v.%55 r>c6|jjSrf)rrrks r=rz(BlipTextLMHeadModel.get_input_embeddingsWsyy--//r>c:|jj|yrf)rrr;new_embeddingss r=rz(BlipTextLMHeadModel.set_input_embeddingsZs &&~6r>cB|jjjSrf)r#rrrks r=get_output_embeddingsz)BlipTextLMHeadModel.get_output_embeddings]sxx##+++r>c||jj_|j|jj_yrf)r#rrrr's r=set_output_embeddingsz)BlipTextLMHeadModel.set_output_embeddings`s,'5$$2$7$7!r>Nr?rtr#r@rurvlabelsrwr return_logitsr reductionlogits_to_keeprxrBc 0|d} |j|f||||||| | d|}|j}t| tr t | dn| }|j |dd|ddf}| r|ddddddfj Sd}||ddddddfj }|ddddfj j|j}t| |j}||jd|jj|jd}| dk(r0|j|jddjd}t!|||j"|j$|j&|j( S) a encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (`torch.LongTensor`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` past_key_values (`Cache`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NF)rtr#r@rurvrwrrr%r)r/r$noner)losslogitsrwrsrr)rrrrOslicer#rrrrr$r~r9r*rDsumrrwrsrr)r;r?rtr#r@rurvr-rwrr.rr/r0rxoutputsrs slice_indicesrlm_lossshifted_prediction_scoresloss_fcts r=rHzBlipTextLMHeadModel.forwarddsP  I@I  A )%'"7#9+! A  A  11 8B>SV8W~ot4]k  HH]1mQ3F%GH $QQY/::< <  (9!SbS!)(D(O(O(Q %AqrE]--/223L3S3STF')TMaMabH8==b$++BXBXY[a[f[fgi[jkGF"!,,'8'='=a'@"EII!L0$#33!//))$55   r>c 8t||f||d|}d|d<|S)N)rwrtTr)r'prepare_inputs_for_generation)r;r?rwrt model_kwargs model_inputsr<s r=r=z1BlipTextLMHeadModel.prepare_inputs_for_generations>w<  +)   &* \"r>) NNNNNNNNNFTmeanr)NN)rIrJrK_tied_weights_keysr(rrr*r,rr6rPrrrrOrrrrHr=rQrRs@r=r!r!Hsy(>*R 07,8*..2,0-1596:&*(,!%%*"& &-.P <<$&P  t+P llT) P ||d* P %||d2 P !& t 3P  t#P P $;P d{P 4KP :P ell*P +,P +!P P d  r>r!)rr!r)>rr6rtorch.nnrrr activationsr cache_utilsrr r generationr masking_utilsr r modeling_layersrmodeling_outputsrrrmodeling_utilsrprocessing_utilsr pytorch_utilsrutilsrrr utils.genericrutils.output_capturingrrconfiguration_blipr get_loggerrIrModulerrTrrrrrrrrrrrrr!__all__r>r=rUs[  %&!CC)J9 .&6BB7E.   H %--bi.BIIi.Z  1 12 299  RYY -.-b/ bii/ f RYY  bii$ ryy "!"))!ioi4v +v tz1?zz Nr>