^jdZddlmZddlmZddlmZddlZddlmZddl m Z dd l m Z dd lmZdd lmZdd lmZdd lmZddlmZmZmZmZmZmZddlmZmZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3ddl4m5Z5m6Z6m7Z7e)jpe9Z:e'eGddeZ;e'deGdde%Z<Gdd ejzZ> dNd!ejzd"ej~d#ej~d$ej~d%ej~dzd&e@d'e@fd(ZAGd)d*ejzZBGd+d,ejzZCGd-d.eZDe'Gd/d0eZEGd1d2ejzZFGd3d4eEZGGd5d6ejzZHGd7d8ejzZIGd9d:ejzZJGd;dejzZLGd?d@eZMGdAdBejzZNGdCdDejzZOGdEdFeEZPe'dGGdHdIeEZQe'dJGdKdLeEeZRgdMZSy)OzPyTorch InstructBLIP model.)Callable) dataclass)AnyN)nn)initialization)ACT2FN)GenerationMixin)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentionsCausalLMOutputWithPastSeq2SeqLMOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)merge_with_config_defaults)OutputRecordercapture_outputs) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipConfigInstructBlipQFormerConfigInstructBlipVisionConfigc:eZdZUdZdZedzed<dZedzed<y)'BaseModelOutputWithVisionQformerOutputsz vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). Nvision_outputsqformer_outputs) __name__ __module__ __qualname____doc__r+r__annotations__r,r/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/instructblip/modeling_instructblip.pyr*r*2s)9=N.5zKInstructBlipForConditionalGenerationModelOutput.to_tuple..\sB WW Gq!**, - s-0)tuplekeysrCs`r4r@z8InstructBlipForConditionalGenerationModelOutput.to_tuple[s% YY[    r3)r-r.r/r0r8rEtorch FloatTensorr1r9r+rr,rr:rrrr@r2r3r4r7r7@s -1D%!! "T )0.2FE%## $t +281D!--1"$,,u{{1d>P>PRVR`R`/a"br3 embeddingsheightwidthr;c|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r%Ng?rrr!bicubicF)sizemode align_cornersdim)shaper_rHjit is_tracingrWrreshapepermuter functional interpolateviewcat) rCrbrcrdr]r^class_pos_embedpatch_pos_embedrl new_height new_widthsqrt_num_positionss r4interpolate_pos_encodingz5InstructBlipVisionEmbeddings.interpolate_pos_encodingxs`!&&q)A- //55a81< yy##%+*F6UZ?** *11!RaR%811!QR%8r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr3 pixel_valuesr{c|j\}}}}|jjj}|j|j |}|j dj dd}|jj|ddj |} tj| |gd} |r|j| ||} n |j} | | ddd| jdddfj |z} | S)N)dtyper!r%rfrk)rmr\weightr~toflatten transposerZexpandrHrur{r_rh) rCr|r{ batch_size_rcrd target_dtype patch_embeds class_embedsrbr_s r4forwardz$InstructBlipVisionEmbeddings.forwards'3'9'9$ Avu++2288 ++LOO,O,OP #++A.88A> ++22:q"EHHV YY l;C #!%!>!>z6SX!Y !%!8!8 "4Q8L*//!:L8La5O"P"S"ST`"aa r3F) r-r.r/r(rSrHTensorintr{rIboolr __classcell__ras@r4rKrKesmc7c$&D5<<&D&DUX&D]b]i]i&DP E$5$5 QU bgbnbn r3rKmodulequerykeyvalueattention_maskscalingdropoutc ptj||jdd|z}|||z}tjj |d}tjj |||j}tj||} | jddj} | |fS)Nrfrk)ptrainingr%r!) rHmatmulrrrrsoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputs r4eager_attention_forwardrs<<s}}R'<=GL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r3ceZdZdZfdZdej dedefdZdej de ej ej d ze ej d zffd Z xZ S) InstructBlipAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ d|_ |j|_ tj|jd|jzd|_|jretj t#j$|j}tj t#j$|j}nd}d}|Qt#j&|t#j(|d|f}tj ||j_tj|j|j|_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).Fr)bias) requires_grad)rRrSrLrTrUnum_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrLinearqkvqkv_biasrXrHzerosru zeros_liker projection)rCrLq_biasv_biasrras r4rSzInstructBlipAttention.__init__su  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!999T^^Q-?eL ??\\%++dnn"=>F\\%++dnn"=>FFF  yy&%*:*:6QV*WY_!`aHLL2DHHM))DNNDNNCr3tensorseq_lenbszc|j|||j|jjddj S)Nr%r!)rtrrrr)rCrrrs r4_shapezInstructBlipAttention._shapes7{{3GQQRSUVWbbddr3 hidden_statesr;Nc |j\}}}|j|}|j||d|j||jzj ddddd}|d|d|d} }}t j |jjt} | |||| fd|jsdn |j|jd|\} } | j||d j} |j| } | | fS) z#Input shape: Batch x Time x Channelrr!rr%N)rrrrf)rhrrprrqr get_interfacerL_attn_implementationrrrrrr) rCrrrtgt_lenrU mixed_qkv query_states key_states value_statesattention_interfacerrs r4rzInstructBlipAttention.forwards##0"4"4"6WiHH]+ %%c7At~~yTXTbTbGbckk q!Q 2;1y|YWX\,j (?(M(M KK , ,.E) %8     %  #}}C$2H2HJJ %  % ! \"))#w;FFH ook2 L((r3) r-r.r/r0rSrHrrrrErrrs@r4rrspGD>eU\\eCece")||") u||U\\D0% 2E2LL M ")r3rcVeZdZfdZdej dej fdZxZS)InstructBlipMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yN) rRrSrLr hidden_act activation_fnrrrTintermediate_sizefc1fc2r`s r4rSzInstructBlipMLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr3rr;cl|j|}|j|}|j|}|Sr)rrrrCrs r4rzInstructBlipMLP.forwards4/ **=9 / r3r-r.r/rSrHrrrrs@r4rrs$KU\\ellr3rcpeZdZdeffd Zedejdee dejfdZ xZ S)InstructBlipEncoderLayerrLcDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yNeps) rRrSrTrUr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2r`s r4rSz!InstructBlipEncoderLayer.__init__$sm ++.v6<<F> '   q&0  ((7 / %0 r3) r-r.r/r&rSrrHrrrrIrrrs@r4rr#sRS1S||+,    r3rczeZdZUeed<dZdZdZdZdZ dZ dZ dZ gdZ ejfdZxZS)InstructBlipPreTrainedModelrLblip)imagetextT)InstructBlipQFormerEmbeddingsrrInstructBlipQFormerLayer%InstructBlipQFormerMultiHeadAttentionInstructBlipQFormerSelfOutputc:t|||jj}t |t rEt j|jd|t j|jd|yt |ttfr t j|jyt |trZt j|j t#j$|j j&dj)dyy)zInitialize the weightsr)meanstdrfr%rfN)rR _init_weightsrLinitializer_range isinstancerKinit trunc_normal_r_rZ$InstructBlipForConditionalGenerationInstructBlipModelzeros_ query_tokensrcopy_ position_idsrHarangermr)rCrfactorras r4rz)InstructBlipPreTrainedModel._init_weightsYs f%.. f: ;   v88s O   v55CV L !EGX Y Z KK++ ,  = > JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h?r3)r-r.r/r&r1base_model_prefixinput_modalitiessupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesrHno_gradrrrs@r4rrCs^ (&*#"&N!U]]_ i ir3rcNeZdZdZdeffd Zedeede e zfdZ xZ S)InstructBlipEncodera Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipEncoderLayer`]. Args: config (`InstructBlipConfig`): The corresponding vision configuration for the `InstructBlipEncoder`. rLct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wNF) rRrSrLr ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)rCrLrras r4rSzInstructBlipEncoder.__init__rsP  mmuU[UmUmOn$o!%=f%E$op &+#%psA#rr;c R|}|jD] }||fi|} t|S)Nlast_hidden_state)r r)rC inputs_embedsrr encoder_layers r4rzInstructBlipEncoder.forwardxs> & ![[ M)M  ??r3) r-r.r/r0r&rSrrrrErrrrs@r4rrhsL,1,  @+, @  @ @r3rc eZdZUdZdZeed<eedZ deffd Z e e de ddejdzd ed eed eezfd Zd ZxZS)InstructBlipVisionModelr|)rrL)r attentionsct||||_|j}t ||_t ||_tj||j|_ |jyr) rRrSrLrTrKrbrencoderrrrpost_layernorm post_init)rCrLrUras r4rSz InstructBlipVisionModel.__init__s]   && 6v>*62  ll9&:O:OP r3F)tie_last_hidden_statesNr{rr;c | td|j||}|jdd|i|}|j}|j |}|dddddf}|j |}t ||S)Nz You have to specify pixel_values)r{rrr pooler_outputr2)rrbrrrr)rCr|r{rrencoder_outputsr pooled_outputs r4rzInstructBlipVisionModel.forwards  ?@ @ Ogh +74<<, ', ,  ,== //0AB)!Q'2 ++M: )/'  r3c|jSr)rbrGs r4get_input_embeddingsz,InstructBlipVisionModel.get_input_embeddingss r3r )r-r.r/main_input_namerr(r1rr_can_record_outputsrSrr rrHrIrrrrErrr"rrs@r4rrs$O! $$1+ 7  E226). ''$. #' +,  + +  3  6r3rc:eZdZdfd Z ddeefdZxZS)rct|||_|j|jzdk7r0t |ds$t d|j|jfz|j|_t|j|jz |_|j|jz|_ |jdz|_ d|_ |j|_ tj|j|j|_|r_tj|j"|j|_tj|j"|j|_ytj|j|j|_tj|j|j|_y)Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)rF)rRrSrLrTrhasattrrrattention_head_size all_head_sizerrattention_probs_dropout_probrrrrencoder_hidden_sizerrrCrLis_cross_attentionras r4rSz.InstructBlipQFormerMultiHeadAttention.__init__sj     : : :a ?PVXhHi^%%v'A'ABC  $*#=#= #&v'9'9F#))#2. CCbC$*B*BC 2N3N*NM^))#2.MMD4L4LMjj/44\BLLQPQR HH^,11(;EEaK jj055h?II!QO (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k))E;+<+S>STzz&"<"<= r3r input_tensorr;cr|j|}|j|}|j||z}|Srr<rrrCrr?s r4rz%InstructBlipQFormerSelfOutput.forward7 =1  ]3 }|'CD r3rrs@r4rr1> U\\RWR^R^r3rc eZdZd fd Z d dej dej dzdej dzdej dzdeedej f d Z xZ S) InstructBlipQFormerAttentioncdt|t|||_t ||_yr)rRrSr attentionroutputr-s r4rSz%InstructBlipQFormerAttention.__init__s) >vGYZ3F; r3Nrrr0r1rr;c ^|jd||||d|\}}|j||}|S)N)rrr0r1r2)rHrI) rCrrr0r1rrrattention_outputs r4rz$InstructBlipQFormerAttention.forwardsL( ')"7#9     Q ;;{MBr3rr9) r-r.r/rSrHrrIrrrrrs@r4rFrFs<48:>;?  || ))D0  %0047  !& 1 1D 8  +,    r3rFcVeZdZfdZdej dej fdZxZS)InstructBlipQFormerIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) rRrSrrrTrr<rrstrr intermediate_act_fnr`s r4rSz(InstructBlipQFormerIntermediate.__init__1s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r3rr;cJ|j|}|j|}|Sr)r<rPrs r4rz'InstructBlipQFormerIntermediate.forward9s& =1 00? r3rrs@r4rMrM0s#9U\\ellr3rMcneZdZfdZdej dej dej fdZxZS)InstructBlipQFormerOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) rRrSrrrrTr<rrr=r>rr`s r4rSz"InstructBlipQFormerOutput.__init__As` YYv779K9KL f&8&8f>S>STzz&"<"<= r3rr?r;cr|j|}|j|}|j||z}|SrrArBs r4rz!InstructBlipQFormerOutput.forwardGrCr3rrs@r4rSrS@rDr3rScFeZdZfdZ ddeefdZdZdZxZ S)rcft||j|_d|_t ||_||_||jzdk(rt |d|_d|_ nd|_ t||_ t||_ t||_t||_y)Nr%rT)r.F)rRrSchunk_size_feed_forward seq_len_dimrFrH layer_idxcross_attention_frequencycrossattentionhas_cross_attentionrM intermediaterSrIintermediate_query output_queryrCrLrZras r4rSz!InstructBlipQFormerLayer.__init__Os '-'E'E$5f=" v77 71 <">vZ^"_D '+D $',D $;FC/7 "A&"I5f=r3rc b|j|fd|i|}|dkDr|ddd|ddf}|jr$| td|j|f|||d|}t |j |j |j|} |jd|kDrjt |j|j |j|dd|dddfj| j} tj| | gd} | St |j|j |j|} | S)Nrrz>encoder_hidden_states must be given for cross-attention layers)rr0r1r%rk)rHr]rr\rfeed_forward_chunk_queryrXrYrmfeed_forward_chunkrdevicerHru) rCrrr0r1 query_lengthrrKquery_attention_output layer_outputlayer_output_texts r4rz InstructBlipQFormerLayer.forwardcsp*4>>  )   ! %5a,6I%J "''(0$%eff)<)<)<**#1*?+A *  *&5--,,  & L %%a(<7$=++00$$$Q q%89 % "\(() " %yy,8I)JPQR  5'',,   L r3cL|j|}|j||}|Sr)r^rIrCrKintermediate_outputrhs r4rdz+InstructBlipQFormerLayer.feed_forward_chunks,"//0@A{{#68HI r3cL|j|}|j||}|Sr)r_r`rks r4rcz1InstructBlipQFormerLayer.feed_forward_chunk_querys."556FG(()<>NO r3NNNr) r-r.r/rSrrrrdrcrrs@r4rrNs6>."# 3+,3j r3rcDeZdZfdZe ddeefdZxZS)InstructBlipQFormerEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}wr ) rRrSrLrr r r rlayerrras r4rSz#InstructBlipQFormerEncoder.__init__sW  ]]JOPVPhPhJi jY %fi 8 j  ',# ksA$rc t|jjD]}|j|}||||f||d|}!t |S)N)r1rfr)r rLr rrr) rCrrr0r1rfri layer_modules r4rz"InstructBlipQFormerEncoder.forwardskt{{445 A::a=L(%(>)   M 9+  r3rn) r-r.r/rSrrrrrrs@r4rprps:,"#  +,  r3rpc2eZdZdZfdZ ddZxZS)rz;Construct the embeddings from word and position embeddings.c,t|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_|j!dt#j$|jj'dd||_y)N) padding_idxrrrF) persistent)rRrSr Embedding vocab_sizerT pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsrr layernormr=r>rregister_bufferrHrrrLr`s r4rSz&InstructBlipQFormerEmbeddings.__init__s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej   r3c||jd}nd}|&|jdd|||zfj}|\|j|}|j |j |j }||z}|tj||fd}n|}|j |jjj}|j|}|j|}|S)Nr%rrk) rhrcloner}rrrerHrurrr~r)rC input_idsr query_embedspast_key_values_length seq_lengthrbrs r4rz%InstructBlipQFormerEmbeddings.forwards  ")!,JJ  ,,Q0FVlIl0l-lmssuL  --i8J"&":":>#8#8#>#>? ^^J/ \\*- r3rn)r-r.r/r0rSrrrs@r4rrsE " r3rceZdZdZdZdZdZdZee e ddge e ddgdZ de ffd Z d Zd Zeee dd ej(dej*d zdej(d zdej,d zdej*d zdej*d zdeedeej*ezfdZxZS)InstructBlipQFormerModelz Querying Transformer (Q-Former), used in InstructBLIP. Slightly modified from BLIP-2 as it also takes the instruction as input. Tr%z .attention)index layer_namez.crossattention)rrcross_attentionsrLct||||_t||_t ||_|jyr)rRrSrLrrbrprrr`s r4rSz!InstructBlipQFormerModel.__init__s9   7?1&9  r3c.|jjSrrbr}rGs r4r"z-InstructBlipQFormerModel.get_input_embeddingss...r3c&||j_yrr)rCrs r4set_input_embeddingsz-InstructBlipQFormerModel.set_input_embeddingss*/'r3Nrrrrr0r1rr;c x| | td||jdnd}|j|||} t|j| |}|.|dkDr| ddd|ddfn| } t|j| ||}|j | f||||d|} | j } | dddddf} t| | S) a$ query_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Hidden states to be used in the attention computation. If cross-attention, will be used for the query (i.e., key and value will use the encoder_hidden_states). Nz7You have to specify query_embeds when input_ids is Noner%r)rrr)rLrr)rLrrr0)rr0r1rfr)rrmrbr rLrrr)rCrrrrr0r1rrfembedding_outputquery_embedding_outputrsequence_outputr s r4rz InstructBlipQFormerModel.forwards$  !5VW W0<0H|))!,a ??%%+  3;;*)  " -O[]^N^%5a,6I%Jdt "%>{{45&; & ",84<< , )"7#9% ,  , *;;'1a0 ;-'  r3)NNNNN)r-r.r/r0rrrrrrrr$r'rSr"rrr rrH LongTensorrIrrrrErrrrs@r4rrsB #'N2 @Vb c  @Vg h 8/0 4804,0:>;?9 ##9 ))D09 &&- 9 llT) 9 %0047 9 !& 1 1D 89 +,9  u  !$P P9  9 r3rz[ InstructBLIP base Model consisting of language model, qformer and vision encoder. ceZdZdZdgZdeffd ZdZdejdejfdZ e e ddejd ejd ejd zdejd zd ejd zd ejd zdejd zdejd zdedeedeezfdZxZS)rr|rrLct||t|j|_t j tjd|j|jj|_ t|j|_t j|jj|j j|_t%j&|j |_|j+yNr%)rRrSr vision_config vision_modelrrXrHrnum_query_tokensqformer_configrTrrqformerr text_configlanguage_projectionr" from_configlanguage_modelrr`s r4rSzInstructBlipModel.__init__gs  3F4H4HILLQ8O8OQWQfQfQrQr)st/0E0EF #%99V-B-B-N-NPVPbPbPnPn#o '33F4F4FG r3c|j}t|dkDr:d|vr6tjj dkDrt j dt|jdrd|jj_ yyz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r%raThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_maplenrHcuda device_countloggerwarningr(rrio_same_devicerCrs r4_preprocess_acceleratez(InstructBlipModel._preprocess_acceleratety ** }  !&6m&KPUPZPZPgPgPilmPm NNM  4&& 3:>D   ( ( 7 4r3rrcj|m||jtj|jjtj |j k(}|jd}n||jjk(}|jdj|j }|SzZ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. r~rerf r"rHrrLimage_token_idlongreall unsqueezerrCrrspecial_image_masks r4get_placeholder_maskz&InstructBlipModel.get_placeholder_mask  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H /99"=@@AUAUV!!r3Nqformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskr{rr;c `|jd || d| } | d} tj| jddtj| j } |j j| jddd}tj|jddtj| j }|tj|}|j|j }tj||gd}|jd |||| | d| }|dddd|jdddf}|7|jj|}|tj|}|j|}|j|j |j }|j#|| }|j%||}|j&j(r|jd ||d | }n|jd ||||d | }t+| || S)a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. )r|r{rNrfrr%rk)rrrr0r1rrr)rrrrr>r2)rrHonesrhrrerrrm ones_likerrurrr"rr~rmasked_scatterrLuse_decoder_only_language_modelr7)rCr|rrrrrrrr{rr+ image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsroutputss r4rzInstructBlipModel.forwardshP+** %%=   &a(  %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "!7!:!:;O;V;V!W!&,@BX+Y_`!a$  '1%".#7    %Q'+A\->->q-A+A1(DE  FD//DDFyQM%!&!;!% 8 8 F 5 8 89M9M}ObOb c!66yP]6^%445GI^_ ;; 6 6)d))+-G *d))+-"3'=   G?))#*  r3)NNNNNNF)r-r.r/r#_keep_in_fp32_modulesr&rSrrHrrIrrrrrrr rEr7rrrs@r4rr^sH %O+, 1 ?( "e.>.> "uO`O` " ;?.22659:>-1)._ ''_ !,,_ !& 0 04 7 _ $$t+ _ ((4/ _ !++d2_ !& 0 04 7_ ||d*_ #'_ -._  @ @_ _ r3ra InstructBLIP Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. ceZdZUeed<dZdZdgZdeffd ZdZ de jfdZ dfd Z d Zd Zee ddej&d ej(dej(d zded zdeedeezf dZdej(dej&fdZee ddej&d ej&dej(d zdej&d zdej(d zdej(d zdej(d zdej&d zdej(d zdedeedeezfdZej< ddej&d ej(d zdej(d zdej(d zdej(d zdej&d zdedej(fdZxZ S)rrLr|Trct||tj|j|_t jtjd|j|jj|_ tj|j|_t j |jj|j"j|_|j&r t)j*|j"}nt-j*|j"}||_|j1yr)rRrSr _from_configrrrrXrHrrrrTrrrrrrrr#rr$rr)rCrLrras r4rSz-InstructBlipForConditionalGeneration.__init__ s  3@@AUAUVLLQ8O8OQWQfQfQrQr)st/<P>PQN2>>v?Q?QRN, r3c:|jj|yr)rset_output_embeddings)rCnew_embeddingss r4rz:InstructBlipForConditionalGeneration.set_output_embeddingss 11.Ar3r;c6|jjSr)rget_output_embeddingsrGs r4rz:InstructBlipForConditionalGeneration.get_output_embeddings"s""88::r3Nc\||jjSt||S)N)modality)r get_encoderrR)rCrras r4rz0InstructBlipForConditionalGeneration.get_encoder%s1  &&224 47&&9 9r3c6|jjSr)r get_decoderrGs r4rz0InstructBlipForConditionalGeneration.get_decoder+s""..00r3c|j}t|dkDr:d|vr6tjj dkDrt j dt|jdrd|jj_ yyrrrs r4rz;InstructBlipForConditionalGeneration._preprocess_accelerate/rr3rrr{rc |jd ||dd|}td i|d|i}|d}tj|j ddtj |j }|jj|jddd} tj| j ddtj |j } |tj|}|j| j }tj| |gd }|jd ||| ||dd |} | |_| dddd| j dddf} |j| } | |_|S) a pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) T)r|r{ return_dictr+rNrfrr%rk)rrrr0r1rr2)rr*rHrrhrrerrrmrrrurr,rr)rCr|rrr{rr+rrrrr,rimage_featuress r4get_image_featuresz7InstructBlipForConditionalGeneration.get_image_featuresCs>6GT5F5F6 %%=6  6  Aq>qbpq%a(  %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "!7!:!:;O;V;V!W!&,@BX+Y_`!a&$,, '1%".#7   *9&&q)!-C|/@/@/C-CQ*FG 11,?'5$r3rrcj|m||jtj|jjtj |j k(}|jd}n||jjk(}|jdj|j }|Srrrs r4rz9InstructBlipForConditionalGeneration.get_placeholder_maskrr3rrrlabelsc |j|||| d} | j} | j}| j}||j |}|t j |}| j|j|j} |j||}|j|| }|jjrS|jd ||d| }|d}d}| j|jd || |jj j"d| }n5d| d<|jd ||||| d | }|j$}|j&}t)||||| S) a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> model.to(device) # doctest: +IGNORE_RESULT >>> url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())).convert("RGB") >>> prompt = "What is unusual about this image?" >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... min_length=1, ... top_p=0.9, ... repetition_penalty=1.5, ... length_penalty=1.0, ... temperature=1, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) The unusual aspect of this image is that a man is ironing clothes on the back of a yellow SUV, which is parked in the middle of a busy city street. This is an unconventional approach to ironing clothes, as it requires the man to balance himself and his ironing equipment on top of the vehicle while navigating through traffic. Additionally, the presence of taxis and other vehicles in the scene further emphasizes the unusual nature of this situation. ```Trrr{rNrrr)r9rr{r)rrrrr)r8r9r+r,r:r2)rrr,r+r"rHrrrer~rrrLrr loss_functionrr{r8r9r7)rCr|rrrrrrrrr{rrrr,r+rrr9r8s r4rz,InstructBlipForConditionalGeneration.forwards^CGBYBY /#9%= CZC !/ < <(88'66  7D557 BM  !"__Y7N 5 8 89M9M}ObOb c!66yP]6^%445GI^_ ;; 6 6)d))+-G QZFD!)t))!&T[[=T=T=_=_ci %)F= !)d))+-"3'=   G<)+#*   r3c Rt|dr|j|jd} |j||||d} | j} |||j j g|j jz} | |j jjgz} tj| gtj|j}|j| d}|j|}|tj |}| j#|j|j$} |j'||}|j)|| }||d}|j*j j,s||d <|j*j.d i||}|S) a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width)): Input images to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. rrTrrr%rrrr2)r(rrmrrrLimage_token_indexrr bos_token_idrHrrrerepeatr"rrr~rrris_encoder_decodergenerate)rCr|rrrrrr{generate_kwargsrrr image_tokens start_tokensrinputsrs r4rz-InstructBlipForConditionalGeneration.generatesD 4 )  ' ' )!''* BFBYBY /#9%= CZC !/ < <   $ = =>A]A]] +t{{/F/F/S/S.TT !LL,uzzR^ReRef %,,Z; 7D557 BM  !"__Y7N 5 8 89M9M}ObOb c!66yP]6^%445GI^_ #0NS""))<<"+F; .$%%..KK?Kr3rr )NNNNNNNF)NNNNNF)!r-r.r/r&r1r#rrrSrrModulerrrrrrrHrIrrrrrEr*rrr7rrrrrs@r4rrs $O!+,1(B;ryy;: 1?( ;?05 @''@!++@!& 0 04 7 @ #'+ @ +, @ 8 8@@D "e.>.> "uO`O` " ;?.22659:>26*.).B ''B !,,B !& 0 04 7 B $$t+ B ((4/ B !++d2B !& 0 04 7B ((4/B   4'B #'B +,B  @ @B B HU]]_6::>-12626).D''D!++d2D!& 0 04 7 D ##d* D ((4/ D((4/D#'D   DDr3r)rrrrr)r)Tr0collections.abcr dataclassesrtypingrrHrrr activationsr generationr masking_utilsr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrrrr utils.genericrutils.output_capturingrr autor"r#r$configuration_instructblipr&r'r( get_loggerr-rr*r7rrKrfloatrrrrrrrrrrFrMrSrrprrrr__all__r2r3r4rs"$! &!)6B9G&6jj7EIIoo   H %  P.H P  P   k