^j+zddlZddlmZddlmZddlZddlmZddlm Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-edGddej\Z/ dFdej\dej`dej`dej`dej`dzde1de1e2zfdZ3Gd d!ej\Z4e!d"#eGd$d%eZ5Gd&d'ej\Z6Gd(d)ej\Z7Gd*d+ej\Z8ejre/d,Z:Gd-d.eZ;Gd/d0ej\Ze!Gd5d6eZ?Gd7d8ej\Z@e!d9#eGd:d;eZAe!d<#Gd=d>e?ZBe!d?#eGd@dAeZCe!dB#GdCdDe?eZDgdEZEy)GN)Callable) dataclass)initialization)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringtorch_compilable_check torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) AutoModel)InternVLConfigInternVLVisionConfigRMSNormcheZdZddeddffd Zdej dej fdZdZxZ S) InternVLVisionRMSNormepsreturnNct|tjt j ||_||_y)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizer" __class__s y/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/internvl/modeling_internvl.pyr&zInternVLVisionRMSNorm.__init__/s1 ll5::k#:; # hidden_statesc"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrT)keepdim) dtypetor)float32powmeanrsqrtr,r+)r-r2 input_dtypevariances r0forwardzInternVLVisionRMSNorm.forward7sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r1c^t|jjd|jS)Nz, eps=)tupler+shaper,r-s r0 extra_reprz InternVLVisionRMSNorm.extra_repr>s* ))*+6$2G2G1HIIr1)gư>) __name__ __module__ __qualname__floatr&r)Tensorr>rC __classcell__r/s@r0r!r!-s7$$$$;U\\;ell;Jr1r!modulequerykeyvalueattention_maskscalingdropoutc x|}|} tj||jdd|z} || |z} tjj | d} tjj | ||j} tj| | } | jddj} | | fS)Nrrr4dim)ptrainingr) r)matmul transposer' functionalsoftmaxrQrV contiguous) rKrLrMrNrOrPrQkwargs key_states value_states attn_weights attn_outputs r0eager_attention_forwardraBsJL<<z';';Aq'ABWLL!#n4 ==((2(>L==((6??([L,,|\:K''1-88:K  $$r1cteZdZdZdeffd Z d dejdejdzdee fdZ xZ S) InternVLVisionAttentionz+Attention Class for InternVL Vision Encoderconfigct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ |j}|j}d|_ tj|j|j |jz|j |_tj|j|j |jz|j |_tj|j|j |jz|j |_tj|j|j|_|dkDrtj*|ntj,|_ |rt/|jntj,|_|rt/|j|_ytj,|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r%r&rdr. embed_dimnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropout use_qk_norm is_causalr'Linearattention_biasq_projk_projv_projprojection_layerDropoutIdentityr!q_normk_norm)r-rd proj_dropoutqk_normr/s r0r&z InternVLVisionAttention.__init___s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!900 $$ii0NU[UjUjk ii0NU[UjUjk ii0NU[UjUjk " $..$.. I>JQ>N"**\":TVT_T_Ta?F+DNN;BKKM ?F+DNN; BKKM r1Nr2rOr\c r|j\}}}|j|}|j|}|j|} |j |}|j |}|j |||j|jjdd}|j |||j|jjdd}| j|||j|jjdd} tj|jjt} | |||| |f|j sdn |j"|j$dd|\} } | j |||j&} |j)| } |j+| } | | fS)NrrF)rQrPrq)sizertrurvrzr{reshaperjrkrXviewr get_interfacerd_attn_implementationrarVrnrmrhrwro)r-r2rOr\ batch_sizeseq_len_ query_statesr]r^attention_interfacer`r_outputs r0r>zInternVLVisionAttention.forward{s "/!3!3!5 GQ{{=1 [[/ {{=1 {{<0 [[, #++JQUQ^Q^_iijkmno '' GT^^T]][eefgijk #((Wdnndmm\ffghjkl (?(M(M KK , ,.E) %8      % $}}C$2H2HJJ %  % ! \"))*gt~~N &&{3((0|##r1N) rDrErF__doc__rr&r)rHrrr>rIrJs@r0rcrc\sN5Z3Z>/3'$||'$ t+'$+, '$r1rcz7 Class for outputs of [`InternVLVisionModel`].  custom_introceZdZdZy)$InternVLVisionModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)rDrErFrr1r0rrsr1rcZeZdZdZfdZdej dej fdZxZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. c^t||j|j}}|j|j }}|d|dz|d|dzz}|d|dz|d|dzf}||_||_||_||_||_tj|||||_ y)Nrr) kernel_sizestride) r%r& image_size patch_size num_channelsr. num_patches patch_shaper'Conv2d projection) r-rdrrrr.rrr/s r0r&z&InternVLVisionPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk !!} 1 5*Q-:VW=:XY !!} 1 5z!} ST 7UV $$(&&))L+:^hir1 pixel_valuesr#c|j\}}}}||jk7r td|j|j |jj j }|jdjdd}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rr) rArrlrr7r+r6flattenrX)r-rrrheightwidth embeddingss r0r>z%InternVLVisionPatchEmbeddings.forwards2>2D2D/ L&% 4,, ,w __\__T__5K5K5Q5Q%RS ''*44Q: r1) rDrErFrr&r)rHr>rIrJs@r0rrs) j ELL U\\ r1rceZdZdZdeddffd Zdejdededejfd Z d d ejd ejdzdejfd Z xZ S)InternVLVisionEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. rdr#Nc2t|tjt j dd|j |_|jr:tjt j dd|j |_ nd|_ t||_ |j|_ t|jtj j"r |jn|j|jf|_|jj$}|j&r=tjt j d|dz|j |_nd|_tj*|j,|_y)Nr)r%r&r'r(r)zerosr. cls_tokenuse_mask_token mask_tokenrpatch_embeddingsr isinstancer collectionsabcIterabler use_absolute_position_embeddingsposition_embeddingsrxhidden_dropout_probrQ)r-rdrr/s r0r&z!InternVLVisionEmbeddings.__init__s$ ekk!Q8J8J&KL   ll5;;q!V=O=O+PQDO"DO =f E ++&++[__-E-EF   ##V%6%67  ++77  2 2')||EKK;QR?TZTfTf4g'hD $'+D $zz&"<"<= r1rrrc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j dz} ||j dz} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr4r?rrbicubicF)rmode align_cornersrS)rArr)jit is_tracingrrrpermuter'rY interpolatercat) r-rrrr num_positionsclass_pos_embedpatch_pos_embedrT new_height new_widthsqrt_num_positionss r0interpolate_pos_encodingz1InternVLVisionEmbeddings.interpolate_pos_encodingsj!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221bqb59221ab59r"tq11 T__Q// &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr1rbool_masked_posc|j\}}}}|j|}|j\}}}|K|jj ||d} |j dj | } |d| z z| | zz}|jj |dd} tj| |fd}|j||j|||z}|j|}|S)Nr4rrS) rArrrexpand unsqueezetype_asrr)rrrrQ) r-rrrrrrrr mask_tokensw cls_tokenss r0r>z InternVLVisionEmbeddings.forwards +001fe**<8 !+!2 GQ  &//00WbIK))"-55kBA#q1u- a?J^^**:r2> YY J7Q?  # # /#d&C&CJPVX]&^^J\\*- r1r) rDrErFrrr&r)rHintr BoolTensorr>rIrJs@r0rrs >3>>,&D5<<&D&DUX&D]b]i]i&DV48ll))D0  r1rcVeZdZfdZdej dej fdZxZS)InternVLVisionMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) r%r&rdr hidden_act activation_fnr'rrr.intermediate_sizefc1fc2r-rdr/s r0r&zInternVLVisionMLP.__init__8sd  #F$5$5699V//1I1IJ99V55v7I7IJr1r2r#cl|j|}|j|}|j|}|Sr)rrr)r-r2s r0r>zInternVLVisionMLP.forward?s4/ **=9 / r1)rDrErFr&r)rHr>rIrJs@r0rr7s$KU\\ellr1r) layer_normrms_normceZdZdZdeddffd ZdejdeejeejejfzfdZ xZ S)InternVLVisionLayerz?This corresponds to the Block class in the timm implementation.rdr#Nct||j|_d|_t ||_t ||_t|j|j|j|_ t|j|j|j|_ |j}tj |t#j$|jzd|_tj |t#j$|jzd|_tj*|j,|_y)Nrr"T) requires_grad)r%r&chunk_size_feed_forward seq_len_dimrc attentionrmlpNORM2FN norm_typer.layer_norm_epslayernorm_beforelayernorm_afterlayer_scale_init_valuer'r(r)r*lambda_1lambda_2rxrrQ)r-rd init_valuesr/s r0r&zInternVLVisionLayer.__init__Ls '-'E'E$08$V, '(8(8 9&:L:LRXRgRg h&v'7'789K9KQWQfQfg33  [5::f>P>P3Q%Qaef  [5::f>P>P3Q%Qaef zz&"<"<= r1r2c|j|j|\}}|j|z}||z}|j|}|j |}|j |}|j |j |z}||z}|Sr)rrrrrrQr)r-r2attention_outputr layer_outputs r0r>zInternVLVisionLayer.forward[s#nn  ! !- 0 ! ==+;;)=8 ++M: xx - ||L1 == $==<7L$m3 r1) rDrErFrrr&r)rHr@r>rIrJs@r0rrIsUI >3 > >|| u|| uU\\5<<%?@ @r1rcReZdZdeddffd Zdej deezfdZ xZ S)InternVLVisionEncoderrdr#Nct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) r%r&rdr' ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r-rdir/s r0r&zInternVLVisionEncoder.__init__xsO  ]]vOgOgIh#iA$7$?#ij &+#$jsA#r2cL|jD] }||} t|S)N)last_hidden_state)rr )r-r2 layer_modules r0r>zInternVLVisionEncoder.forward~s3!JJ 8L(7M 8+  r1) rDrErFrr&r)rHr@r r>rIrJs@r0rrws7,3,,  ||      r1rceZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ eedZej$fdZxZS) InternVLVisionPreTrainedModelrdinternvl_visionr)imagevideoTr)r2 attentionsc$t||t|trwt j |j |jt j |j|j t j |jyyt|trit j|j|jjt j|j|jjyy)zInitialize the weightsN)r% _init_weightsrrinitzeros_rrrr constant_rrdrr)r-rKr/s r0rz+InternVLVisionPreTrainedModel._init_weightss f% f6 7 KK(( )  , F--.))5 F6676  3 4 NN6??DKK,N,N O NN6??DKK,N,N O5r1)rDrErFr__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrrc_can_record_outputsr)no_gradrrIrJs@r0rrsn  )$O)&*#./N"&-- U]]_ P Pr1rc eZdZdeddffd ZdZeede d de jd e jdzde e zfd ZxZS) InternVLVisionModelrdr#Nc2t||||_t||_t ||_|jrtjn*tj|j|j|_ |jy)Nr)r%r&rdrrrencoderuse_mean_poolingr'ry LayerNormr.r layernorm post_initrs r0r&zInternVLVisionModel.__init__so   26:,V4 $44BKKM",,vGYGY_e_t_t:u  r1c.|jjSr)rrrBs r0get_input_embeddingsz(InternVLVisionModel.get_input_embeddingss///r1F)tie_last_hidden_statesrrc |j||}|j|}|d}|j|}t||j|j S)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )rr)rr2r)rrrrr2r)r-rrr\embedding_outputencoder_outputssequence_outputs r0r>zInternVLVisionModel.forwards` ??<?Y,,'78)!,..93-)77&11  r1r)rDrErFrr&rrrrr)rHrr@rr>rIrJs@r0rrsu 3  0 E2UY !LL ;@;K;Kd;R 5 5 3  r1rc>eZdZUeed<dZdZdZdgZdZ dZ dZ dZ dZ y)InternVLPreTrainedModelrdmodel)rtextrTpast_key_valuesN)rDrErFrrrr r _skip_keys_device_placementr r _can_compile_fullgraphrrrr1r0r"r"s@ 1&*##4"5N!"&r1r"c*eZdZdeffd ZdZxZS)InternVLMultiModalProjectorrdc*t|tj|jj t d|jz dzz|_tj|jj t d|jz dzz|jj |_ t|j|_tj|jj |jj |_y)Nrr)r%r&r'r vision_configr.rdownsample_ratiorrr text_configlinear_1rprojector_hidden_actactlinear_2rs r0r&z$InternVLMultiModalProjector.__init__s ,,v';';'G'G#aRXRiRiNiJjnoJo'op  , ,s1v7N7N3N/OST/T TV\VhVhVtVt  &556 &"4"4"@"@&BTBTB`B`a r1c|j|}|j|}|j|}|j|}|Sr)rr.r0r1)r-image_featuresr2s r0r>z#InternVLMultiModalProjector.forwards@7  m4 /  m4 r1)rDrErFrr&r>rIrJs@r0r)r)sb~br1r)zM Base class for InternVL outputs, with hidden states and attentions. c:eZdZUdZdZej dzed<y)InternVLModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)rDrErFrr6r) FloatTensorrrr1r0r5r5s 59**T18r1r5zx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. c:eZdZdeffd Zeeed ddejde e e ze e zdzde dzd e ed eezf d Zd ej$d ejdejfdZee dd ej$dzdejdzdej(dzdej$dzdedzd ejdzde e e ze e zdzde dzd e ed eezfdZddej(defdZxZS) InternVLModelrdct||tj|j|_t ||_tj|j|_ |jyr) r%r&r from_configr+ vision_towerr)multi_modal_projectorr-language_modelrrs r0r&zInternVLModel.__init__sY  %11&2F2FG%@%H"'33F4F4FG r1zWObtains image last hidden states from the vision tower and apply multimodal projection.rNrvision_feature_layervision_feature_select_strategyr\r#c &|j|j}|jj}|dk7rd|d<|jd |dd|}|dk(r |j }n|j |}|dk(r|ddddddf}|jd}t|d z} |jd } |j| | | d}|j|| }|j| d|jd}|j|}||_ |S) a! pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `list[int]`): Layer index or list of layer indices to extract features from. )r6r4Toutput_hidden_states)r return_dictdefaultNrrr) scale_factorr) r7r6rdr,r<rr2rArr pixel_shuffler= pooler_output) r-rr?r@r\r,vision_outputsvision_featureschannels feature_sizers r0get_image_featuresz InternVLModel.get_image_featuress:$$TZZ8 ;;77 2 %-1F) ****a RVaZ`a 2 %,>>O,::;OPO )Y 6-aQh7O#((+8S=) $**1- *11*lLZ\],,_K[,\*11*b/BWBWXZB[\44_E'6$r1 input_ids inputs_embedsr3c.|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd|jdz}|jdj|j }t||jdz|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )r6devicer4rrz6Image features and image tokens do not match, tokens: z , features: )rr)tensorrdimage_token_idlongrPallsumrArr7rnumel)r-rMrNr3special_image_maskn_image_tokensn_image_featuress r0get_placeholder_maskz"InternVLModel.get_placeholder_maskMs   !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2^5I5I!5LL/99"=@@AUAUV ]004 48L8L8N ND^DTT`aq`r s "!r1rO position_idsr%c  |du|duz r td||j|}|k|j|||dj} | j |j |j } |j||| } |j| | }|jd||||d| } t| j| j| j| j| SdS)Nz:You must specify exactly one of input_ids or inputs_embedsT)rr?r@rC)rNr3)rOr[r%rN)rr%r2rr6r)rlrrLrGr7rPr6rZmasked_scatterr>r5rr%r2r) r-rMrrOr[r%rNr?r@r\r3rWoutputss r0r>zInternVLModel.forwardes9  -t"; <YZ Z  7D557 BM  #!44)%9/M 5 m  ,..}/C/C]EXEXYN!%!:!:~";" *889K^\M%$%% )%+'    +%77#33!//))2>2J   QU   r1rIrEc |j\}}}}||zdk7s||zdk7r td|j||t||zt||z }|j ddddj }|j|t||zt||zt||dzz }|j ddddj }|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)rrlrrrr[)r-rIrErrrrJs r0rFzInternVLModel.pixel_shuffles/>.B.B.D+ E68 L A %)=)Bjk k*.. s6L#893x,?V;W *11!Q1=HHJ*.. F\12C 8L4MsS[_kmn_nSoOp  *11!Q1=HHJr1NN)NNNNNNNN)r)rDrErFrr&rrrr)r7rliststrrrr@rrL LongTensorrZrHrr5r>rGrFrIrJs@r0r9r9 s ~ n DH59 ,'',"DIoS 9D@,),d , +, , + + , ,\"))":?:K:K"]b]n]n"0.215.204(,26CG59- ##d*- ''$.-  t+ - &&- -  - ((4/- "DIoS 9D@- ),d - +,-  , ,- - ^!U\\!!r1r9zT Base class for InternVL causal language model (or autoregressive) outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) InternVLCausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsr%r2rr6)rDrErFrrfr)r7rrgr%rr2r@rr6rr1r0reres &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r1rezV The INTERNVL model which consists of a vision backbone and a language model. cBeZdZddiZdeffd ZdejfdZe dde jd e e e ze e zdzd edzd eedeezf d Zee dd e j*dzde jdzde j,dzde j*dzdedzde jdzd e e e ze e zdzd edzde j*dzde e j,zde j,dzd eedeezfdZ dfd ZxZS) InternVLForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightrdct||t||_t j |j j|j jd|_ |jy)NFrf) r%r&r9r#r'rrr-r. vocab_sizelm_headrrs r0r&z)InternVLForConditionalGeneration.__init__sS  "6* yy!3!3!?!?ASASA^A^ejk  r1r#c|jSr)rlrBs r0get_output_embeddingsz6InternVLForConditionalGeneration.get_output_embeddingss ||r1Nrr?r@r\c B|jjd|||d|S)N)rr?r@r)r#rL)r-rr?r@r\s r0rLz3InternVLForConditionalGeneration.get_image_featuress5-tzz,, %!5+I    r1rMrOr[r%rNlabelslogits_to_keep image_sizesc |jd||||||||| d | } | d}t| tr t| dn| }|j |dd|ddf}d}| 4|j d|| |j jjd| }t||| j| j| j| jS)ac Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```) rMrrOr[r%rNr?r@rrrN)rgrprk)rfrgr%r2rr6r)r#rrslicerl loss_functionrdr-rkrer%r2rr6)r-rMrrOr[r%rNr?r@rprqrrr\r^r2 slice_indicesrgrfs r0r>z(InternVLForConditionalGeneration.forwardsh$**  %)%+'!5+I#      8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD.#33!//)) ' ; ;   r1c ft ||f|||||d|} |s|jdds|| d<| S)N)r%rNrOrqis_first_iteration use_cacheTr)r%prepare_inputs_for_generationget) r-rMr%rNrrOrqrxr\ model_inputsr/s r0rzz>InternVLForConditionalGeneration.prepare_inputs_for_generationMsYw<  +'))1    VZZ T%B ,8L (r1r`) NNNNNNNNNrN)NNNNNF)rDrErF_tied_weights_keysrr&r'Modulernrr)r7rrarbrrr@rrLrrcrHrrer>rzrIrJs@r0riris +,VW~ ryyDH59  ''  "DIoS 9D@  ),d  +,  + +    .215.204(,26CG59*.-.+/Q ##d*Q ''$.Q  t+ Q &&- Q  Q ((4/Q "DIoS 9D@Q ),d Q   4'Q ell*Q \\D(Q +,Q  / /Q Q l r1ri)rrr"r9ri)r)Fcollections.abcrr dataclassesrr)torch.nnr'rr activationsr cache_utilsr generationr integrationsr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrrutils.output_capturingrautorconfiguration_internvlrrr~r!rHrGrrarcrrrrrrrrrrr"r)r5r9reri__all__rr1r0rs,$! &! )79ddF&ggI5HY'JBIIJ(J6% II% <<% % << % LL4' %  %S[%4F$biiF$R  +E   BII J[ryy[|  3H I+4+\ BII &POPP@& 7& & R 'o ' '"))$   9"9 9  9 f+f  fR  9[9  90 N'>N  Nb r1