^jLdZddlmZddlZddlmZddlmZddlmZddl m Z dd l m Z m Z mZdd lmZdd lmZmZdd lmZdd lmZddlmZmZmZmZmZmZddlm Z ddl!m"Z"ejFe$Z%edeGddeZ&edeGddeZ'GddejPZ)eGddeZ*edGdd e*Z+edGd!d"e*e Z,gd#Z-y)$zPyTorch PaliGemmamodel.) dataclassN)nn)Cache)PreTrainedConfig)GenerationMixin)create_causal_maskcreate_masks_for_generate!create_sliding_window_causal_mask)FlashAttentionKwargs)BaseModelOutputWithPastBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check) AutoModel)PaliGemmaConfigzN Base class for Paligemma outputs, with hidden states and attentions.  custom_introc:eZdZUdZdZej dzed<y)PaligemmaModelOutputWithPasta image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__{/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/paligemma/modeling_paligemma.pyrr,s 59**T18r(rzU Base class for PaliGemma causal language model (or autoregressive) outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) PaliGemmaCausalLMOutputWithPasta8 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)r r!r"r#r,r$r%r&r-r.rr/tupler0rr'r(r)r+r+<s &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r(r+c*eZdZdeffd ZdZxZS)PaliGemmaMultiModalProjectorconfigct|tj|jj |jj d|_y)NTbias)super__init__rLinear vision_config hidden_sizeprojection_dimlinearselfr4 __class__s r)r9z%PaliGemmaMultiModalProjector.__init__[s; ii 4 4 @ @&BVBVBeBelpq r(c(|j|}|SN)r>)r@image_featuresr/s r)forwardz$PaliGemmaMultiModalProjector.forward_s N3 r()r r!r"rr9rE __classcell__rAs@r)r3r3Zsrrr(r3cDeZdZUeed<dZdZdZdgZdgZ dZ dZ dZ dZ dZy) PaliGemmaPreTrainedModelr4model)imagetextTr3r.FN)r r!r"rr&base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraph_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendr'r(r)rIrIesI (&*#78#4"5"N"&r(rIz| The Base Paligemma model which consists of a vision backbone and a language model without language modeling head., ceZdZdZdeffd Zeeddejde e de e zfd Zd ejd ejd ejfd Zee dd ejdzdejdzdej"dzdejdzdedzdejdzd ejdzdejdzdedzde ede ezfdZxZS)PaliGemmaModelFr4ct||tj|j|_t ||_|jj|_ tj|j}||_ |jjjxs |j|_|jy)N)r4)r8r9r from_configr; vision_towerr3multi_modal_projector text_config vocab_sizelanguage_modelr4get_text_configdtypetext_config_dtype post_init)r@r4r_rAs r)r9zPaliGemmaModel.__init__}s  %119M9MN%A&%I" ,,77"..f6H6HI,!%!!D!D!R  r(zWObtains image last hidden states from the vision tower and apply multimodal projection.r pixel_valueskwargsreturnc t|j|fi|}|j}|j|}||_|SrC)r[last_hidden_stater\ pooler_output)r@rdre image_outputsselected_image_featurerDs r)get_image_featuresz!PaliGemmaModel.get_image_featuressF*)),A&A !.!@!@334JK&4 #r( input_ids inputs_embedsrDc.|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd|jdz}|jdj|j }t||jdz|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )radevicerrz6Image features and image tokens do not match, tokens: z , features: )get_input_embeddingsr$tensorr4image_token_idlongrpallsumshape unsqueezetornumel)r@rmrnrDspecial_image_maskn_image_tokensn_image_featuress r)get_placeholder_maskz#PaliGemmaModel.get_placeholder_masks   !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2^5I5I!5LL/99"=@@AUAUV ]004 48L8L8N ND^DTT`aq`r s "!r(Nattention_mask position_idsr.token_type_idslabels use_cachec |du|duz r td|R|jj|jk\r/||jjk(} |j } d| | <n|} ||j | }|Y||j nd} tj|jd|j| z}|jddz}|g|j|j}|j|j|j}|j!|||} |j#| |}|jj%||||d}|duxs|j& xs|du}||rtj(|dk(dd|d <t+di|}t-|jj.d d|j1}|t3di|d }|j4d||||| d | }t7|j8|j:|j<|j>| Sd S) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, PaliGemmaForConditionalGeneration >>> model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/paligemma2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ```Nz:You must specify exactly one of input_ids or inputs_embedsrrrp)rnrD)r4rnrr.rrqblock_sequence_idssliding_window)full_attentionsliding_attention)rrr.rnr)rhr.r/r0rr') ValueErrorr4rtr^clonerrget_seq_lengthr$arangerxrpryrlrirzrarmasked_scatterr`is_initializedwherer getattrr]copyr r_rrhr.r/r0)r@rmrdrrr.rrnrrrer| llm_input_idspast_seen_tokensrD mask_kwargsis_first_iteration causal_masksliding_mask_kwargsoutputss r)rEzPaliGemmaModel.forwardsZ  -t"; <YZ Z  T[[%?%?4??%R!*dkk.H.H!H %OO-M01M, -%M  7D557 FM  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4q8L  #!44\BPPN+..}/C/C]EXEXYN!%!:!:~";" *889K^\Mkk113*,.(  -4vO2J   QU   r() NNNNNNNNN)r r!r"accepts_loss_kwargsrr9rrr$r%rrr1rrl LongTensorrTensorrboolr rrErFrGs@r)rXrXts   n!--9?@R9S + +"))":?:K:K"]b]n]n"0.215.204(,2626*.!%l ##d*l ''$.l  t+ l &&- l  l ((4/l ((4/l   4'l $;l -.l  - -l l r(rXcpeZdZddiZdeffd Zedejde e fdZ e e dd ejdzdejdzd ejdzd ejdzd edzd ejdzdejdzdejdzdedzdeejzde e deezfdZ dfd Ze ddedejd ejdzd edzd ejdzd ejdzdedzdefdZxZS)!PaliGemmaForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightr4ct||t||_t j |j j|j jd|_ |jy)NFr6) r8r9rXrJrr:r]r<r^lm_headrcr?s r)r9z*PaliGemmaForConditionalGeneration.__init__(sS  #F+ yy!3!3!?!?ASASA^A^ejk  r(rdrec <|jj|fi|SrC)rJrl)r@rdres r)rlz4PaliGemmaForConditionalGeneration.get_image_features.s,tzz,,\DVDDr(Nrmrrr.rrnrrlogits_to_keeprfc |jd|||||||| |d | } | d} t| tr t| dn| }|j | dd|ddf}d}|4|j d|||j jjd| }t||| j| j| j| jS)r) rmrdrrrr.rnrrrN)r-rr^)r,r-r.r/r0rr')rJ isinstanceintslicer loss_functionr4r]r^r+r.r/r0r)r@rmrdrrr.rrnrrrrerr/ slice_indicesr-r,s r)rEz)PaliGemmaForConditionalGeneration.forward2sZ$**  %))%+'      8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD/#33!//)) ' ; ;   r(c t||f|||||| || d| } | jd | ddz| d<| s|s|| d<| S)N)r.rnrrrrrrrrrd)r8prepare_inputs_for_generationget)r@rmr.rnrrdrrrrrrre model_inputsrAs r)rz?PaliGemmaForConditionalGeneration.prepare_inputs_for_generations| w<   +')%))1       N + 7+7+G!+KL ( Y+7L (r(rc tjg|jddd|j}|tj|dk(dd}t |j |||||S)Nrqrr)r4rnrrr.r)r$fullsizerprr r`) r4rnrr.rrrre group_idss r)r z;PaliGemmaForConditionalGeneration.create_masks_for_generatesyJJ;!3!3!5cr!:;R H\H\]  % Na$7B?I())+'()+%   r() NNNNNNNNNr) NNNNNNTNNF)NF)r r!r"_tied_weights_keysrr9rr$r%rrrlrrrrrrr1r+rEr staticmethodrdictr rFrGs@r)rr s- +,VW Eu/@/@EFSeLfEE.215.204(,2626*.!%-.J ##d*J ''$.J  t+ J &&- J  J ((4/J ((4/J   4'J $;J ell*J +,J  0 0J J ^ )V/3*/   ||  t+   llT)   t+  !4K    r(r)rrIrX).r# dataclassesrr$r cache_utilsrconfiguration_utilsr generationr masking_utilsr r r modeling_flash_attention_utilsr modeling_outputsr rmodeling_utilsrprocessing_utilsrutilsrrrrrrautorconfiguration_paligemmar get_loggerr loggerrr+Moduler3rIrXr__all__r'r(r)rs<!  3)mmBS-&4   H %  9#:9  9  9k9  90299 ' ' ' d -d  d N ^ (@/^  ^ B ^r(