^j:;:dZddlZddlmZddlmZddlmZddlmZm Z ddl m Z dd l m Z dd lmZdd lmZmZmZmZmZd d lmZej0eZeGdde ZedGddeZedGddeeZgdZy)zPyTorch Fuyu model.N)nn)Cache)GenerationMixin)BaseModelOutputWithPoolingCausalLMOutputWithPast)PreTrainedModel) AutoModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check) FuyuConfigc>eZdZUeed<dZdZdZdZdZ dZ dZ gZ dgZ y)FuyuPreTrainedModelconfigmodel)imagetextTpast_key_valuesN)__name__ __module__ __qualname__r__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_no_split_modules_skip_keys_device_placementq/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/fuyu/modeling_fuyu.pyrr s@ (&*#"&N#4"5r(rzt The Fuyu model which consists of a vision backbone and a language model, without a language modeling head. ) custom_introc4eZdZdeffd Zdej deej dej dej fdZe e dejd e e deezfd Zd ej"d ejd ejfdZe e dd ej"dzdej dzdej dzdej dzdej"dzdedzd ejdzdedzd e e deezfdZxZS) FuyuModelrct|||j|_|jj |_t j|j|_tj|j|jz|jz|j|_d|_|j!y)NF)super__init__ pad_token_id padding_idx text_config vocab_sizer from_configlanguage_modelrLinear patch_size num_channels hidden_sizevision_embed_tokensgradient_checkpointing post_initselfr __class__s r)r/zFuyuModel.__init__4s  !.. ,,77'33F4F4FG#%99    1 1 1F4G4G GI[I[$  ',# r(word_embeddingscontinuous_embeddingsimage_patch_input_indicesreturnc |jdt|k(s'tdt|d|jd|j}t |jdD]}t j ||dk\dd}|||}|jd||jdkDr,td||jd|jd|d |||j|j|||f<|S) aThis function places the continuous_embeddings into the word_embeddings at the locations indicated by image_patch_input_indices. Different batch elements can have different numbers of continuous embeddings. Args: word_embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Tensor of word embeddings. continuous_embeddings (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Tensor of continuous embeddings. The length of the list is the batch size. Each entry is shape [num_image_embeddings, hidden], and num_image_embeddings needs to match the number of non-negative indices in image_patch_input_indices for that batch element. image_patch_input_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Tensor of indices of the image patches in the input_ids tensor. rz7Batch sizes must match! Got len(continuous_embeddings)=z and word_embeddings.shape[0]=T)as_tuplezGNumber of continuous embeddings continuous_embeddings[batch_idx].shape=zA does not match number of continuous token ids src_indices.shape=z in batch element .) shapelen ValueErrorclonerangetorchnonzerotodevice)r>r@rArBoutput_embeddings batch_idx dst_indices src_indicess r)gather_continuous_embeddingsz&FuyuModel.gather_continuous_embeddingsAsZ( %%a(C0E,FFJs3H/I.KKjQ`QfQfghQiPkl ,11344Q78 I --(A)(LPQ(Q\`abcdK4I>{KK  #&;I&F&L&LQ&OO ^7LY7W7]7]6_`I6A6G6G5II[\e[ffgi9Ni8XYd8e8h8h!((9 i4 5  ! r( pixel_valueskwargsc <|j|}t|S)z pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. )last_hidden_state)r:r)r>rUrVpatch_embeddingss r)get_image_featureszFuyuModel.get_image_featuresms! 33LA)r[r\r]special_image_maskn_image_tokensn_image_featuress r)get_placeholder_maskzFuyuModel.get_placeholder_maskys   !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2^5I5I!5LL/99"=@@AUAUV ]004 48L8L8N ND^DTT`aq`r s "!r(N image_patchesimage_patches_indicesattention_mask position_idsr use_cachec l|du|duz r td| |jj|}|jd} |i| |jn |j} ||j nd} t j| | | zt j| }|jd}|i|j|dj} | j|j|j} |j||| }|j|| }|jd |||||d | }|S) a image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. Nz:You must specify exactly one of input_ids or inputs_embedsrrr_T) return_dict)r\r])r\rprqrrrr')rIr5rbrGrOget_seq_lengthrLarangererhrZrXrNr`rmmasked_scatter)r>r[rnrorprqrr\rrrVseq_lenrOpast_key_values_lengthrYrjoutputss r)forwardzFuyuModel.forwardsl,  -t"; <YZ Z  FD//DDFyQM%%a(  )2)>Y%%MDXDXFIXId_%C%C%Ejk " <<&2H(HPUPZPZciL(11!4L  $#66}RV6Wii /22=3G3GI\I\] !%!:!:GW";" *889KM]^M%$%% ')%+    r()NNNNNNNN)rrrrr/rLTensorlistrTrr FloatTensorr r tuplerrZ LongTensorrmrboolrr{ __classcell__r?s@r)r,r,.s z *!*! $ELL1*!$)<< *!  *!XN!--N9?@R9SN + +NN"))":?:K:K"]b]n]n"0.2-159.204(,26!%5##d*5||d* 5 %||d2 5  t+ 5&&-55((4/5$;5+,5 ' '55r(r,zz Fuyu Model with a language modeling head on top for causal language model conditioned on image patches and text. czeZdZddiZdeffd Zee ddejdzdejdzdejdzd ejdzd ejdzd e dzd ejdzd e dzdejdzdedzdeedeezfdZ dfd ZxZS)FuyuForCausalLMzlm_head.weightz(model.language_model.embed_tokens.weightrct||t||_t j |j j|j jd|_ |jy)NF)bias) r.r/r,rrr6r2r9r3lm_headr<r=s r)r/zFuyuForCausalLM.__init__sS  v& yy!3!3!?!?ASASA^A^ejk  r(Nr[rnrorprqrr\rrlabelslogits_to_keeprVrCc |jd||||||||d| } | d} t| tr t| dn| }|j | dd|ddf}d}| 4|j d|| |j jjd| }t||| j| j| jS)a image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Examples: ```python >>> from transformers import FuyuProcessor, FuyuForCausalLM >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> processor = FuyuProcessor.from_pretrained("adept/fuyu-8b") >>> model = FuyuForCausalLM.from_pretrained("adept/fuyu-8b") >>> url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bus.png" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> prompt = "Generate a coco-style caption.\n" >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> generated_ids = model.generate(**inputs, max_new_tokens=7) >>> generation_text = processor.batch_decode(generated_ids[:, -7:], skip_special_tokens=True) >>> print(generation_text[0]) A blue bus parked on the side of a road. ```)r[rnror\rprqrrrrN)logitsrr3)lossrr hidden_states attentionsr') r isinstanceintslicer loss_functionrr2r3rrrr)r>r[rnrorprqrr\rrrrrVrzr slice_indicesrrs r)r{zFuyuForCausalLM.forwardsj$**  '"7')%+      8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD&#33!//))   r(c rt ||f||||||d|} |s|jddr d| d<d| d<| S)N)rrpr\rnrois_first_iterationrrTrorn)r.prepare_inputs_for_generationget) r>r[rrpr\rnrorrV model_inputsr?s r)rz-FuyuForCausalLM.prepare_inputs_for_generation-sew<   +)''"71    "fjjd&C48L0 1,0L )r() NNNNNNNNNr)NNNNNF)rrr_tied_weights_keysrr/rr rLrr|rr~rrr r rrr{rrrs@r)rrsY +,VWz .2-159.204(,26!%&*%&P ##d*P ||d* P %||d2 P  t+ P &&-P P ((4/P $;P  t#P d P +,P  ' 'P P j" r(r)rrr,)__doc__rLr cache_utilsr generationrmodeling_outputsrrmodeling_utilsr models.auto.modeling_autor processing_utilsr utilsr r rrrconfiguration_fuyur get_loggerrloggerrr,r__all__r'r(r)rs  )R-2&jj*   H % 6/ 6 6 U#U  Up z)?z  zz Br(