^j:6ddlZddlmZddlmZmZddlmZm Z m Z m Z ddl m Z mZddlmZmZmZddlmZdd lmZerd d lmZmZej4eZGd d e dZeedGdde ZdgZy)N) AudioInput) ImageInputmake_nested_list_of_images)MultiModalDataProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringis_vision_availablelogging)requires) VideoInput)Gemma4ImageProcessorKwargs get_aspect_ratio_preserving_sizec4eZdZUeed<dddddiiddidZy)Gemma4ProcessorKwargs images_kwargsT)paddingreturn_mm_token_type_idsdo_convert_rgbreturn_metadata) text_kwargsr audio_kwargs videos_kwargsN)__name__ __module__ __qualname__r__annotations__ _defaultsw/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/gemma4/processing_gemma4.pyrr"s5--(, d +T2 Ir%rF)total)vision)backendsc heZdZeZ ddededeffd Z ddedzdee ze eze e zde d e ffd Z ddee ezdzdee ze eze e zde d e d eef fd Zd ededefdZdededefdZdededefdZddZdedefdZefdZede efdZxZS)Gemma4ProcessorNimage_seq_lengthaudio_seq_lengthaudio_ms_per_tokenc ||_|j|_|j|_|j|_|j|_|j ddgid|_|j|j |_||_ ||_ t|dd|_ t|dd|_ t|dd|_t|dd|_t! |Dd |||||d| y) u image_seq_length (`int`, *optional*, defaults to 280): The number of soft tokens per image used for placeholder expansion. audio_seq_length (`int`, *optional*, defaults to 750): The maximum number of audio soft tokens per audio segment. Serves as an upper-bound cap when dynamic audio token counts are computed. audio_ms_per_token (`int`, *optional*, defaults to 40): Milliseconds of audio per output soft token. Used to dynamically compute the number of audio placeholder tokens as ``ceil(duration_ms / audio_ms_per_token)``. The default of 40 comes from the SSCP convolution's 4× time reduction on 10ms frames. additional_special_tokensz <|video|>audio_token_idN audio_token boa_token eoa_token)feature_extractorimage_processor tokenizervideo_processor chat_templater$)r,image_token_id boi_token eoi_token image_tokenadd_special_tokens video_tokenconvert_tokens_to_idsvideo_token_idr-r.getattrr1r2r3r4super__init__) selfr5r6r7r8r9r,r-r.kwargs __class__s r&rDzGemma4Processor.__init__6s.!1'66",,",,$00 $$&AK=%QR&'==d>N>NO !1#5%i1A4H"9mTB K> K>  /++'    r%imagestextvideosaudioc t|d||||d|\}}}}| t|}|r7|s5|Dcgc]*}dj|jgt |z,}}|r|s|j gt |z}||||fScc}w)N)rHrIrJrK r$)rCprepare_inputs_layoutrjoinr=lenr2)rErHrIrJrKrF image_listrGs r&rNz%Gemma4Processor.prepare_inputs_layoutns',g&C' V5' DJ' #fe  /7F $U[\zCHHd../#j/AB\D\ $$%E 2DtVU** ]s/BrFc t |d||d|| | td| |j|j |j td||Dcgc]}|j |j}}|t|t|k7r$tdt|dt|d|Dcgc] }t|} }|| k7r,td|jd|d |jd | d y|1t|r%td t|d |jd yyycc}wcc}w)N)rHrIz+You must provide either `text` or `images`.zUAudio inputs were provided, but the tokenizer does not have an `audio_token` defined.z1Received inconsistently sized batches of images (z ) and text (z).zThe total number of zP tokens in the prompts should be the same as the number of images passed. Found rMz tokens and z images per sample.zFound z. tokens in the text but no images were passed.r$) rCvalidate_inputs ValueErrorr2r3r4countr=rPanysum) rErHrIrJrKrFsamplen_images_in_textsublistn_images_in_imagesrGs r&rSzGemma4Processor.validate_inputss CvDCFC R R!v;#d)+$KCPVK=XdehimendooqrCI%Iwc'l%I"%I#'99$.t/?/?.@A""2!31T5E5E4FlSeRffy{: C(8$9 S!1231T5E5E4FFtu%: R&Js "D:?D? image_inputs image_idxreturncd|d|}|j|j|z|jS)Nnum_soft_tokens_per_image)r;r=r<)rEr\r]num_soft_tokenss r&replace_image_tokenz#Gemma4Processor.replace_image_tokens;&'BCIN..!$"2"2_"D!EdnnEUVVr% video_inputs video_idxc |d|}|d|}|jtjd|jdn |j|_|jDcgc]#}t |dzddt |dzd%}}dj |Dcgc].}|d|j |j|z|j0c}}|Scc}wcc}w) Nnum_soft_tokens_per_videovideo_metadataaGemma4 requires frame timestamps to construct prompts, but the `fps` of the input video could not be inferred. Probably `video_metadata` was missing from inputs and you passed pre-sampled frames. Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results.<02d:rM) fpslogger warning_once timestampsintrOr;r?r<) rErcrdrametadataseconds timestamp_strtvideo_replacements r&replace_video_tokenz#Gemma4Processor.replace_video_tokens&'BCIN 01)< <<    e  &\\1rx|| ]e\o\opQXC2 .s31S25Fs4KLp pHHbo p]^s!DNN#D$4$4$F#GGW X p ! q ps (C3C audio_inputs audio_idxc|d|}t|}tdD]&}|dzdz dzdz}|dddd|}t|}(|j|jt |j z|j S)Ninput_features_maskrr)rPranger3r2rprWr4)rErwrxmaskrt_t_outs r&replace_audio_tokenz#Gemma4Processor.replace_audio_tokens129= Iq AUQY1$q(E!9Ve$DD A  ..!$"2"2S_"D!EdnnEUVVr%c &tjjdi}|j||jddxs|jj }|jddxs|jj }|jddxs|jj}||dzz}i} |ig} |D]?} t| d| d||| \} } | |z}| |z}| j||z|dzzAdgt|z}| j| |d |\t|jd d }|Dcgc]'}|jtj||)}}| jd |it!di| Scc}w)av Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. audio_lengths (`list[int]`, *optional*): The lengths of audio inputs in number of samples. Used to dynamically compute per-audio token counts. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. r patch_sizeNpooling_kernel_sizemax_soft_tokensr{rr)heightwidthr max_patchesr)num_image_tokensnum_image_patches sampling_ratei>num_audio_tokensr$)rr#getupdater6rrrrappendrPrBr5_compute_audio_num_tokensnpzerosr)rE image_sizes audio_lengthsrFrrrrr vision_datar image_sizetarget_htarget_w patch_height patch_widthrrlengthrs r&_get_num_multimodal_tokensz*Gemma4Processor._get_num_multimodal_tokenss .77;;ORP V$"&&|T:]d>R>R>]>]   3T : fd>R>R>f>f (++,=tDlH\H\HlHl%(;Q(>>   "! ) ^ %E%a=$Q-) +(; &"( (:5 &*4  '' {(BFY[\F\(\] ^"#c+&6 6    4D[lm n  $$D$:$:OVTM^k TZ..rxx/?O       24DE F, ,,  s,FrcNt|}|jjdz}|jjdz}||z|z |jjzdz}|dkryd}d\}} } |} t |D]} | d| zz|z | zdz} t | |j S)aNumber of audio soft tokens, replicating the encoder's seq-length arithmetic. Mirrors Gemma4AudioFeatureExtractor mel framing + the two stride-2 Conv2d subsampling layers in Gemma4AudioSubSampleConvProjection, capped at ``audio_seq_length``. Must match ``audio_mask.sum()`` from the audio tower or vLLM's ``_merge_multimodal_embeddings`` will raise on a length mismatch. Args: audio_waveform: A 1-D array or list containing the raw audio samples. sampling_rate: The sampling rate of the audio waveform in Hz. Returns: The number of audio soft tokens to insert as placeholders. rr{r)rr{r)rPr5 frame_length hop_lengthr|minr-) rEaudio_waveformr num_samplesframe_size_for_unfoldpad_leftnum_mel_framessscp_num_layers sscp_kernel sscp_stride sscp_paddingrtr~s r&rz)Gemma4Processor._compute_audio_num_tokenss.) !% 6 6 C Ca G))66!;%03HHTMcMcMnMnnqrr Q 18. [, ' HAQ%% 3 CaGA H 1d++,,r%c t|dgzS)Nmm_token_type_ids)rCmodel_input_names)rErGs r&rz!Gemma4Processor.model_input_names,sw(,?+@@@r%c ddgS)Nr`rfr$)rEs r&unused_input_namesz"Gemma4Processor.unused_input_names0s+-HIIr%)Nii()NNNN)NN)rr r!rvalid_processor_kwargsrprDrr r listrrrNr rrSdictstrrbrvrrrpropertyrr __classcell__)rGs@r&r+r+1s3 # #"$6 6 6  6 t%)Z^! +T!+++d9o=EV@WW+ +  +48rs %AXXCAA*%e   H % ,E  ;JnJ JD  r%