^jX%ddlZddlmZmZmZddlmZmZeje Z GddedZ eGd d eZ d gZy) N)MultiModalDataProcessingKwargsProcessorMixin)auto_docstringloggingc"eZdZddddddidZy)Glm4vProcessorKwargsFT)paddingreturn_token_type_idsreturn_mm_token_type_idsreturn_metadata) text_kwargs videos_kwargsN)__name__ __module__ __qualname__ _defaultsu/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/glm4v/processing_glm4v.pyr r s#%*(, ,T2 Irr F)totalceZdZeZdfd ZdededefdZ dededefdZ dd Z dd Z e fd Zd edeeefd ZddefdZxZS)Glm4vProcessorc t|dsdn |j|_t|dsdn |j|_t|ddr |jn|j |j|_t|ddr |j n|j |j|_t|!|||||j d|_ |j d |_ y) N image_tokenz <|image|> video_tokenz <|video|>image_token_idvideo_token_id) chat_templatez<|begin_of_video|>z<|end_of_video|>) hasattrrrgetattrrconvert_tokens_to_idsrsuper__init__video_start_id video_end_id)selfimage_processor tokenizervideo_processorr kwargs __class__s rr%zGlm4vProcessor.__init__-s.5i.O;U^UjUj.5i.O;U^UjUjy"2D9  $ $001A1AB y"2D9  $ $001A1AB  )_Tab'==>RS%;;sI++669 '(89)DIIK|["222r video_inputs video_idxcl|jjdz}|d|d}|d|j|z|z}|d|}d}|jtj d|jdn |j|_|j ddd}g} tdt|D]} | j|| | d|} t| |kr'| j| r| dndt| |kr't|D]} | | } |j| | }||z }!|S) Nr2video_grid_thwrvideo_metadataaGLM4V requires frame timestamps to construct prompts, but the `fps` of the input video could not be inferred. Probably `video_metadata` was missing from inputs and you passed pre-sampled frames. Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results.)r7) r+r4r5fpslogger warning_once timestampsrangelenappendreplace_frame_token_id)r(r9r:r6 num_framesr7metadatavideo_structurerDunique_timestampsidxselected_timestamps frame_idx timestamp_secframe_structures rreplace_video_tokenz"Glm4vProcessor.replace_video_tokenCsr++669 !"23I>qA '(89)DIIK|[_ii 01)< <<    e  &\\1rx|| ((1- C O, 6C  $ $Z_ 5 60 <%&3  & &BU':2'>[\ ]%&3z* /I/ :M"99-Zj9kO  .O / rc i}|tjjdi}|j||jddxs|jj }|Dcgc] }|jj g||"}}|D cgc] } | |dzz } } |j| |d|vtjjdi} | j||D cgc] } |jjg| | "} } | D cgc] } | dzz }} ||d<tdi|Scc}wcc} wcc} wcc} w) aK Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. video_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (num_frames, height, width) per each video. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. N images_kwargsr4r2)r7num_image_patchesrnum_video_tokensr) r rgetupdater)r4get_number_of_image_patchesr+get_number_of_video_patchesr)r( image_sizes video_sizesr, vision_datarTr4 image_sizerU num_patchesr7r video_sizenum_video_patchesrVs r_get_num_multimodal_tokensz)Glm4vProcessor._get_num_multimodal_tokensbs  "0::>>PRSM   (&**<>a$BVBVBaBaJ#.!A$$@@\*\m\! !Sdd; A !=d d   4D[lm n  "0::>>PRSM   (#.!A$$@@\*\m\! !Sdd; A !=d d.>K* +, ,,#! e ! es$%D7D<+%EEc B|jj|f||d|S)a Post-process the output of the model to decode the text. Args: generated_outputs (`torch.Tensor` or `np.ndarray`): The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)` or `(sequence_length,)`. skip_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to clean up the tokenization spaces. Argument passed to the tokenizer's `batch_decode` method. **kwargs: Additional arguments to be passed to the tokenizer's `batch_decode method`. Returns: `list[str]`: The decoded text. )skip_special_tokensclean_up_tokenization_spaces)r* batch_decode)r(generated_outputsrdrer,s rpost_process_image_text_to_textz.Glm4vProcessor.post_process_image_text_to_texts5(+t~~**  3)E    rc t|dgzS)Nmm_token_type_ids)r$model_input_names)r(r-s rrkz Glm4vProcessor.model_input_namessw(,?+@@@r input_idscg}|D]}tj|}tj|}tj||jk(d}tj||j k(d}||kD}d|||j k(|z<d|||j k(|z<|j|j|S)Nr)axisr2) nparray zeros_likecumsumr&r'rrGtolist) r(rlrjinput array_idsmm_token_typesstartsendsis_video_modalitys rcreate_mm_token_type_idsz'Glm4vProcessor.create_mm_token_type_idss >EI]]51N YYyD,?,??aHF99Y$*;*;;!DD & UVNI)<)<<@QQ RXYNI)<)<! rr7c>d|j|zdt|S)Nz<|begin_of_image|>z<|end_of_image|>)rint)r(rPr7s rrHz%Glm4vProcessor.replace_frame_token_ids+#D$4$47G$G#HHXY\]jYkXlmmr)NNNN)NN)TF)ro)rrrr valid_processor_kwargsr%dictr}strr8rRrbrhpropertyrklistr{rH __classcell__)r-s@rrr)s1P"3333 >$-NY^ 6AA!$!4S ?!*ncnrr)numpyrpprocessing_utilsrrrutilsrr get_loggerrrBr r__all__rrrrsb*PP,   H %+5Sn^SnSnl  r