^js`ddlZddlmZddlmZmZmZddlZddl Z ddl m Z ddl m Z mZddlmZmZmZmZddlmZmZmZdd lmZdd lmZerdd lmZGd d edZedeGddeZdgZy)N) accumulate) TYPE_CHECKINGOptionalUnion) BatchFeature) ImageInputis_valid_image)MultiModalDataProcessingKwargsProcessorMixinUnpack) AddedToken BatchEncoding TextInput)auto_docstring)requires)PreTokenizedInputc(eZdZddiddddddidZy ) ColModernVBertProcessorKwargspaddinglongestTchannels_first)return_row_col_info data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaults/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/colmodernvbert/processing_colmodernvbert.pyrr(s/ y $(+" +D1 Ir'rF)total)torch)backendsc "eZdZeZ d dededzdedzffd Ze d!de e e ze e e zde e de e e dfdedzd e ed ef d Z d"de dzde e de e e dfd e efd Z d"de dzde e de e e dfd e effd Zdeded efdZde de ed e e efdZd#dZ d#de dzd e ed efdZde e e zd e ed efdZ d$de de dfde de dfdededde defd df dZxZS)%ColModernVBertProcessorN image_seq_lenvisual_prompt_prefix query_prefixc d}tdddj|_tdddj|_tdddj|_d|_||_|j|j|_|j|j|_ |j|j |_ td D cgc]0}td D] } |jd |d zd | d zd "2c} }|_ tjd|_d|j|j|jgi} |j!| |j|j|_t# |H||fd|i||xsd|jd|_|xsd|_|j|_ycc} }w)a image_seq_len (`int`, *optional*, defaults to 64): The length of the image sequence i.e. the number of tokens per image in the input. visual_prompt_prefix (`str`, *optional*): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*): A prefix to be used for the query. NzFT) normalizedspecialzzz z*(\n?\n?|\n?)+additional_special_tokens chat_templatez<|begin_of_text|>User:z0Describe the image. Assistant:)rcontentfake_image_token image_tokenend_of_utterance_tokenglobal_image_tagr.convert_tokens_to_idsimage_token_idfake_image_token_idglobal_image_token_idrange row_col_idsrecompile%_regex_to_remove_extra_special_tokensadd_special_tokenssuper__init__r/r0query_augmentation_token) selfimage_processor tokenizerr:r.r/r0kwargsij tokens_to_add __class__s r(rLz ColModernVBertProcessor.__init__;s$ *+FSXbf g o o%iE4PXX&01ERWae&f&n&n# .*'==d>N>NO#,#B#B4CXCX#Y %.%D%DTEZEZ%["SXYZS[ NOejklem `aI + +eAE7%Awa,H I I  68ZZ@m5n2 (%%  ++*  $$]3'==d>N>NO )[=[TZ[$8% $T%5%5$66g h !).B(,(C(C%1 s5GimagestextrrQreturnc |jd||d|\}}|jd||d||jtfd|jj i|}||n |j }|djdd}|djdd}|djdd}ix} } ||j|fi|d \} } | jd d| jd d||j|| \}} |j|fi|d} |r| | d <g} t| D]e\}}g}|D]H}|d\}}| j||}| j||dz }|j||z dzJ| j|g|r|j| d| | d<|j|| dgn||jdd|i|d} ti| | |S)a  image_seq_len (`int`, *optional*): The length of the image sequence. If not provided, the default value of self.image_seq_len is used. image_seq_len should be equal to int(((image_size // patch_size) ** 2) / (scale_factor**2)) )rVrWtokenizer_init_kwargsNrreturn_text_replacement_offsetsFreturn_mm_token_type_idsrr rowscols)images_replacementstext_replacement_offsetsnew_spanr6 input_idsmm_token_type_idsimage) modalitiesrW)data tensor_typer&)prepare_inputs_layoutvalidate_inputs _merge_kwargsrrP init_kwargsr.pop_process_imagesget_text_with_replacements enumerate char_to_tokenappendcreate_mm_token_type_ids_check_special_mm_tokensr)rNrVrWr.rQ output_kwargsr[r\r image_inputs text_inputsr_r`batch_image_seq_lengthsbatch_idtext_replacement_offsetimage_seq_lensrfstartend start_id_pos end_id_poss r(__call__z ColModernVBertProcessor.__call__ps2t11UdUfU @F@@*** ) "&.."<"<  *7)B HZHZ *7 *F*J*JKlns*t'#0#?#C#CD^`e#f &}599:JDQ%'' {  0D0D0DV0n}]lOm0n -L-   VT *   VT *151P1P.A2Q2..-dnnTR]=5QR 2>VK :;*,'9BC[9\C5H5%'N 7M%)*%5 s'2'@'@5'Q %0%>%>xq%Q &--j<.G!.KL M ,22>BC,7;7T7T#K02I8K 34--dKWI-V  ($..SdSmM6RSK!@K!@rrElen) rNrVrWrQsamplen_images_in_textcumsum_images_in_textrR split_imagess r(rhz-ColModernVBertProcessor.prepare_inputs_layoutsk  $$v99;D  ))66v>Ff%!($t|#FT5M2~fQi7P#UY'Z6 T5E5E(F'Z$'Z-.C$zBR7S2T,T)"'s+;'>!-8Mb8Q8S1T0U!U t| ".t|%XFt|([$s "DDc t|||fi|| | td||Dcgc]}|j|j}}|J|Dcgc] }t |}}||k7r,td|jd|d|jd|d y|1t |r%tdt|d|jdyyycc}wcc}w) Nz+You must provide either `text` or `images`.zThe total number of zP tokens in the prompts should be the same as the number of images passed. Found  z tokens and z images per sample.zFound z. tokens in the text but no images were passed.)rKri ValueErrorrr>ranysum) rNrVrWrQrrsublistn_images_in_imagesrUs r(riz'ColModernVBertProcessor.validate_inputss 77 R R!BH%Iwc'l%I"%I#'99$.t/?/?.@A""2!31T5E5E4FlSeRffy{: C(8$9 S!1231T5E5E4FFtu%: R%Is "C Cru image_idxc |dDcgc] }|D]}| c}}|}|dDcgc] }|D]}| c}}|}|dk(rI|dk(rD|j|jz|j|jzz|jzSd} t |D]R} t |D]=} | |jd| dzd| dzdz|j|jzzz } ?| d z } T| d |j|jz|j|jzz|jzz } | Scc}}wcc}}w) Nr]r^rr;r5r6r7r8 )r=r@r>r.rE) rNrurrow_listrow image_rowscol_listcol image_colstext_split_imagesn_hn_ws r(replace_image_tokenz+ColModernVBertProcessor.replace_image_tokens*6v*>Sh(S3cScST]^ *6v*>Sh(S3cScST]^ ?zQ(()**+-%%&$*<*<<=**+- !# Z( * ,C%001!#'%ay:;!--.$2D2DDE% "T)! * T**+,**+-%%&$*<*<<=**+-   % $5TSs D:Erbrwcg}t|D]\}}tj||}tj|}tj||j k(d}d} |D]7} | t |k\rn'|| } | | z} d|| | tj|| } 9|j|j|S)Nrr6) ronparray zeros_likewhererCr searchsortedrqtolist) rNrbrwrcrR seq_lengths array_idsmm_token_typesimage_start_positionsrSseq_lenr{r|s r(rrz0ColModernVBertProcessor.create_mm_token_type_ids s'(?@ >NA{1.I]]95N$&HHY$:R:R-R$STU$V !A& @122-a0go,-uS)OO$93?  @  $ $^%:%:%< = >! r'c i}|tjjdi}|j||Dcgc] }|jj g||"}}|j dz}|j dz}t|jddddz } g} g} |D]B\} } }||zdz}| d kDr| nd }| j||z|| zz| j| D|j| | d td i|Scc}w) a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. r rz F)rJrbr6r)num_image_tokensnum_image_patchesr&) rr%getupdaterOget_number_of_image_patchesr.rrPrqr )rN image_sizesrQ vision_datar image_sizenum_image_row_colsbase_image_length col_lengthextra_split_newlinerr num_patchesnum_rowsnum_cols row_length split_extras r(_get_num_multimodal_tokensz2ColModernVBertProcessor._get_num_multimodal_tokenssR  "9CCGGY[\M   (#."A$$@@\*\m\" " !% 2 2Q 6 ++a/J #&dnnVPUn&VWb&c"dgh"h !  " 3E 6/ Xx'(2Q6 5=\1q  ''(9K(G:X`K`(ab!((5  6   4D[lm n, ,,/"s%D c v|jtfd|jji|}|dj dd}|du}t |r|g}n^t |trt |drn?t |tr$t |dtrt |dds td|Dcgc]}|jd}}|j|jgt|z||d|d }|r.|d j|d dk(d }|jd |i|Scc}w)a Prepare for the model one or several image(s). Handles input validation, RGB conversion, and prepends the `visual_prompt_prefix` to each image. Optionally computes labels from `token_type_ids` when a `suffix` is provided in `text_kwargs`. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. rZrsuffixNrzAimages must be an image, list of images or list of list of imagesRGBr )rWrVr rrbtoken_type_idsilabels)rjrrPrkrlr rrrconvertrr/r masked_fillr) rNrVrQrtrreturn_token_type_idsrd batch_docrs r(process_imagesz&ColModernVBertProcessor.process_imagesIs[<+** ) "&.."<"<  }-11(DA &d 2 & !XF  %.*C VT*z&)T/J~^def^ghi^jOk`a a5;;5%--&;;MM++,s6{:'8%m4 "  !{+77 BR8SWX8XZ^_F   h/ 022R7FSW!W$"3"3e";f"D!W !Wmm"'%m4$ "XsCquery_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec t|dk(r tdt|dk(r td|dj|djk7r td|dj|djk7r td||dj}g}t dt||D]%}g}t j jjj||||zdd} t dt||D]} t j jjj|| | |zdd} |jt jd| | jd djd |jt j|d j|j|(t j|d S) a[ Compute the late-interaction/MaxSim score (ColBERT-like) for the given multi-vector query embeddings (`qs`) and passage embeddings (`ps`). For ColQwen2, a passage is the image of a document page. Because the embedding tensors are multi-vector and can thus have different shapes, they should be fed as: (1) a list of tensors, where the i-th tensor is of shape (sequence_length_i, embedding_dim) (2) a single tensor of shape (n_passages, max_sequence_length, embedding_dim) -> usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeT) batch_first padding_valuez bnd,csd->bcnsr)dimrr6)rrdevicedtyperEr*nnutilsrnn pad_sequencerqeinsummaxrcatto) rNrrrrrscoresrR batch_scores batch_queriesrSbatch_passagess r(score_retrievalz'ColModernVBertProcessor.score_retrievals@  A %23 3 ! "a '34 4 A  % %);A)>)E)E ENO O A  $ $(:1(=(C(C CLM M  +A.44L%'q#./< ]A/1L!HHNN..;; Q^4$VW<M1c"45zB !&!3!3!@!@&q1z>:\]"A"##LL-PTTYZT[\]^bbghbi   MM%))La8;;LILL][ \ ]yyQ''r')NN@NN)NNN)NN)N)Ncpu)r"r#r$rvalid_processor_kwargsintrrLrr rrrrrrrhr ridictrrrrrrrrr __classcell__)rUs@r(r-r-6s; +/#'3D  3D "Dj 3DDj3DjJNbf$( >^T*--T*5E0FF>^I2DOTJ]E^^_>^Tz >^ 67 >^  >^>^D%)bf T! I2DOTJ]E^^_ 67 H%)bfT!I2DOTJ]E^^_)* 2%%%%:!$!QUVYQZ!_cdhildm_n!*)-Z%)@T!@67@  @D7$y/)7677  7z0449 >(^0D DE>(".$~2F"FG>( >( }- >( ^S01 >( >(r'r-) rG itertoolsrtypingrrrnumpyrr*feature_extraction_utilsr image_utilsr r processing_utilsr r r rtokenization_utils_baserrrrrutils.import_utilsrrrr-__all__r&r'r(rs~* 11 45XXKK#*< $4E  :J(nJ(J(Z % %r'