^jl8 (dZddlZddlmZddlmZmZddlm Z m Z m Z ddl m Z mZddlmZGd d e d Zd eededeeefdZdeeeedeeedededej*f dZdedededefdZeGdde ZdgZy)zProcessor class for Mllama.N) BatchFeature) ImageInputmake_nested_list_of_images)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)auto_docstringceZdZdddiiZy)MllamaProcessorKwargs image_kwargsmax_image_tilesN)__name__ __module__ __qualname__ _defaultsw/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/mllama/processing_mllama.pyrrs q IrrF)total input_idsimage_token_idreturnct|Dcgc] \}}||k(s |}}}t|dk(rgSt|dk(r|ddggSt|dd|ddDcgc] \}}||g }}}|j|dt|g|dd}|dddD]} | d| ddz k(r|| d<| d}|Scc}}wcc}}w)a Generate a cross-attention token mask for image tokens in the input sequence. This function identifies the positions of image tokens in the input sequence and creates a mask that defines which subsequent tokens each image token should attend to. Args: input_ids (list[int]): A list of token ids representing the input sequence. image_token_id (int): The id of the token used to represent images in the sequence. Returns: list[list[int]]: A list of [start, end] pairs, where each pair represents the range of tokens an image token should attend to. Notes: - If no image tokens are present, an empty list is returned. - For a single image token, it attends to all subsequent tokens until the end of the sequence. - For multiple image tokens, each attends to tokens up to the next image token or the end of the sequence. - Consecutive image tokens are treated as a group and attend to all subsequent tokens together. rN) enumeratelenzipappend) rritokenimage_token_locationsloc1loc2 vision_masks last_mask_end vision_masks rget_cross_attention_token_maskr,"s,09/C_81euP^G^Q__ !Q&  !Q&&q)2.//367LSb7QShijikSl3mnZT4T4LnLn.r2C NCD !$Q'M#DbD)' q>[^a/ /*KN#A ' /`os B>B>Ccross_attention_token_mask num_tiles max_num_tileslengthc zt|}td|D}tj||||ftj}t t ||D]\\}\}} t t || D]<\} \} } t| dk(s| \} }t||}|dk(r|}d||| || d| f<>^|S)a Convert the cross attention mask indices to a cross attention mask 4D array. This function takes a sparse representation of cross attention masks and converts it to a dense 4D numpy array. The sparse representation is a nested list structure that defines attention ranges for each image in each batch item. Args: cross_attention_token_mask (list[list[list[int]]]): A nested list structure where: - The outer list represents the batch dimension. - The middle list represents different images within each batch item. - The inner list contains pairs of integers [start, end] representing token ranges for each image. num_tiles (list[list[int]]): A nested list structure specifying the number of tiles for each image in each batch item. max_num_tiles (int): The maximum possible number of tiles. length (int): The total sequence length of the input. Returns: np.ndarray: A 4D numpy array of shape (batch_size, length, max_num_images, max_num_tiles) The array contains `1` where attention is allowed and `0` where it is not. Note: - Special handling is done for cases where the end token is -1, which is interpreted as attending to the end of the sequence. c32K|]}t|ywNr!).0maskss r z?convert_sparse_cross_attention_mask_to_dense..psLUL)shapedtyperrN)r!maxnpzerosint64r r"min)r-r.r/r0 batch_sizemax_num_imagescross_attention_mask sample_idx sample_maskssample_num_tilesmask_idx locationsmask_num_tilesstartends r,convert_sparse_cross_attention_mask_to_denserLRs:/0JL1KLLN886>=Ahh 9B#F`bkBl8m[4 4\#35>s>> build_string_from_input("Hello world", "", "<|image|>") 'Hello world' >>> build_string_from_input("<|image|>Hello world", "", "<|image|>") '<|image|>Hello world' >>> build_string_from_input("Hello world", "", "<|image|>") 'Hello world' rNr) startswithr!)rMrNrOnum_image_tokens_on_starts rbuild_string_from_inputrSsn4F !   K (K(*+!Q&!   K (55 6yk& JJrc <eZdZeZdfd Ze ddedzdee ze eze e zdzde ede fdZ ddedzdee ze eze e zdzffd Z ddedzdee ze eze e zdzde effd Zd ed edefd Z dd ZedZxZS)MllamaProcessorNcFt|ds(d|_|j|j|_n"|j|_|j|_d|_|j|j|_|j |_t|!|||y)NrOz <|image|>z<|python_tag|>) chat_template) hasattrrOconvert_tokens_to_idsr python_tokenpython_token_idrNsuper__init__)selfimage_processor tokenizerrW __class__s rr]zMllamaProcessor.__init__sy-0*D "+"A"A$BRBR"SD (44D "+":":D ,(>>t?P?PQ",, )=Qrimagestextkwargsrc |j||\}}|jd||d||jtfd|jj i|}|dj dd}i}|+|j|fi|d}|j||dgi}|*|j|fi|d\}}|j d } |c|a|d D cgc]} t| |j} } t|  |jjtd |d D } | |d <ti|||Scc} w)a Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. TODO: add aspect_ratio_ids and aspect_ratio_mask and cross_attention_mask rbrctokenizer_init_kwargs text_kwargsreturn_tensorsNimage) modalities images_kwargsr.rc32K|]}t|ywr3r4)r5rs rr7z+MllamaProcessor.__call__..sTi3y>Tr8)r.r/r0rC)data tensor_typer)prepare_inputs_layoutvalidate_inputs _merge_kwargsrr` init_kwargspop_check_special_mm_tokens_process_imagesr,rrLr_rr<r) r^rbrcrd output_kwargsri text_inputs image_inputs_r. token_idsr-rCs r__call__zMllamaProcessor.__call__s$11d1K @F@@*** ! "&.."<"<  '}599:JDQ  ($..N}1MNK  ) )$  ) R  2d226\]?=[\OL!$((5I  $"2"-[!9*/y$:M:MN* &*$P*#"22BBT;{;STT $ 3GK. /!@K!@.sH)yA~Hc3&K|] }|dk( ywrrrs rr7z2MllamaProcessor.validate_inputs..sU'0INUrzaIf a batch of text is provided, there should be either no images or at least one image per samplezZMake sure to pass your images as a nested list, where each sub-list holds images per batchz)The number of image tokens in each text (zA) should be the same as the number of provided images per batch (z). ) r\rqcountrOsum ValueErrorrr!anyall) r^rbrcrdtn_images_in_textsamplen_images_in_images add_messageras rrqzMllamaProcessor.validate_inputss@ 77  CGHa(8(8 9H H#$q(V^ !cdd#3F;@F%Gfc&k%G"%GH7GHHQTU4DUR%{&)99"$K-.#6F2GGL^brLr'C $CDTCUV@@R?SSVWbVce :$ H &Hs "C()C-ry image_idxc|jSr3)rO)r^ryrs rreplace_image_tokenz#MllamaProcessor.replace_image_token!src B|jj|f||d|S)a Post-process the output of the model to decode the text. Args: generated_outputs (`torch.Tensor` or `np.ndarray`): The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)` or `(sequence_length,)`. skip_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to clean up the tokenization spaces. Argument passed to the tokenizer's `batch_decode` method. **kwargs: Additional arguments to be passed to the tokenizer's `batch_decode method`. Returns: `list[str]`: The decoded text. )skip_special_tokensclean_up_tokenization_spaces)r` batch_decode)r^generated_outputsrrrds rpost_process_image_text_to_textz/MllamaProcessor.post_process_image_text_to_text&s5(+t~~**  3)E    rc|jj}|jj}|Dcgc] }|dk7s | }}t||zdgzScc}w)Nr.rC)r`model_input_namesr_list)r^tokenizer_input_namesimage_processor_input_namesnames rrz!MllamaProcessor.model_input_namesAse $ @ @&*&:&:&L&L#9T&kW[_jWjt&k#&k),GGKaJbbcc'ls AAr3)NN)TF)rrrrvalid_processor_kwargsr]r rr r rr rr|rprrqdictintstrrrpropertyr __classcell__)ras@rrUrUsX2 R%)ae3^T!3^++d9o=EV@WWZ^^3^./ 3^  3^3^n%)aeT!++d9o=EV@WWZ^^&%)ae T! ++d9o=EV@WWZ^^ )* D    Y^ 6ddrrU)__doc__numpyr=feature_extraction_utilsr image_utilsrrprocessing_utilsrrr tokenization_utils_baser r utilsr rrrr,ndarrayrLrrSrU__all__rrrrs"4AHHC#,E-d3i--QUVZ[^V_Q`-`- $T$s)_ 5- DI- -   - ZZ - `"KC"KC"Kc"Kc"KJadnadadH  r