^jrNFddlmZddlZddlmZddlmZddlmZddlm Z ddl m Z dd l m Z mZmZdd lmZdd lmZdd lmZmZmZdd lmZmZddlmZddlmZe dGddej>Z Gddej>Z!Gddej>Z"edeGddeZ#edeGdde Z$eGd d!eZ%ed"Gd#d$e%Z&ed%Gd&d'e%e Z'gd(Z(y))) dataclassN)nn)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)BaseModelOutputWithPastBaseModelOutputWithPooling ModelOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringtorch_compilable_check)can_return_tuplemerge_with_config_defaults) AutoModel)Mistral3ConfigRMSNormcheZdZddeddffd Zdej dej fdZdZxZ S) Mistral3RMSNormepsreturnNct|tjt j ||_||_y)z> Mistral3RMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer __class__s y/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/mistral3/modeling_mistral3.pyrzMistral3RMSNorm.__init__*s1 ll5::k#:; # hidden_statesc"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrT)keepdim) dtypetor!float32powmeanrsqrtr$r#)r%r* input_dtypevariances r(forwardzMistral3RMSNorm.forward2sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r)c^t|jjd|jS)Nz, eps=)tupler#shaper$r%s r( extra_reprzMistral3RMSNorm.extra_repr9s* ))*+6$2G2G1HIIr))gư>) __name__ __module__ __qualname__floatrr!Tensorr6r; __classcell__r's@r(rr(s7$$$$;U\\;ell;Jr)rcxeZdZdZdeffd ZdejdejdejfdZxZ S)Mistral3PatchMergerz< Learned merging of spatial_merge_size ** 2 patches configc"t|||_|jj}|j |_|jjj |_tj||j dzz|d|_ y)NrFbias) rrrE vision_configr&spatial_merge_size patch_sizerLinear merging_layer)r%rEr&r's r(rzMistral3PatchMerger.__init__Bsr  **66 "(";";++33>>YY{T5L5La5O'OQ\chir)image_features image_sizesrc|Dcgc]&}|d|jz|d|jzf(}}|Dcgc] \}}||z }}}|jd}g}t|j|D]\} } || \}}| j |||j dddj d} tjjj| |j|j} | j ||jdzzdj} |j| tj|d}|j|}|Scc}wcc}}w)Nrrr,r) kernel_sizestridedim)rKr9 enumeratesplitviewpermute unsqueezer!r functionalunfoldrJtappendcatrM) r%rNrO image_sizehwtokens_per_imagedpermuted_tensor image_index image_tokens image_gridgrids r(r6zMistral3PatchMerger.forwardKslcn U_Z]doo -z!}/O P  /::daAE::   $)2>3G3GHX3Y)Z ) %K{+DAq%**1a3;;Aq!DNNqQJ88&&--(?(?H_H_.D99Q!8!8!!;;R@BBDD  " "4 ( )?:++N;) ;s +E"E') r<r=r>__doc__rrr!r@r6rArBs@r(rDrD=s?j~jellRWR^R^r)rDc\eZdZdeffd Zdej dej fdZxZS)Mistral3MultiModalProjectorrEct|t|jj|j j |_t||_ t|jtrdnt|j|_tj |jj|jz|j j|j"|_t&|j(|_tj |j j|j j|j"|_y)N)rrrG)rrrrIr& text_config rms_norm_epsnormrD patch_merger isinstancevision_feature_layerintlennum_feature_layersrrLmultimodal_projector_biaslinear_1rprojector_hidden_actactlinear_2r%rEr's r(rz$Mistral3MultiModalProjector.__init__ds #F$8$8$D$D&J\J\JiJij /7F77=A3vGbGbCc   , ,t/F/F F    * *11  &556    * *F,>,>,J,JQWQqQq  r)rNrOc|j|}|j||}|j|}|j|}|j |}|SN)rorprwryrz)r%rNrOr*s r(r6z#Mistral3MultiModalProjector.forwardvsR>2**>;G n5 /  m4 r)) r<r=r>rrr!r@r6rArBs@r(rkrkcs* ~ $ellr)rkzT Base class for Mistral3 causal language model (or autoregressive) outputs.  custom_introceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) Mistral3CausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitspast_key_valuesr* attentionsimage_hidden_states)r<r=r>rirr! FloatTensor__annotations__rrrr*r8rrr)r(rrs &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r)rzM Base class for Mistral3 outputs, with hidden states and attentions. c:eZdZUdZdZej dzed<y)Mistral3ModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nr)r<r=r>rirr!rrrr)r(rrs 59**T18r)rc>eZdZUeed<dZdZdZdgZdZ dZ dZ dZ dZ y)Mistral3PreTrainedModelrEmodel)imagetextTrN)r<r=r>rrbase_model_prefixinput_modalitiessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_flex_attn_supports_attention_backendrr)r(rrs@ (&*##4"5N!"&r)rzx The Mistral3 model which consists of a vision backbone and a language model, without a language modeling head. cVeZdZdeffd Zeeed ddejdejde e e ze e zdzd e dzd eed eezf d Zd ej&dejdejfdZeee dd ej&dzdejdzdejdzdej&dzdedzdejdzde e e ze e zdzde dzdejdzd eed eezfdZxZS) Mistral3ModelrEct||tj|j|_t ||_tj|j|_ |jyr}) rrr from_configrI vision_towerrkmulti_modal_projectorrmlanguage_model post_initr{s r(rzMistral3Model.__init__sY  %11&2F2FG%@%H"'33F4F4FG r)zWObtains image last hidden states from the vision tower and apply multimodal projection.r~N pixel_valuesrOrroutput_hidden_stateskwargsrc |jDcic] \}}| || }}}|j|f|ddd|}t|tr|j|} n3|D cgc]} |j| } } t j | d} |j| jd|} |jj|jjz} t j|| j| zjdj}t j | jd|} | |_|Scc}}wcc} w)NT)rOr return_dictr,rSr)device)itemsrrqrsr*r!r^rsqueezerKrErJ as_tensorrprodtolistrV pooler_output)r%rrOrrrrkv image_outputsselected_image_feature layer_idxhs_poolrNdownsample_ratio split_sizess r(get_image_featuresz Mistral3Model.get_image_featuressS$*<<>C41aQ]!Q$CC)))  #!%     *C 0%2%@%@AU%V "Ocd)}229=dGd%*YYwB%? "334J4R4RST4UWbc,,77$++:X:XX __[1F1F GK[ [ a afh a i p p r ^%;%;A%> L&4 #3Des EE"E input_ids inputs_embedsrNc.|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd|jdz}|jdj|j }t||jdz|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )r.rr,rrz6Image features and image tokens do not match, tokens: z , features: )get_input_embeddingsr!tensorrEimage_token_idlongrallsumr9rYr/rnumel)r%rrrNspecial_image_maskn_image_tokensn_image_featuress r(get_placeholder_maskz"Mistral3Model.get_placeholder_masks   !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2^5I5I!5LL/99"=@@AUAUV ]004 48L8L8N ND^DTT`aq`r s "!r)attention_mask position_idsr use_cachec |du|duz r td||j|}||j||| dj} t j | dj |j|j} |j||| } |j| | }|jd |||||d| } t| j| j| j| j | SdS) Nz:You must specify exactly one of input_ids or inputs_embedsT)rrrrOrrrS)rrN)rrrrr)last_hidden_staterr*rrr) ValueErrorrrrr!r^r/rr.rmasked_scatterrrrrr*r)r%rrrrrrrrrrOrrNroutputss r(r6zMistral3Model.forwardsH  -t"; <YZ Z  7D557 BM  #!44)%9' 5 m  #YY~1=@@AUAUWdWjWjkN!%!:!:~";" *889K^\M%$%% )%+'    +%77#33!//))2>2J   QU   r))NN) NNNNNNNNN)r<r=r>rrrrrr!rr@rslistboolrrr8r r LongTensorrrrr6rArBs@r(rrs ~ nDH,0 !''!\\!"DIoS 9D@ ! #Tk ! +, ! + +! !F"))":?:K:K"]b]n]n"0 .215.204(,26CG!%+// ##d*/ ''$./  t+ / &&- /  / ((4// "DIoS 9D@/ $;/ \\D(/ +,/  , ,/  / r)rzV The MISTRAL3 model which consists of a vision backbone and a language model. cHeZdZddiZdeffd ZdejfdZe e e dde jd e jd eeezeezdzd eedeezf d Ze e e dd e j,dzde jdzde jdzde j,dzdedzde jdzde j,dzdedzdee jzd e jdzd eedeezfdZ dfd ZxZS) Mistral3ForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightrEct||t||_t j |j j|j jd|_ |jy)NFrG) rrrrrrLrmr& vocab_sizelm_headrr{s r(rz)Mistral3ForConditionalGeneration.__init__MsS  "6* yy!3!3!?!?ASASA^A^ejk  r)rc|jSr})rr:s r(get_output_embeddingsz6Mistral3ForConditionalGeneration.get_output_embeddingsSs ||r)NrrOrrrc B|jjd|||d|S)N)rrOrrr)rr)r%rrOrrrs r(rz3Mistral3ForConditionalGeneration.get_image_featuresVs5-tzz,, %#!5    r)rrrrrlabelsrlogits_to_keepc |jd|||||||| d| } | d} t| tr t| dn| }|j | dd|ddf}d}|4|j d|||j jjd| }t||| j| j| j| jS)a1 Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, Mistral3ForConditionalGeneration >>> model = Mistral3ForConditionalGeneration.from_pretrained("mistralai/Mistral-Small-3.1-24B-Instruct-2503") >>> processor = AutoProcessor.from_pretrained("mistralai/Mistral-Small-3.1-24B-Instruct-2503") >>> prompt = "[INST][IMG]What is the image?[/INST]" >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=15) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "What is the image?The image depicts two cats lying on a pink blanket." ```)rrrrrrrrOrN)rrr)rrrr*rrr)rrqrsslicer loss_functionrErmrrrr*rr)r%rrrrrrrrrrOrrr* slice_indicesrrs r(r6z(Mistral3ForConditionalGeneration.forwardgsR$**  %)%+'#      8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD.#33!//)) ' ; ;   r)c ft ||f|||||d|} |s|jdds|| d<| S)N)rrrris_first_iterationrTr)rprepare_inputs_for_generationget) r%rrrrrrrr model_inputsr's r(rz>Mistral3ForConditionalGeneration.prepare_inputs_for_generationsYw<  +'))1    VZZ T%B ,8L (r)r}) NNNNNNNNrN)NNNNNF)r<r=r>_tied_weights_keysrrrModulerrrrr!rr@rsrrrr8r rrrrrr6rrArBs@r(rrEs +,VW~ ryy  DH  ''  \\  "DIoS 9D@  +,  + +      .215.204(,26*.!%-.+/C ##d*C ''$.C  t+ C &&- C  C ((4/C   4'C $;C ell*C \\D(C +,C  / /C  C P r)r)rrr)) dataclassesrr!r activationsr cache_utilsr generationr integrationsr modeling_outputsr r r modeling_utilsr processing_utilsrutilsrrr utils.genericrrautorconfiguration_mistral3rrrrDrkrrrrr__all__rr)r(rsW," ! )7``-&OOI2Y'JbiiJ(J(#"))#L"))8  9[9  90   9"9 9  9 'o ' ' { +{  { | C'>C  CL [r)