^j?=ddlmZddlZddlmZddlmZddlmZ ddl m Z ddl m Z mZddlmZmZdd lmZdd lmZmZmZd d lmZmZmZmZmZmZd d lm Z d dl!m"Z"m#Z#ejHe%Z&edeGdde Z'GddeZ(Gdde"Z)GddeZ*GddeZ+GddeZ,GddeZ-eGdd eZ.Gd!d"eZ/gd#Z0y)$)CallableN)strict)initialization)PreTrainedConfig)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)CLIPMLP CLIPAttention CLIPEncoderCLIPEncoderLayerCLIPVisionEmbeddingsCLIPVisionModel)eager_attention_forward)VisionRotaryEmbeddingapply_rotary_pos_emb_visionz&DeepGlint-AI/mlcd-vit-bigG-patch14-336) checkpointceZdZUdZdZdZdZeed<dZ eed<dZ eed <d Z eed <d Z eed <dZ eed<dZeeezeeefzed<dZeeezeeefzed<dZeed<dZeed<dZeezed<dZeed<dZeed<y)MLCDVisionConfigav num_key_value_groups (`int`, *optional*, defaults to 1): Number of key-value groups used in Attention. Example: ```python >>> from transformers import MLCDVisionConfig, MLCDVisionModel >>> # Initializing a MLCDVisionConfig with DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> configuration = MLCDVisionConfig() >>> # Initializing a MLCDVisionModel (with random weights) from the DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> model = MLCDVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```mlcd_vision_model vision_configi hidden_sizei intermediate_size0num_hidden_layersnum_attention_headsnum_key_value_groupsr num_channelsiP image_size patch_sizegelu hidden_actgh㈵>layer_norm_epsattention_dropoutg{Gz?initializer_range?initializer_factorN)__name__ __module__ __qualname____doc__ model_typebase_config_keyrint__annotations__r r"r$r&r'r(listtupler*r,strr-floatr/r0r2p/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/mlcd/modular_mlcd.pyrr)s &%J%OK!s!s!! !#!L#47Jd3i%S/1746Jd3i%S/16J NE %(us{(#u# ##r@rc eZdZy)MLCDMLPNr3r4r5r?r@rArCrCQr@rCc eZdZy)MLCDRotaryEmbeddingNrDr?r@rArGrGUrEr@rGc\eZdZdeffd Zdej dejfdZxZ S)MLCDVisionEmbeddingsconfigc(t|||`yN)super__init__position_embeddingselfrJ __class__s rArNzMLCDVisionEmbeddings.__init__Zs   #r@ pixel_valuesreturncT|jd}|jjj}|j|j |}|j dj dd}|jj|dd}tj||gd}|S)Nrdtyperr%dim) shapepatch_embeddingweightrWtoflatten transposeclass_embeddingexpandtorchcat)rQrS batch_size target_dtype patch_embeds class_embeds embeddingss rAforwardzMLCDVisionEmbeddings.forward^s!''* ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY l;C r@) r3r4r5rrNrc FloatTensorTensorrj __classcell__rRs@rArIrIYs-$/$ E$5$5 %,, r@rIceZdZdZdeffd Z d dejdeejejfdejdzde e d eejejdzff d Z xZ S) MLCDAttentionzMulti-headed attention with RoPE. Refer to papers: - Attention is all you need: https://huggingface.co/papers/1706.03762 - RoFormer: Enhanced Transformer with Rotary Position Embedding: https://huggingface.co/papers/2104.09864 rJcTt|||j|_d|_y)NF)rMrNr& is_causalrPs rArNzMLCDAttention.__init__ss%  $*$?$?!r@N hidden_statesposition_embeddingsattention_maskkwargsrTc |jdd\}}|j|j|||j|jf}|j |j|||j|jf}|j |j|||j|jf} |djdj} |djdj} t||| | \}}|jddddj}|jddddj}| jddddj} tj|jjt } | |||| |f|j"sdn |j$|j&|j(d|\} }| jddddj} | j+||d} |j-| } | jdddj} | |fS)NrXrr%rrr.)dropoutscalingrr)r[q_projreshape num_headshead_dimk_projv_proj unsqueezer>rpermute contiguousr get_interfacerJ_attn_implementationrtrainingrxscalerrviewout_proj)rQrsrtrurvre seq_length query_states key_states value_statescossinattention_interface attn_output attn_weightss rArjzMLCDAttention.forwardxs;"/!4!4Sb!9 J{{=199:zSWSaSacgcpcp:qr [[/77ZQUQ_Q_aeanan8op {{=199:zSWSaSacgcpcp:qr "!$..q1779!!$..q1779#>|ZY\^a#b j$++Aq!Q7BBD ''1a3>>@ #++Aq!Q7BBD (?(M(M KK , ,.E) %8      % $}}C$,,JJnn %  % ! \"))!Q15@@B !&&z:rB mmK0 !))!Q2==? L((r@rL) r3r4r5r6rrNrcrlr<r r rjrmrns@rArprpks//3 ,)||,)#5<<#=>,) t+ ,) +, ,) u||U\\D00 1 ,)r@rpc eZdZdeffd Z d dej deej ej fdej dzdee deejf d Z xZ S) MLCDEncoderLayerrJcDt||t||_yrL)rMrNrp self_attnrPs rArNzMLCDEncoderLayer.__init__s  &v.r@NrsrtrurvrTc |}|j|}|jd|||d|\}}||z}|}|j|}|j|}||z}|S)a Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. Represents the hidden states from the previous layer or the input embeddings. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. )rsrtrur?) layer_norm1r layer_norm2mlp)rQrsrtrurvresidual_s rArjzMLCDEncoderLayer.forwards$!((7 )4>> ' 3)    q !=0  ((7 /  =0 r@rL) r3r4r5rrNrcrlr<r r rkrjrmrns@rArrsz////3 "||"#5<<#=>" t+ " +, " u  ! "r@rc eZdZdZdeffd Z d dejdeejejfdejdzde e d ee zf d Z xZS) MLCDEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MLCDEncoderLayer`]. Args: config: MLCDVisionConfig rJc$t||y)z3Overwrite dummy `MLCDConfig` to `MLCDVisionConfig`.N)rMrNrPs rArNzMLCDEncoder.__init__s  r@N inputs_embedsrtrurvrTc V|}|jD] }||||fi|}t|S)a= Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) )last_hidden_state)layersr)rQrrtrurvrs encoder_layers rArjzMLCDEncoder.forwardsK,& ![[ M)# M +  r@rL)r3r4r5r6rrNrcrkr<rlr r rrjrmrns@rArrs{!/!/3 ! ((! #5<<#=>!  t+ ! +, !  ! r@rc~eZdZUeed<dZdgZdZdZdZ dZ dZ dZ e edZej"fdZxZS)MLCDPreTrainedModelrJ vision_modelrTF)rs attentionsct|||jj}t |t r|jj}t j|jd|jdz|zt j|jj|jj|zt j|jtj |jj"dj%dy t |t&r|jj}|jdzd|jj(zdzz|z}|jdz|z}t j|j*j|t j|j,j|t j|j.j|t j|j0j|y t |t2r|jj}|jj4dzd|jj(zdzz|z}d|jj4zdz|z}t j|j6j|t j|j8j|y t |t:ro|jj}|jj4|jj<zdzdz|z}t j|j>d|y t |t@rod|jBtj d |jDdtjF |jDz zz }t j|jH|y y ) zInitialize the weightsr.g)meanstd)rrX)r%rXrr1rrVN)%rM _init_weightsrJr2 isinstancerIinitnormal_ra embed_dimr\r]r0copy_ position_idsrcaranger[rbrpr"rzr~rrrCrfc1fc2MLCDVisionModelr$ class_pos_embrGthetarZr>inv_freq) rQmodulefactor in_proj_std out_proj_stdfc_std pos_emb_stdrrRs rArz!MLCDPreTrainedModel._init_weightss f%// f2 3[[33F LL//cv?O?OQU?UX^?^ _ LL//66FMM<[<[^d>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> from transformers import AutoProcessor, MLCDVisionModel >>> model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs, output_attentions=True) >>> features = outputs.last_hidden_state >>> print(f"Extracted features shape: {features.shape}") >>> print(f"Number of attention layers: {len(outputs.attentions)}") >>> print(f"Attention shape: {outputs.attentions[0].shape}") ```Nz You have to specify pixel_valuesrX)devicer%rrY)rrt)r pooler_outputr?) ValueErrorr[rJr*rcrrrrbstackr_rrdrrrri pre_layrnormencoderpost_layernormr )rQrSrvnum_patches_heightnum_patches_widthhpos_idswpos_idspos_idsrotary_pos_embembrtrsencoder_outputsr pooled_outputs rArjzMLCDVisionModel.forward9s:  ?@ @)//3t{{7M7MM(..r2dkk6L6LL LL+L4G4G H R RST U \ \]_ar s  LL*<3F3F G Q QRS T [ [\npr s ++x//183C3C3EFBO55g>D$6$6#GQOii8bA"wwy#'')4 5 ))-8 &$,, ' 3   ,A.)!Q'2 ++M: )/'  r@rL) r3r4r5rrNrcrkr r r<r rjrmrns@rArr3sOq/q26> ''$.> +,>  + + > r@r)rrr)1collections.abcrrctorch.nnrhuggingface_hub.dataclassesrrrconfiguration_utilsrmodeling_outputsrr modeling_utilsr r processing_utilsr utilsr rrclip.modeling_cliprrrrrrllama.modeling_llamarqwen2_vl.modeling_qwen2_vlrr get_loggerr3loggerrrCrGrIrprrrr__all__r?r@rArs% .&3KF&@@;[   H %CD#$'#$E#$L g  / /$9)M9)x'''T. +. b-2/-2-2`D oD N r@