^j"ddlZddlZddlmZddlmZddlmZmZddl Z ddl m Z ddl m cm Zddl mZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddl m!Z!ddl"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4ddl5m6Z6m7Z7m8Z8m9Z9ddl:m;Z;ddlZ>ddl?m@Z@mAZAmBZBedGdde jZDGdde jZEGdd e jZFGd!d"e jZGGd#d$e jZHGd%d&e jZId'ZJd(e jd)e jd*e jd+e jd,eLe je jff d-ZMd.e jd/eNd,e jfd0ZO dWd1e jd2e jd3e jd4e jd5e jdzd6ePd7ePd8e-e/fd9ZQGd:d;e jZRGd<d=e!ZSGd>d?e jZTd@ZUdXdAZVGdBdCe jZWGdDdEe jZXGdFdGe!ZYe0eGdHdIe#ZZe0GdJdKe+Z[GdLdMe[Z\e0GdNdOe[Z]e0GdPdQe[Z^e0eGdRdSe%Z_GdTdUe[eZ`gdVZay)YN)Callable) dataclass)AnyOptional) LayerNorm)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)deprecate_kwarg)accepts_precomputed_kwargsis_flash_attention_requestedmaybe_autocastmerge_with_config_defaults)capture_outputs)get_vision_cu_seqlensget_vision_position_ids) Glm4vConfigGlm4vTextConfigGlm4vVisionConfigRMSNormcheZdZddeddffd Zdej dej fdZdZxZ S) Glm4vRMSNormepsreturnNct|tjt j ||_||_y)z; Glm4vRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizer- __class__s s/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/glm4v/modeling_glm4v.pyr1zGlm4vRMSNorm.__init__:s1 ll5::k#:; # hidden_statesc"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NT)keepdim) dtypetor4float32powmeanrsqrtr7r6)r8r= input_dtypevariances r;forwardzGlm4vRMSNorm.forwardBsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r<c^t|jjd|jS)Nz, eps=)tupler6shaper7)r8s r; extra_reprzGlm4vRMSNorm.extra_reprIs* ))*+6$2G2G1HIIr<)gư>) __name__ __module__ __qualname__floatr1r4TensorrJrN __classcell__r:s@r;r,r,8s7$$$$;U\\;ell;Jr<r,c,eZdZddeffd ZdZxZS) Glm4VisionMlpbiasct||j|_|j|_t j |j|j||_t j |j|j||_t j |j|j||_ t|j|_ yNrX) r0r1r9out_hidden_sizeintermediate_sizer2Linear gate_projup_proj down_projr hidden_actact_fn)r8configrXr:s r;r1zGlm4VisionMlp.__init__Ns !--!'!7!74#3#3T5K5KRVWyy!1!143I3IPTU 4#9#94;K;KRVWV../ r<c|j|j|j||j|zSN)rarcr_r`r8 hidden_states r;rJzGlm4VisionMlp.forwardWs2~~dkk$..*FG$,,WcJddeer<F)rOrPrQboolr1rJrTrUs@r;rWrWMs0T0fr<rWc`eZdZdeddffd Zdej dej fdZxZS)Glm4vVisionPatchEmbedrdr.NcTt||j|_|j|_|j|_|j |_|j|j|jg}tj|j|j |||_ y)N) kernel_sizestride) r0r1 patch_sizetemporal_patch_size in_channelsr9 embed_dimr2Conv3dproj)r8rdrnr:s r;r1zGlm4vVisionPatchEmbed.__init__\s  ++#)#=#= !--++//$//R IId..K`kl r<r=c6|jjj}|jd|j|j |j |j }|j|j|jd|j}|S)Nr@rB) rur6rBviewrrrqrprCrs)r8r= target_dtypes r;rJzGlm4vVisionPatchEmbed.forwardfs~yy''-- %**   $":":DOOT__  -"2"2"2"FGLLRQUQ_Q_` r< rOrPrQr)r1r4rSrJrTrUs@r;rlrl[s5m0mTmU\\ellr<rlceZdZUejed<d dededdffd ZdejdejfdZ xZ S) Glm4vVisionRotaryEmbeddinginv_freqdimthetar.Nct|||_||_d|t j d|dtj |z zz }|jd|dy)N?rr?rwr}F persistent)r0r1r~rr4arangerRregister_buffer)r8r~rr}r:s r;r1z#Glm4vVisionRotaryEmbedding.__init__rsY  %ELLC%++$NQT$TUV ZeDr< position_idsc\|jd|jzjdS)Nr@r&) unsqueezer}flatten)r8rs r;rJz"Glm4vVisionRotaryEmbedding.forwardys'&&r*T]]:CCAFFr<)g@) rOrPrQr4rS__annotations__intrRr1rJrTrUs@r;r|r|osIllECEEDEGELLGU\\Gr<r|c neZdZd dededededdf fd Zdejdejfd Z xZ S) Glm4vVisionPatchMergerr~ context_dimrbrXr.Ncxt|tj||||_t ||_tj||||_tj||||_tj||||_ tj|_ t||_ yrZ)r0r1r2r^rurpost_projection_normr_r`raGELUact1r rc)r8r~rrbrXr:s r;r1zGlm4vVisionPatchMerger.__init__~s IIc3T2 $-cN!3 $?yyk= ;$?GGI Z( r<rhc|j|}|j|j|}|j|j |j ||j |zSrf)rurrrarcr_r`rgs r;rJzGlm4vVisionPatchMerger.forwardsYyy. yy!:!:1D!--"$,,t/A/A4>>"R#, r<r.c|jj}|jd}|j}t |t r&t j||t j}|jd} t| dz} |j| | |jdddjdj|t j} |jd} t j| |j} | jd|j!djdk\j#d}||dfjt j}||dfjt j}|dz|z dzdz }|dz|z dzdz }t j$||fd jdjd}t'j(| ||j*d d }|j-dj-djdd}|j|j.j|j}||z}|S) a Forward pass with integrated position encoding adaptation using 2D interpolation. Args: embeddings: Input embeddings tensor lengths (torch.Tensor): Sequence lengths for each image in the batch. image_shapes (torch.Tensor): Tensor of shape [batch_size, 3] representing the image shapes (t, h, w). h_coords (torch.Tensor): Tensor of shape [total_seq] representing the h coordinate for each patch. w_coords (torch.Tensor): Tensor of shape [total_seq] representing the w coordinate for each patch. Returns: torch.Tensor: Embeddings with adapted position encoding added. r&devicerBrg?r?rrwr@r~Fborder)mode align_corners padding_mode)rr6rMr isinstancelistr4tensorlongrrxpermuterrCrDrcumsumsumstackF grid_samplersqueezerB)r8 embeddingslengths image_shapesh_coordsw_coordspos_embed_weightr9r orig_size_sq orig_size pos_embed_2d num_tokenstoken_positionsseq_idstarget_htarget_wnorm_wnorm_hgridinterpolated_embed_fp32adapted_pos_embed_fp32adapted_pos_embeds r;rJzGlm4vVisionEmbeddings.forwards= 2299&,,Q/ !(( gt $ll76LG(--a0  c)*  ! !)Y D WQ1  Yq\ RvU]]R 3  %%a( ,,z*:K:KL",,Q/7>>!3D3N3Nq3QQVVWXY +..U]].C +..U]].Cc>X-2Q6c>X-2Q6{{FF+4>>qAKKAN#$-- $T%=%=Uai#  "9!@!@!C!K!KB!O!W!WXY[\!]2556F6L6LMPPQ[QbQbc "33 r<rzrUs@r;rrs# -0 -:PUP\P\:r<rc|dd|jddzf}|d|jddzdf}tj| |fdS)*Rotates half the hidden dims of the input..Nr@r?r)rMr4catxx1x2s r; rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r<qkcossinr.c|j}|j}|j|j}}|jdj|jdj}}||zt||zz}||zt||zz}|j |}|j |}||fS)N)rBrRrrrC)rrrr orig_q_dtype orig_k_dtypeq_embedk_embeds r;apply_rotary_pos_emb_visionrs77L77L 779aggiqA}}R &&(#--*;*A*A*CC3w;q>C/0G3w;q>C/0Gjj&Gjj&G G r<r=n_repc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r&N)rMexpandreshape)r=rbatchnum_key_value_headsslenhead_dims r; repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr<modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||j dd|z} || |z} t j j| dtjj|j} t j j| ||j} tj| | } | j ddj} | | fS)Nr?rr@)r~rB)ptrainingr&)rnum_key_value_groupsr4matmul transposer2 functionalsoftmaxrDrCrBrr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r;eager_attention_forwardrs3 ; ;}}}}t'j.|d } | j|dj1} |j3| } | Scc}wcc}}}w) Nrrr@r&r?F)rrr cu_seq_lens_q cu_seq_lens_k max_length_q max_length_krr)rrrr)rMrrrrunbindrrrr get_interfacerd_attn_implementationrr maxrrrr4splittolistziprrru)r8r=rrr seq_length query_statesrrrrattention_interface max_seqlenr_rrsplitsrrv attn_outputss r;rJzGlm4vVisionAttention.forward s#((+ HH] # + +J4>>2 N V VWXZ[]^`a b i ijk l / j,'S#>|ZY\^a#b j#--a3==a@ ))!Q/99!< #--a3==a@ (?(M(M KK , ,.E)  ( 4$QR.:cr?:??AJ0  $ #'==d6L6L((''NK"!nz#26GLXZdfrKsAG FGNN$4!<F #F|  Aq!$  $( LL'+}}C$:P:P#    L  ))La8K!))*b9DDF ii , - s -I?AIrf) rOrPrQr)r1rr4rSrLrJrTrUs@r;rrs 0 T %w7 IM A||ALLA#5<<#=>E A  A8Ar<rc eZdZd fd Zedde d dejdejd eejejfdzdejfd Z xZ S) Glm4vVisionBlockr.Nct|t|j|j|_t|j|j|_t||_t|d|_ y)Nr-Fr[) r0r1r,r9 rms_norm_epsnorm1norm2rattnrWmlprs r;r1zGlm4vVisionBlock.__init__fs\ !&"4"4&:M:MN !&"4"4&:M:MN (0  e4r<rrrr=rrc ||j|j|f||d|z}||j|j|z}|S)z cu_seqlens (`torch.Tensor`): Cumulative sequence lengths used for packed variable-length attention in Flash Attention kernels. rr)r%r#r&r$)r8r=rrrs r;rJzGlm4vVisionBlock.forwardms`&  JJ} %) ! 3)  )  &M1J(KK r<r.Nrf) rOrPrQr1rrr4rSrLrJrTrUs@r;rres|5%w7 IM ||LL#5<<#=>E   8r<rc eZdZUejed<d deffd Ze ddedzde dde dzde d e ffd Z ejed Zd ZxZS)Glm4vTextRotaryEmbeddingr}Nrdct||j|_|j|_||_|j j d|_|j}|jdk7rt|j}||j |\}|_ |jd|d|jd|jd|j jdgd|_y) N rope_typedefaultr}Froriginal_inv_freq mrope_section) r2)r0r1max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrdrope_parametersr-compute_default_rope_parametersrattention_scalingrclonegetr0)r8rdr rope_init_fnr}r:s r;r1z!Glm4vTextRotaryEmbedding.__init__s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L($( ZeD 0(..2BuU#3377Ur<rz torch.deviceseq_lenr.z torch.Tensorcn|jd}|jjdd}t|ddxs|j|jz}t ||z}d}d|t jd|dt jj|t j |z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorrrNrr?rwr) r6r:getattrr9num_attention_headsrr4rint64rCrR) rdrr<baser?rr~attention_factorr}s r;r7z8Glm4vTextRotaryEmbedding.compute_default_rope_parameterss&%%l3 & 6 6 : :;RTW X6:t4h8J8JfNhNh8h(223 U\\!S!5;;?BB&X]XcXcBdgjj k )))r<c^|jdddddfjjd|jddd}|dddddddfj}t |j j tr/|j j dk7r|j j nd}t|d5|j|jzjdd}|j||j}tj||fd }|j|jz}|j!|jz} dddj#|j$  j#|j$ fS#1swYAxYw) Nrr&r@mpscpuF) device_typeenabledr?rrw)r}rRrrMrrtyperr!r apply_mroper0r4rrr8rrCrB) r8rrinv_freq_expandedposition_ids_expandedrHfreqsembrrs r;rJz Glm4vTextRotaryEmbedding.forwards` !MM$a*=>DDFMMaQ]QcQcdeQfhjlmn ,Q4] ; A A C'1!((--'E!((--[`J`ahhmmfk  U C 5&,,.1F1L1L1NNYYZ[]^_E$$UD,>,>?E))UEN3C'')d444C'')d444C  5vvAGGv$cff177f&;;; 5 5s B!F##F,c |}|j|d}tjt|Dcgc] \}}||dzc}}d}|Scc}}w)Nr@rr)rr4r enumerate)r8rNr0sectionchunksichunkresults r;rKz$Glm4vTextRotaryEmbedding.apply_mropesQW"-69JKXQE!a%LKQST LsA rfNNN)rOrPrQr4rSrr(r1 staticmethodrrrLrRr7no_gradrrJrKrTrUs@r;r+r+sllVV")-+/"*$&*(*t* ~u$ % **>U]]_<< r<r+c||ddddf}|ddddf}tj| |fdjdS) r.rNr?r&r@rr)r4rrrs r;rotate_half_llmr[sJ 319B 319B ;;Ryb ) 1 1" 55r<c|j|}|j|}|dd|jddzfjdd}|dd|jddzfjdd}|jd}|dd|f|d|df}}|dd|f|d|df} }||zt||zz} ||zt||zz} t j | |gd} t j | | gd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .Nr@r?r)rrMrepeat_interleaver[r4r) rrrr unsqueeze_dim rotary_dimq_rotq_passk_rotk_passrrs r;apply_rotary_pos_embrdsD$ -- &C -- &C c'SYYr]a''' ( : :1" : EC c'SYYr]a''' ( : :1" : EC2Jc;J;&'3 +;)<6Ec;J;&'3 +;)<6Es{u5;@P@PW\] r<r=rrpast_key_valuesrr.c F|j\}}}|j|} |j|} |j|} | j ||d|j j dd} | j ||d|j j dd} | j ||d|j j dd} |\} } t| | | | \} } | |j| | |j\} } tj|jjt}||| | | |f|jsdn |j |j"d|\}}|j%||dj'}|j)|}||fS)Nr@r&r?r )rr)sizerirjrkrxrrrdupdatergrrrdrrrrrrrrl)r8r=rrrnrbszq_lenrrrrrrrrrs r;rJzGlm4vTextAttention.forwards&**, UA{{=1 [[/ {{=1 #((eRGQQRSUVW __S%T]]CMMaQRS #((eRGQQRSUVW &S#7 jRUWZ#[ j  &'6'='=j,X\XfXf'g $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \"))#ub9DDF kk+. L((r<rfrW)rOrPrQ__doc__r(rr1r4rSrLr rrrJrTrUs@r;rfrfs ^^3:^.IM.2(, ))||))#5<<#=>E)) t+ ))  )) -. )) u||U\\D0% 2E2LL M))r<rfcVeZdZfdZdej dej fdZxZS) Glm4vTextMLPc*t|||_tj|j d|j zd|_tj|j |j d|_t|j|_ y)Nr?Fr[) r0r1rdr2r^r9r] gate_up_projrar rb activation_fnrs r;r1zGlm4vTextMLP.__init__Gsp  IIf&8&8!f>V>V:V]bc6#;#;V=O=OV[\#F$5$56r<r=r.c|j|}|jdd\}}||j|z}|j|S)Nr?r@r)rxrUryra)r8r= up_statesgates r;rJzGlm4vTextMLP.forwardOsL%%m4 #//!/4i 2 24 88 ~~i((r<)rOrPrQr1r4 FloatTensorrJrTrUs@r;rvrvFs'7)U%6%6)5;L;L)r<rvcDeZdZdedeffd Ze d dejde ejejfdzdejdzdejdzd e dzd e dzd e eje ejejfdzffd ZxZS)Glm4vTextDecoderLayerrdrgct||j|_t|||_t ||_t|j|j|_ t|j|j|_ t|j|j|_ t|j|j|_ y)Nr!) r0r1r9rf self_attnrvr&r,r"input_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernormrms r;r1zGlm4vTextDecoderLayer.__init__Ys !--+FI>'+F,>,>FDWDWX(4V5G5GVM`M`(a%(4V5G5GVM`M`(a%".v/A/AvGZGZ"[r<Nr=rrrrn use_cacher.c |}|j|}|jd||||||d|\}} |j|}||z}|}|j|}|j |}|j |}||z}|S)N)r=rrrrnr)rrrrr&r) r8r=rrrrnrrresidualrs r;rJzGlm4vTextDecoderLayer.forwardcs!,,]; *4>> ' 3)%+    q55mD  =0 !55mD / // >  =0 r<)NNNNF)rOrPrQr(rr1rr4rSrL LongTensorr rjr}rJrTrUs@r;rrXs\\3\IM.204(,!&#||##5<<#=>E# t+ # &&- #  #$;# u  %(9(95;L;L(L"MPT"TT U##r<rc:eZdZUdZdZej dzed<y)Glm4vModelOutputWithPast rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. The attribute is deprecated and will be removed in v5.20, use `model.base_model.rope_deltas` instead. N rope_deltasrOrPrQrtrr4rrrr<r;rr ,0K!!D(/r<rcTeZdZUeed<dZdZdZddgZdgZ dZ dZ dZ dZ fdZxZS) Glm4vPreTrainedModelrdmodel)imagevideotextTrrrnc"t||t|trod|jt j d|jdt j|jz zz }tj|j|yy)Nrrr?rw) r0 _init_weightsrr|rr4rr~rRinitcopy_r})r8rr}r:s r;rz"Glm4vPreTrainedModel._init_weightssk f% f8 9fllu||Avzz1TYT_T_/`cicmcm/mnoH JJv 1 :r<)rOrPrQr'rbase_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_attention_backendrrTrUs@r;rrsQ 1&*#02DE#4"5N!"&22r<rc eZdZUeed<dZdgZeedZ d fd Z dZ e e edej d ej d eedeezfd ZxZS) Glm4vVisionModelrd)rrrr= attentionsr.cFt|||j|_|j|_t ||_t ||_|j|jz}t|dz|_ tjt|jDcgc] }t!|c}|_t%|j&|j(|j*|_t/|j|j0|_tj4|j|j&|j|j|_t/|j|j0|_d|_|j=ycc}w)Nr?)r~rrbr!)rr out_channelsrnroF)r0r1spatial_merge_sizerprrrl patch_embedr9rr|rr2 ModuleListrangedepthrblocksrr\r]rbmergerr,r"post_conv_layernormConv2d downsamplepost_layernormgradient_checkpointing post_init)r8rdrrr:s r;r1zGlm4vVisionModel.__init__sA  "(";"; ++/708%%)9)998QGmmuV\\GZ$[!%5f%=$[\ ,&&F4L4LY_YjYj  $00B0BH[H[#\ ))**//11,,   +6+=+=6CVCVW&+# %\s%Fctjd|jjdtdt ||j }|j|}||fS)N`z.rot_pos_emb` is deprecated and will be removed in v5.11. Use `get_vision_position_ids` from `transformers.vision_utils` and apply the rotary embedding module.r?) stacklevel)warningswarnr:rO FutureWarningr%rr)r8grid_thwrrs r; rot_pos_embzGlm4vVisionModel.rot_pos_embsa ''()H I  /x9P9PQ ,,\:|++r<r=rrc xt||j|}t||}|j|}|j |}|j |}t j||fd}|j|jf}|dd|ddz } |j|| ||dddfj|j|dddfj|j}|jD]} | |f||d|}|j|}|jd|j|j|j d}|j#dddd }|j%|jd|j&j(}|j+|} t-|| S) a\ hidden_states (`torch.Tensor` of shape `(seq_len, hidden_size)`): The final hidden states of the model. grid_thw (`torch.Tensor` of shape `(num_images_or_videos, 3)`): The temporal, height and width of feature shape of each image in LLM. Returns: `torch.Tensor`: hidden_states. )rr@rr&Nrr(rr?)last_hidden_state pooler_output)r%rr$rrrr4rrrrrCrrrrxrMrrrdr\rr) r8r=rrrr rotary_embrOrseqlensblkmerged_hidden_statess r;rJzGlm4vVisionModel.forwards/x9P9PY_` *8FC ((7 00? ((6 iiZ0b9"wwy#'')4QR.:cr?2    A  ! !-"6"6 7 A  ! !-"6"6 7  ;; C%$7 M ++M: %** '')@)@-BUBUVXBY &--aAq9  6;;B @[@[\ #{{=9)+.  r<r))rOrPrQr)rrrrr_can_record_outputsr1rr"r#rr4rSrrrLrrJrTrUs@r;rrs )+,)* 8, 3 "\\3 5:\\3 MSTfMg3 + +3  3 r<rceZdZUeed<dZeedZdeffd Z e e e dde jdzde jdzde jdzd edzd e j"dzd edzd eed eezfdZxZS)Glm4vTextModelrd)rrc  t|||j|_|j|_t j |j|j|j|_t jt|jDcgc]}t||c}|_ t|j|j|_t#||_d|_|j)ycc}w)Nr!rdF)r0r1 pad_token_id padding_idx vocab_sizer2rr9 embed_tokensrrnum_hidden_layersrlayersr,r"normr+rrrrms r;r1zGlm4vTextModel.__init__s  !.. ++LL):):FO   --i8M  CRC^==?de  << (;(;A(>}G[G[\_ooL',,Q26==aATATUVAWY[\L   ! #' 299!\=O=OPQ=RTVWL    !l&8&8&;q&@ ,Q '+L!% kk*,.-  )7;7 % "oom,oW![[ *M)*. /$7   M*M * -0 &++  r<)NNNNNN)rOrPrQr(rrrrfrr1rr"r#r4rrSr r}rjrrrLrrJrTrUs@r;rrs  .(  .2.204(,26!%J ##d*J  t+J &&- J  J ((4/ J $;J -.J  ( (J  J r<rcheZdZdZdZddgZfdZ d(dedeeeefe jzd ed ed ed e e jzdzf d Z d)de jde jde jdzde jdzde jdzdee je jff dZedee d*de j*de jdzdeedeezfdZedee d*de j*de jdzdeedeezfdZ d+de jde j*de j*dzde j*dzfd Z d,de jdzde jdzde jdzde jdzde jdzd!e jdzde jdzde jdzfd"Zed#d$%ee d-de jdzde jdzd&e jdzd!edzde j*dzde jdzde j*dzde jdzde jdzde jdzdeedeezfd'Z xZ!S). Glm4vModelrFrrct||tj|j|_t j|j|_d|_ |jyrf) r0r1r _from_config vision_configvisualr text_configlanguage_modelrrrs r;r1zGlm4vModel.__init__sU  &33F4H4HI ,99&:L:LM r<Nstart_positionrtemp_merge_sizer time_intervalrc|dj|z|dj|z|dj|z} }}tj|||z} tj|||z} tj| ||z} tj| | | d\} }}tj| ||gdj dd }|dxx|z cc<|S) a Compute 3D positional indices for vision tokens derived from a single image or video input. The positions are generated from the input grid defined by temporal (T), height (H), and width (W) dimensions. Temporal and spatial dimensions can be downscaled according to the merge sizes used in the vision backbone. The resulting positions are offset by `start_position`. Args: start_position (`int`): Offset added to all computed positional indices. grid_thw (`Sequence[int]` or `torch.Tensor` of shape `(3,)`): The (T, H, W) grid representing the feature layout of the current image or video after patch embedding. temp_merge_size (`int`, *optional*): Factor by which the temporal dimension is reduced in the backbone. The temporal grid size is divided by this value. Defaults to 1. spatial_merge_size (`int`, *optional*): Factor by which the spatial dimensions (H and W) are reduced in the backbone. Both H and W are divided by this value. Defaults to 1. time_interval (`int`, *optional*): Spacing factor applied between consecutive temporal position indices.Defaults to 1. device (`str` or `torch.device`, *optional*): Device on which the resulting tensor is allocated. If `None`, uses the current default device. Returns: torch.LongTensor of shape (3, sequence_length): Positional indices for temporal, height, and width dimensions, flattened into sequence form and offset by `start_position`. rr&r?rij)indexingrrr@)itemr4rmeshgridrr)r8rrrrrr llm_grid_t llm_grid_h llm_grid_wposition_temporalposition_heightposition_widthT_gridH_gridW_gridvision_position_idss r;r%z"Glm4vModel.get_vision_position_idssL QK   / 1 QK   "4 4 QK   "4 4!+J "LLFCmS,,z&ANRj@>Q!&0A?Tbmq!r#kk666*BJRRSTVXYA.0""r<rmm_token_type_idsimage_grid_thwvideo_grid_thwrr.c "|(tj||dddfd}d|dddf<|jjj}g}tj d|j d|j d|j|j} | t|nd| t|ndd} t|D]\} } || } |,| || j} | || j} g}tjt| jdD]7\}}t|}|dd}|d ddz}|j!|||f9d}g}|D]\}}}|dk(r^||z }|j!tj"||j j%dd j'dd |z||z }jt)| |}|j+||d||j }|j!||t-|d|d |zz }tj.|dj1dd }|5|j3| j| dd| || jf<n"|j3| j| dd| f<|j!|j-dzt5| z tj6||j j9d}| |fS) a Difference from Qwen2VL/Qwen2.5VL's get_rope_index: - GLM4V uses timestamps to separate each video frame, so the video_grid_thw should also be split too. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. mm_token_type_ids (`torch.IntTensor` of shape `(batch_size, sequence_length)`): Token type ids matching each modality to a different value in the input sequence, i.e. text (0), image (1), video (2). image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Returns: position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`) mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`) Nrrr&rrBr)r&r?c |dS)Nr&r)rs r;z+Glm4vModel.get_rope_index..s `abc`dr<r@rr?)r4r]rdrrzerosrMrBriterrQrj itertoolsgroupbyrrappendrrxrnextr%rrrrClenrr)r8rrrrrrrmrope_position_deltasr grid_iters batch_idxcurrent_input_idsinput_token_typeinput_type_grouprgroup start_index end_index current_posllm_pos_ids_list modality_type start_idxend_idxtext_lenrr llm_positionss r;get_rope_indexzGlm4vModel.get_rope_indexsOF  %"44^^TUWXTXEY_`aN#$N1a4 ![[66II "{{ OOA  OOA //##  (6'AtN#t'5'AtN#t -6i,@$ [ (I(0; )$5nY6O6T6T6V$W!#3N94M4R4R4T#U ! '// :J:Q:Q:S0TVde G UU #Ahqk !"IaL1,  ''k9(EF  G K! 5E W1 y' A%&2H$++ Xi6F6FGLLQPRSZZ[\^`adoo 8+K $J}$=>H*.*F*F#Xq2DYM]M]+G+'%++,?@3x{HQK#@DV#VVK W "II&6A>FFq"MM)O\O_O_`l`s`sOt Q >)+D+I+I+KKL-:-=-=l>Q>Q-R Q \* ! ( ():):)#?#?M '3$r<r pixel_valuesc :|j|jj}|j|fd|i|}|jd|jjdzzj }t j|j|}||_|S)T pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. rr@r?) rJrrBr"rrr4rr)r8r,rrr(r) image_embedss r;get_image_featureszGlm4vModel.get_image_features=s$(():):; $\UNUfU%**2.$++2P2PRS2SS[[] {{>#?#?M '3$r<rimage_featuresvideo_featuresc|||jtj|jjtj |j k(}|jd}||jtj|jjtj |j k(}|jd}n2||jjk(}||jjk(}|j}|jdj|j }|@t||jdz|jk(d|d|jd|j}|jdj|j }|@t||jdz|jk(d|d|jd||fS)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. rr@z6Image features and image tokens do not match, tokens: z , features: rz6Video features and video tokens do not match, tokens: )get_input_embeddingsr4rrdimage_token_idrrallvideo_token_idrrrCrrMnumel) r8rrr1r2special_image_maskspecial_video_maskn_image_tokensn_video_tokenss r;get_placeholder_maskzGlm4vModel.get_placeholder_maskTs  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; "+dkk.H.H!H !*dkk.H.H!H +//1/99"=@@AUAUV  % "!4!4R!88NTXY>Y(,(;(;---"3 )<) %L+ +D &   )/E/IYM^(5(;(; %J A)-224;;B?!C +77!8KQO +00JCJJ1aQRSVVWdWkWkl $||,BDZ]gDgh +00Ar:AA!ZQSTWWXeXlXlm $$66zTEUEUE[E[\]E^7^de6fE'%((-:N:N(*OOL Lr<rrrrc |du|duz r td||j|}||j||fddi| j} t j | dj |j|j} |j||| \} }|j| | }||j|| fddi| j}t j |dj |j|j}|j|||\}}|j||}||j||| |||| }|jd d||||d | }td i|d |jiS) aU image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. Nrr Trr)r1)r2)rrrrrrnr)rrrrnrrr)rr4r0rr4rrCrrBr=masked_scatterr+rFrrr)r8rrrrnrr,rrrrrr/ image_maskrr* video_maskoutputss r;rJzGlm4vModel.forwards.  -t"; <YZ Z  7D557 BM  #2422n:>BHm !99\q9<<]=Q=QS`SfSfgL 55i_k5lMJ)88\RM  *2422#^AEIOm !99\q9<<]=Q=QS`SfSfgL 55i_k5lMAz)88\RM  77#--+- /"38L&$%% %)+'    (  ((  r<)r&r&r&NrWrf)NN)NNNNN) NNNNNNNNNN)"rOrPrQraccepts_loss_kwargsrr1rrr4rSrrr%r IntTensorrLrrrrr}rrrr+r0r=rFrr rrJrTrUs@r;rr|s>02DE !"#,02#2#sC}% 42# 2#  2#  2#ell"T)2#p3726.2 [3##[3!??[3((4/ [3 ((4/ [3  t+ [3 u||U\\) *[3z 137"..((4/+,  + + 2: 137''((4/+,  + + 204837 (6##(6(((6))D0 (6 ))D0 (6\/3.2.2/348/<<$&/||d*/ t+ /  t+ /  t+ /,/!??T1/  /b]G4.2.204(,26,08<262648A ##d*A  t+A &&- A  A ((4/ A llT)A #..5A ((4/A ((4/A !??T1A +,A  ) )A 5A r<rc:eZdZUdZdZej dzed<y)Glm4vCausalLMOutputWithPastrNrrrr<r;rOrOrr<rOc!eZdZddiZdZfdZe d dejdejdzde e d e e zfd Ze d d ejd ejdzde e d e e zfd Zeddee d!dejdzdej$dzdejdzdedzdejdzdejdzd ej$dzdejdzd ejdzdejdzdej(dzdeej$zde e d e ezfdZ d"fd ZfdZ d dejdzdej$dzd e ej$ej$ffdZ d#dededejdzd e ejeeefffdZxZ S)$Glm4vForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightFct||t||_t j |j j|j jd|_ |jy)NFr[) r0r1rrr2r^rr9rlm_headrrs r;r1z&Glm4vForConditionalGeneration.__init__sS  ' yy!3!3!?!?ASASA^A^ejk  r<Nrrrr.c >|jj||fi|S)r)rr+)r8rrrs r;r+z0Glm4vForConditionalGeneration.get_video_featuress$-tzz,,-@.[TZ[[r<r,rc >|jj||fi|S)r.)rr0)r8r,rrs r;r0z0Glm4vForConditionalGeneration.get_image_featuress"-tzz,,\>TVTTr<rrrrrrrnrlabelsrlogits_to_keepc  |jd|||| | | ||||d | }|d}t| tr t| dn| }|j |dd|ddf}d}|2|j |||j jj}t|||j|j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, Glm4vForConditionalGeneration >>> model = Glm4vForConditionalGeneration.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> processor = AutoProcessor.from_pretrained("zai-org/GLM-4.1V-9B-Thinking") >>> messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}, {"type": "text", "text": "What is shown in this image?"}, ], }, ] >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) >>> inputs = processor(text=[text], images=[image], vision_infos=[vision_infos]) >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "The image shows a street scene with a red stop sign in the foreground. In the background, there is a large red gate with Chinese characters ..." ```) rr,rrrrrrrnrrN)logitsrVr)lossrYrnr=rrr)rrrslicerS loss_functionrdrrrOrnr=rr)r8rrrrnrrVr,rrrrrWrrKr= slice_indicesrYrZs r;rJz%Glm4vForConditionalGeneration.forward,sz$**  % 3))/%)+'      9C>SV8W~ot4]k mA}a,?@A  %%VFt{{OfOfOqOq%rD*#33!//))++   r<c  Zt||f||||||| | || d | } | s |r d| d<d| d<| S)N) rnrrrr,rrrris_first_iterationr,r)r0prepare_inputs_for_generation)r8rrnrrrrr,rrrr_r model_inputsr:s r;r`z;Glm4vForConditionalGeneration.prepare_inputs_for_generationsf"w<   +)'%% 3))1    "i+/L (26L. /r<ct |||}d}|jdx}|j}|dk7r4|jj |d|jj z}|Sd|vr|dj ddkDr|d}t|j dk(xr,|jtjtjfv}|r|jd}|jd|jd [|jD cic]\}} |dk7s || }}} |jj|fi|\} } | |j_no|jdjd d d } tj |j ddtj|j" |j_|d}tj$|| gd }|Scc} }w)Nrrnrrr&r?rrrrr@rr)r0$_prepare_position_ids_for_generationr:rrrrMr rBr4rritemsrrrrrr) r8 inputs_tensor model_kwargstext_positions past_lengthcacher is_input_idsrrvision_positionsrr:s r;rczBGlm4vForConditionalGeneration._prepare_position_ids_for_generationsEmUab !%%&78 8E E..0K !  6 6 B))4tzz7M7MML  , &< +D+J+J1+MPQ+Q(5M=../14g9L9LQVQZQZ\a\f\fPg9g   !45A!!"23?#5#5#7,:J:J:L#LRST "Q % 6),,,3 %))a)0 \))r< expand_sizeis_encoder_decoderc dk(rfSgdfd}fd}|jd||r*jd td|dd<fS) Nr&)r,rrrsecond_per_grid_tsc4 jdd} jdd} j jdd\}}d}|D]9}|dk(rctj|t |}|Dcgc]'}tj |dj )} }|||| ||<l|dk(rt |} |||| ||<|d k(rctj|t |}|Dcgc]'}tj |dj )} }|||| ||<|dk(rt |} |||| ||<|d k(s |||t | ||<<|Scc}wcc}w) Nrrr)rctj||}|gdg|jdz zz}tj|Dcgc]}|j|c}d}|Scc}w)Nr&rr)r4rr~rr@)rr repeat_timessamples repeat_argssamplerVs r;_repeat_interleave_sampleszGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual.._repeat_interleave_samples&sa++a1+nsaeegk/BB g#VFMFMM;$?#V\]^ $WsA&r,r&r)rrrr|)r:rxr4rrr"r)dict_to_expandrr image_nums video_numsrrrrrryrrfr8s r;"_expand_dict_for_generation_visualzgGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visuals)--.>EN)--.>EN%)%H%H)9)9/4)P&I& "J   & .(#kk.$z:JKGMTU6uzz&a8<<>UGU*D&s+W;+N3',,":.G*D&s+W;+N3'11#kk.$z:JKGMTU6uzz&a8<<>UGU*D&s+W;+N3',,":.G*D&s+W;+N3'00*D&s+T*5ET_+N3'7 <" !3VVs =,F,Fc|D]u}|dk(r,||jdk(r||jd||<4||:t||tjsX|vs]||jd||<w|S)Nrrr&rr)rr]rr4rS)rrry visual_keyss r;_expand_dict_for_generationz`Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generationLs% d.(^C-@-E-E-J*8*=*O*OP[ab*O*cN3'"3'3">##6 E;.*8*=*O*OP[ab*O*cN3' d" !r<rrencoder_outputszMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)r]r:r)r8ryrzrrfrrrs`` `` @r;_expand_inputs_for_generationz;Glm4vForConditionalGeneration._expand_inputs_for_generations ! l* *w + "Z ":,G  !33KQ3GI2<@  12: !pqq.I,WhJi.jL* +,&&r<rf) NNNNNNNNNNNr) NNNNTNNNNF)r&FN)!rOrPrQ_tied_weights_keysrLr1rr4r}rrrrLrr+r0rrrSr rMrrOrJr`rcrxrjdictrrrrTrUs@r;rQrQs"*,VW37 \".. \((4/ \+, \ + + \ \37 U'' U((4/ U+, U + + U U]G4.2.204(,26*.,08<262648-.Y ##d*Y  t+Y &&- Y  Y ((4/ Y   4'Y llT)Y #..5Y ((4/Y ((4/Y !??T1Y ell*Y +,Y  , ,Y 5Y |  $L$R.26*##d*6*||d*6* u||U\\) * 6*t#(-1 V'V'!V'##d* V' uc3h/ 0 V'r<rQ)rQrrrr)r )r&)brrcollections.abcr dataclassesrtypingrrr4torch.nnr2torch.nn.functionalrrrr r activationsr cache_utilsr r generationr integrationsr masking_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrutils.deprecationr utils.genericrr r!r"utils.output_capturingr# vision_utilsr$r%configuration_glm4vr'r(r)Moduler,rWrlr|rrrrSrLrrrrRrrrr+r[rdrfrvrrrrrrrOrQ__all__rr<r;rs($! &!.)7/B9kkKF&aa0 6JPPY'J299J(J( fBII fBII( G GfRYYf"GBIIGT(  ||   +0<< >Cll  5<< %&  UU\\ U# U%,, U&% II% <<% % << % LL4' %  %%'(%2P299Pf1>JryyJZ6%PC)C)L)299)$/6/d 060 02?22(e +e Pe )e e Pv %v v r  0"80 0b'$8/b'J xr<