^j<ddlmZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z mZddlmZd d lmZej&eZd ZGd d eZGddeZdZddZeGdde ZGdde ZGdde ZGdde ZgdZy))OptionalN)logging)no_inherit_decorator)LlamaAttentionLlamaForCausalLMLlamaForSequenceClassificationLlamaForTokenClassificationLlamaRotaryEmbedding)Phi3MLP) GlmConfigzTHUDM/glm-4-9bc eZdZy)GlmMLPN__name__ __module__ __qualname__n/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/glm/modular_glm.pyrr&rrc ReZdZe d dedzdeddedzdedeffdZ y) GlmRotaryEmbeddingNconfigdevicez torch.deviceseq_lenreturnz torch.Tensorcn|jd}|jjdd}t|ddxs|j|jz}t ||z}d}d|t jd|dt jj|t j |z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorg?head_dimNrr)dtype)rr$) rope_parametersgetgetattr hidden_sizenum_attention_headsinttorcharangeint64tofloat) rrrbaser"r#dimattention_factorinv_freqs rcompute_default_rope_parametersz2GlmRotaryEmbedding.compute_default_rope_parameters+s&%%l3 & 6 6 : :;RTW X6:t4h8J8JfNhNh8h(223 U\\!S!5;;?BB&X]XcXcBdgjj k )))r)NNN) rrr staticmethodrrr*tupler/r4rrrrr*sY#'+/"*D *(*t* ~u$ % **rrc||ddddf}|ddddf}tj| |fdjdS) z*Rotates half the hidden dims of the input..rNrrr1)r+stackflatten)xx1x2s r rotate_halfr@LsJ 319B 319B ;;Ryb ) 1 1" 55rc|j|}|j|}|dd|jddzfjdd}|dd|jddzfjdd}|jd}|dd|f|d|df}}|dd|f|d|df} }||zt||zz} ||zt||zz} t j | |gd} t j | | gd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .Nr8rr9) unsqueezeshaperepeat_interleaver@r+cat) qkcossin unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embeds rapply_rotary_pos_embrRSsD$ -- &C -- &C c'SYYr]a''' ( : :1" : EC c'SYYr]a''' ( : :1" : EC2Jc;J;&'3 +;)<6Ec;J;&'3 +;)<6Es{{51C78Gs{{51C78Gii&)r2Gii&)r2G G rc0eZdZddededzffd ZxZS) GlmAttentionNr layer_idxct|||tj|j|j z|j d|_y)NF)bias)super__init__nnLinearr)r#r(o_proj)selfrrU __class__s rrYzGlmAttention.__init__}s< +ii : :T]] JFL^L^ejk r)N)rrrrr*rY __classcell__)r^s@rrTrT{s#lylS4ZllrrTc eZdZy)GlmForCausalLMNrrrrrararrrac eZdZy)GlmForSequenceClassificationNrrrrrcrcrrrcc eZdZy)GlmForTokenClassificationNrrrrrererrre)GlmPreTrainedModelGlmModelrarcre)r) typingrr+torch.nnrZutilsr utils.genericrllama.modeling_llamarr r r r phi3.modeling_phi3r configuration_glmr get_loggerrlogger_CHECKPOINT_FOR_DOCrrr@rRrTrarcre__all__rrrrss 1)(   H %& W *-*D6%Pl>ll  %  #A  ;  r