^j-c dZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z ddl mZdd lmZmZdd lmZdd lmZmZdd lmZdd lmZddlmZmZmZddlm Z ddl!m"Z"m#Z#ddl$m%Z%ejLe'Z(GddejRZ*GddejVZ,GddejVZ-GddeZ.eGddeZ/eGdde/Z0ed Gd!d"e/eZ1gd#Z2y)$zPyTorch XGLM model.N)nn)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)merge_with_config_defaults)OutputRecordercapture_outputs) XGLMConfigc `eZdZdZd dededededzffd Zdejffd Z xZ S) XGLMScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. num_embeddings embedding_dim padding_idx embed_scaleNc6t||||||_yN)super__init__r)selfrrrr __class__s q/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/xglm/modeling_xglm.pyr#z XGLMScaledWordEmbedding.__init__-s  D& input_idsc<t|||jzSr!)r"forwardr)r$r(r%s r&r*zXGLMScaledWordEmbedding.forward1swy)D,<,<<r@rAregister_buffer)r$rrr emb_weightss r&r<z.XGLMSinusoidalPositionalEmbedding.make_weights@s[(( T 4 #%..t||/A/A$,,J]J].^K Y Fr'c|dz}tjd|dz z }tjtj|tj j | z}tj|tj j jd|jdz}tjtj|tj|gdj|d}|dzdk(r-tj|tj|dgd}| d||ddf<|jtjS) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". r:i'r)r@rdimN)mathlogr2exparangeint64r1 unsqueezecatsincosviewzerosrEget_default_dtype)rrrhalf_dimembs r&rCz/XGLMSinusoidalPositionalEmbedding.get_embeddingHs!A%hhuoA.ii XU[[AGGISDPQll>=CCEOOPQRUXUbUbcdUeeii338a@EEnVXY 1  !))S%++na"@AqIC  ""#C Q vve--/00r' position_idspast_key_values_lengthc|j\}}||jz}d|z|z}||jjdkDr'|j||j|j |jj d|jdj|||jjdjS)Nr:rrK) sizer;r>r<rr index_selectrUshapedetach)r$rZr[bszseq_lenmax_poss r&r*z)XGLMSinusoidalPositionalEmbedding.forward]s#((* W#dkk1 g+ 66 T\\&&q) )   gt'9'94;K;K L||((L,=,=b,ABGGWVZVbVbVhVhikVlmttvvr'r!)Nr)r,r-r.r/r0r#r< staticmethodrCr2no_gradr3r*r4r5s@r&r7r75sNScS#SCRVJSG3GsGQTW[Q[G1c1#1CRVJ11(U]]_wELL4$7wX[wwr'r7c4eZdZdZ ddedededzdedzdedzdedzf fd Z dd ejd ejdzd e dzd ejdzde e de ejejdze ejdzff dZxZS) XGLMAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsdropout is_decoderbias layer_idxct|||_||_||_||z|_|j |z|jk7rt d|jd|d|j dz|_||_||_ tj||||_ tj||||_ tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rl)r"r#rhrirjhead_dim ValueErrorscalingrkrmrLineark_projv_projq_projout_proj)r$rhrirjrkrlrmr%s r&r#zXGLMAttention.__init__ls "" !Y. MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r' hidden_stateskey_value_statespast_key_valuesattention_maskkwargsreturnc  |du}|j\}}} |r|jdn|} |j||jz} d} |St |t rA|j j|j} |r |j} n|j} n|} |r|n|}|rK|I| rG j|jj}| j|jj}n|j|}|j|}|j!|| d|j"j%dd}|j!|| d|j"j%dd}|K j'|||j\}}|r)t |t rd|j |j<||j(zd|j"f}| j!|||j(|j"j%dd} | j*|} |j*|}|j*|}|jd} t-j.| |j%dd}|j||j(z|| fk7r/t1d||j(z|| fd|j||j|d|| fk7r#t1d |d|| fd|j|j!||j(|| |z}t-j2|t-j4t-j6|j8j:|j< }|j!||j(z|| }|j8t,j>k(rNt@jBjE|dt,jF jIt,j>}n!t@jBjE|d }|j!||j(|| }|j!||j(z|| }t@jBjK||jJ|jL }t-j.||}|j||j(z||j"fk7r7t1d||j(||j"fd|j|j!||j(||j"}|j%dd}|j+|||jN}|jQ|}||fS)z#Input shape: Batch x Time x ChannelNrFrKr:Tz$Attention weights should be of size z , but is z!Attention mask should be of size )rA)rJr@rIptrainingz `attn_output` should be of size ))r]r_rvrr isinstancer is_updatedgetrmcross_attention_cacheself_attention_cachelayerskeysvaluesrtrurUrp transposeupdaterireshaper2bmmrqmaxtensorfinfor@minrAfloat16r functionalsoftmaxfloat32rErjrrhrw)r$rxryrzr{r|is_cross_attentionratgt_len_src_len query_statesrcurr_past_key_valuescurrent_states key_states value_states proj_shape attn_weightsattn_weights_reshaped attn_probs attn_outputs r&r*zXGLMAttention.forwards.T9',,.Wa/A"((+w{{=1DLL@   &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML^4J;;~6L#gr4==ISSTUWXYJ',,S'2t}}MWWXY[\]L*+?+F+FzS_aeaoao+p( L%*_FY*ZAEO..t~~>DNN*B > #((gt~~t}}U__`acde +|++Z8 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL 99ell5;;|7I7I+J+N+NWcWjWjkL(,,S4>>-A7GTL    .==002U]]0[^^_d_l_lmL==0020FL !- 1 1#t~~wPW X,11#2FQXY ]]**<4<$>?"(";";  % % -.. 0000# !D ,.<<+GD ($&LL$@!99T^^V^^<99V^^T^^< " T^^ <r'rxr{encoder_hidden_statesencoder_attention_maskrz use_cacher|r}c |}|j|}|j|f||d|\}} tjj ||j|j }||z}|h|}|j |}|j|f|||d|\}} tjj ||j|j }||z}|}|j|}|j|j|}tjj ||j|j }|j|}tjj ||j|j }||z}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states )rzr{r)ryr{rz) rrrrrjrrrrrrrr) r$rxr{rrrzrr|residualrs r&r*zXGLMDecoderLayer.forwards*!11-@ *4>>  +)    q  --mt||VZVcVc-d  =0  ! ,$H 88GM0t00 !65 /      M1MM11-4<)r$modulerGr%s r&rz!XGLMPreTrainedModel._init_weights\sg f% f? @ ..$$v}}4f6J6JFL^L^K JJv~~{ 3 Ar') r,r-r.r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrr4r5s@r&rrUs( &*#+,44r'rceZdZeeeddeedddZdeffd Ze e e dd e jdzd e jdzd e jdzd e jdzd e jdzdedzde jdzdedzdeedee jezfdZxZS) XGLMModelrr)index layer_namer)rx attentionscross_attentionsrc t|||j|_|j|_|j|_|j |_|jrtj|jnd}t|j|j|j ||_t|j |j|j|_t#j$t'|j(Dcgc]}t+||c}|_t#j.|j|_d|_|j5ycc}w)Nr+)r)rmF)r"r#rj layerdrop pad_token_idrmax_position_embeddingsmax_target_positionsscale_embeddingrLsqrtrr vocab_size embed_tokensr7embed_positionsr ModuleListrange num_layersrrr layer_normgradient_checkpointing post_init)r$rrir%s r&r#zXGLMModel.__init__ms   ~~ ))!..$*$B$B!393I3Idii/s 3   v~~t/?/?[  A  * * NN     mmTYZ`ZkZkTl$mq%5f%J$mn ,,v~~6&+#  %ns E&Nr(r{rZrrrz inputs_embedsrr|r}c  D|du|duz r td||j|}|rd|b||jjr4t t |jt |jnt |j}||j nd} t|j|||}|`tj| |jd| ztj| |jn |j}|jd}||t|j|||}||j|| j!|jz} t"j$j'| t)|j&|j* } t-|j.D]E\} } |j*r%tj0g}||j2kr7| | ||f|||d | } G|j5| } t7| | S) a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Nz:You must specify exactly one of input_ids or inputs_embeds)rr)rrr{rzrr?)rrr{rr)rrzr)last_hidden_staterz)rqrris_encoder_decoderr rget_seq_lengthr r2rOr_longrArQr rrErrrjr1r enumeraterrandrrr)r$r(r{rZrrrzrrr|r[rxidx decoder_layerdropout_probabilitys r&r*zXGLMModel.forwards08  -t"; <YZ Z   --i8M 0)4 8V8V$L $DlZ^ZeZeFfg!5  FUE`!?!?!Afg+;;')+     <<&##A&)??jj+4+@y''mFZFZ L (11!4L ! ,1G1S%>{{+5&; & "&(<(<\Ka(b(e(e  )   --muT\\?R]a]j]j-k "+DKK"8  C}}&+jjn#&7)%(> /# M " 6 8++  r')NNNNNNNN)r,r-r.rrrg_can_record_outputsrr#rrrr2r3rrrrrrr*r4r5s@r&rres3*$]! T*=n] z0 *..2,0596:(,-1!%] <<$&]  t+] llT) ] %||d2 ] !& t 3 ] ] ||d*] $;] +,]  u|| H H]  ] r'rz The XGLM Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introceZdZdZddiZfdZeee dde jdzde jdzde jdzd e jdzd e jdzd e dzd e jdzd e jdzde dzde e jzdeedee jezfdZxZS)XGLMForCausalLMrzlm_head.weightzmodel.embed_tokens.weightct||t||_t j |j |jd|_|jy)NFro) r"r#rrrrs hidden_sizerlm_headr)r$rr%s r&r#zXGLMForCausalLM.__init__sH  v& yy!3!3V5F5FUS  r'Nr(r{rZrrrzrlabelsrlogits_to_keepr|r}c |jd|||||||| d| } | j} t| tr t | dn| }|j | dd|ddf}d}|=|j |||jj|jj}t||| j| j| j| jS)ai encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. )r(r{rZrrrzrrN)rr)losslogitsrzrxrr)rrrr0slicer loss_functionrrrrrzrxrr)r$r(r{rZrrrzrrrrr|outputsrx slice_indicesrrs r&r*zXGLMForCausalLM.forwardsF>HTZZ > )%"7#9+' >  >  11 8B>SV8W~ot4]k mA}a,?@A  %%;;11![[55 &D1#33!//))$55   r') NNNNNNNNNr)r,r-r.r_tied_weights_keysr#rrrr2r3rrr0rrrrr*r4r5s@r&rrsI *,GH *..2,0596:(,-1&*!%-.A <<$&A  t+A llT) A %||d2 A !& t 3 A A ||d*A  t#A $;A ell*A +,A  u|| @ @A  A r'r)rrr)3r/rLr2rrr activationsr cache_utilsrrr generationr masking_utilsr r modeling_layersr modeling_outputsrrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrutils.output_capturingrrconfiguration_xglmr get_loggerr,logger EmbeddingrModuler7rgrrrr__all__rr'r&rs  &!CC)J9l-&@@7E*   H % =bll =1w 1whL2BIIL2^Z1Zz 4/ 4 4 #  D P )?P  P f Br'