^jFdZddlZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlmZmZmZddlmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/e'j`e1Z2GddejfZ4dejjde6dejjfdZ7 dGd/d0eZ?e%Gd1d2e Z@e%Gd3d4e@ZAGd5d6e@eZBe%d78Gd9d:e@ZCgd;ZDy)=zPyTorch Zamba model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)lazy_load_kernel)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)resolve_internal_import)capture_outputs) ZambaConfigcheZdZddeddffd Zdej dej fdZdZxZ S) ZambaRMSNormepsreturnNct|tjt j ||_||_y)z; ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer" __class__s s/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/zamba/modeling_zamba.pyr&zZambaRMSNorm.__init__2s1 ll5::k#:; # hidden_statesc"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NT)keepdim) dtypetor(float32powmeanrsqrtr+r*)r,r1 input_dtypevariances r/forwardzZambaRMSNorm.forward:sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r0c^t|jjd|jS)Nz, eps=)tupler*shaper+)r,s r/ extra_reprzZambaRMSNorm.extra_reprAs* ))*+6$2G2G1HIIr0)gư>) __name__ __module__ __qualname__floatr&r(Tensorr>rB __classcell__r.s@r/r!r!1s7$$$$;U\\;ell;Jr0r!r1n_repr#c|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rAexpandreshape)r1rJbatchnum_key_value_headsslenhead_dims r/ repeat_kvrRFso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr0modulequerykeyvalueattention_maskscalingdropoutc t||j}t||j} tj||j dd|z} || |z} t j j| dtjj|j} t j j| ||j} tj| | } | j ddj} | | fS)Nr3rr4)dimr6)ptrainingr)rRnum_key_value_groupsr(matmul transposer functionalsoftmaxr8r7r6rYr] contiguous) rSrTrUrVrWrXrYkwargs key_states value_states attn_weights attn_outputs r/eager_attention_forwardriRs3 ; ;$>&B\B\$\!'-'E'E$ )d2 !'!9!9ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii : :T]] JFL^L^ejk r0Nr1rWpast_key_valuesrdr#c |jdd}g|d|j}|j|j|j dd}|j |j|j dd} |j |j|j dd} ||j| | |\} } tj|jjt} | ||| | |f|jsdn |j|jd|\} } | j g|dj#} |j%| } | | fS)Nr4rr3)rYrX)rArQryviewr`rzr{updater get_interfacerl_attn_implementationrir]rwrXrMrcr|)r,r1rmrWr~rd input_shape hidden_shape query_statesrerfattention_interfacerhrgs r/r>zZambaAttention.forwardsk$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST  &'6'='=j,Xa'b $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r0N)rCrDrE__doc__rintr&r(rGr rrr@r>rHrIs@r/rkrkks l{lsl.)- #)||#)#) t+ #)  #) +, #) u||U\\D0% 2E2LL M#)r0rkc~eZdZdZdeffd Z d dejdedzfdZ d dedzfdZ d dedzfd Z xZ S) ZambaMambaMixeruE Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. rlc 8 t|||_||_|j|_|j |_|j|_|j|jz|_ |j|_ |j|_ |j|jz|_|j|_|j"|_t'j(|j|j|j |j|j|jdz |_|j,|_t0|j,|_|j4|_t'j8|j|jdz|j$|_t'j<t?j@|j|j|j dzz|j|_!t'j<t?j@|j|j|jdz dz|jdzz |_"t'j<t?j@|j|j|_#t?jHd|j dzt>jJdddf}|jM|jdjO}t'j<t?jP|jS|j|jd|_*t'j<t?jV|j|j|_,t'j8|j|j|j$|_-t]da/tat^d da1tat^d da2t]d a3titfd a5tatfdda6tatfdda7tqtjtltdtbtnfa9trsttjwd|jx||_=y)Nr) in_channels out_channelsrq kernel_sizegroupspaddingr3rpg?r6r4z causal-conv1dcausal_conv1d_updatecausal_conv1d_fnz mamba-ssmz8ops.triton.selective_state_update.selective_state_update) chained_pathselective_scan_fnmamba_inner_fnaqThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)>r%r&rlrmr- mamba_d_statessm_state_size mamba_d_convconv_kernel_size mamba_expandintermediate_size mamba_dt_ranktime_step_rank n_mamba_headsmamba_head_dimmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasrConv1dconv1dhidden_mamba_act activationr actuse_mamba_kernelsuse_fast_kernelsrxin_projr'r(zeros x_proj_weightdt_proj_weight dt_proj_biasaranger8rLrclogrMA_logr)Dout_projr causal_conv1dgetattrrr mamba_ssmrselective_state_updaterrallis_fast_path_availablelogger warning_once layer_types layer_type)r,rlrmAr.s r/r&zZambaMambaMixer.__init__s  "!--$22 & 3 3!'!4!4v7I7I!I$22#11"448J8JJ#33.. ii..//##--))))A-   !11&112 & 8 8yy!1!143I3IA3MTXTaTab  \\ KK""##d&9&9A&==##  !ll [[++T-@-@$BUBU VY\ \ !!3& '  LLT5G5GI\I\)]^ LLD//!35== I$PQ' R HHT++R 0 ; ; =\\%))A,"6"6t7I7I4K^K^`b"cd ejj););T=P=PQR $"8"8$:J:JQUQ^Q^_ )9 &}6LdS"=2DdK%[1 !8 $^" $I/BDI ,|j@|j@gd\} }}|jBdddf| j ddz}t)jD|jFjI }|jJ|jJjInd}t)jL|d|f|jN|jP }|rtS|jD]}tU|j|jjVddd|f||ddf||ddf||||dddf||dddf|jX|| |ddf||d j'd}t)jZ||fd}n5t)jL|d|j8|j@f|jN|jP }tS|jD]}t]||||||||j dd||j dd|jX|jI| |||d d \}}t)jZ||fdj}t)jZ||j'dfd}|||j_||j|ja|j dd}|S) Nrr3r4r[r.)rdevicer6T) dt_softplus)delta_softplusreturn_last_state)1rAhas_previous_statermrr`rchunksqueezercrMrrr*sizerlayers conv_statesrqr unsqueezer(rrrapadrupdate_conv_staterrrsplitrrrexprrFremptyrr6rangerrecurrent_statesrcatrupdate_recurrent_stater)r,r1rrW batch_sizeseq_len_use_precomputed_statesprojected_statesgate conv_weightsrssm_parameters time_stepBCdiscrete_time_steprtime_proj_bias scan_outputsn scan_outputs_ ssm_state ssm_state_contextualized_statess r/cuda_kernels_forwardz$ZambaMambaMixer.cuda_kernels_forwards"/!4!4 GQ  $ i)H)H)X i]dhi]i   << 6@@AF.33JAwOUUVW]^U_ t%--a0;;= ||A||J(:(:BHRRSTVWX{{))..t{{/A/A/F/Fq/I4;;K]K]KbKbcdKef !0%%b)##DNN3??B    M*33B7M)%))NajA| |j*| jdz df} |j)| |jd|j* df} |j7|j/| dd|f} || |dd| jd dfj;dz} nS|| |j;dz} |j7|j/| dd|f} || |j;dz} | jd|j|j"|jdd} |jBdddddddf| zjdd} tjD| |jF|j$|j$gd\}}}|jHdddf|jddz|jJddddddfz}t<j>jM|}tjN|jPjS }tjN|ddddddddf|dddddddddfz}|dddddddddf|dddddddddfjSz}|| dddddddddfjSz}g}tU|D]}|dddddd|ddfjdd| z|dddddd|ddfjddz} tjV| jddj9||dddd|ddfj;d}|jY|dddddddftjZ|d}|| |j\ddddddfzz}||j7| z}||j_| |j|ja|jddj|d|jdd}|S) Nrr3r4rrr.r)1rAr6rr`rrrrcrMrrrmrrcloner(rrrrrrsumrr*rrqrr7rrrarrrrrrsoftplusrrrFrr_appendstackrrr)r, input_statesrrWrrrr6rr1rr conv_staterrrrrr discrete_A discrete_BdeltaB_uri scan_outputrs r/ slow_forwardzZambaMambaMixer.slow_forwardtsM!-!3!3 GQ""<< 5??1E.33JAwOUUVW]^U_ t%--a0;;= ||A||J(:(:BHRRSTVWX  # (G(G(W$++DNN;LLQOUUWI T//1D1DdFYFYZ$++I  #..t~~>7a<);;M4>>Z$///11 !& *t{{7I7I!QPQ'7R*RXZ [ %%!T[[%5%55M $ 7 : :5 A K KB O !-$1N1}GZGZ[]G^F^F`C`4a4k4klm4n$nM]]..}t?T?TWdWjWjkmWn?npq>rs );;JWX[^b^s^s]s]uXuv $])CC'M)R S !-$1N1}GZGZ[]G^F^F`C`4a4k4klm4n$nM) -0H0H0K K  HHT[[%?XgX %NOM) -0H0H0K K &--b$2D2DdFYFY[bcmmnoqrs ,,Qa];mKVVWY[]^++ T00$2E2EtGZGZ[ac  1a#11!T':Y=P=PQSUW=XX\`\m\m tQ ]   ]]334FGYYtzz'') * *YYqD!T1!458J1aQRTUW[K[8\\] '1aD(89AaD!Q>N ? E E GG w 9A"1aAq=1;;AqAIMPXYZ\]_`bcefYfPgPqPqrsuvPwwI,,y':':1a'@'C'CE'JAaQRTUWXjMLcLcdfLghK    Aq!QJ 7 8 9kk,B7 !]TVVAtQ> J!%  ! !!Q ' / / B H R RSTVW X! %$r0c tttttt f}|j rC|r"d|jjjvr td|j|||S|j|||S)NcudazFast Mamba kernels are not available. Make sure to they are installed and that the mamba module is on a CUDA device. lease run 'pip install causal-conv1d>=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)rW) rrrrrrrrrtype ValueErrorrr)r,r1rrWrdrs r/r>zZambaMambaMixer.forwards!$ #%68HJ^`n o"   )V4;M;M;T;T;Y;Y-Y i ,,]LYg,h h   ^ \\r0)NN) rCrDrErrr&r(rGr rrr>rHrIs@r/rrsd O8{O8d^bc%"\\c%9>c%JR%ut|R%h ]54< ]r0rc$eZdZfdZdZxZS)ZambaMLPct|||_|j|_|j|_t j |j|jd|_t j |j|jd|_t j |j|jd|_ t|j|_ yNFrp) r%r&rlr-rrrx gate_projup_proj down_projr hidden_actact_fnr,rlr.s r/r&zZambaMLP.__init__s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ r0c|j|j|j||j|z}|Sr)r r rr)r,xr s r/r>zZambaMLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r0)rCrDrEr&r>rHrIs@r/rrs 0r0rc eZdZd dededzffd Z ddejdejdedejdzdedzd e dzd e e d e eje ejejfdzffd ZxZS)ZambaAttentionDecoderLayerNrlrmct|t|||_t ||_t |j|j|_ t |j|j|_ y)Nr") r%r&rk self_attnr feed_forwardr!rr rms_norm_epsinput_layernormr-pre_ff_layernormr}s r/r&z#ZambaAttentionDecoderLayer.__init__s_ ' :$V,+F,H,HfNaNab ,V-?-?VEXEX Yr0r1original_hidden_statesrWr~ use_cacherdr#c tj||gd}|j|}|jd|||||d|\}}|j |}|j |}|S)ac Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://huggingface.co/papers/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). r4r)r1rmrWr~r)r( concatenaterrrr) r,r1rrmrWr~rrdrs r/r>z"ZambaAttentionDecoderLayer.forwards2))=:P*QWYZ ,,]; )4>> ')+     q--m< ))-8 r0r)NNF)rCrDrErrr&r(rGr boolrrr@ FloatTensorr>rHrIs@r/rrsZ{ZsTzZ/3(,!&'||'!& ' '  t+ '  '$;'+,' u  %(9(95;L;L(L"MPT"TT U'r0rcreZdZdedeffd Z ddejdejdzdedzdejdzdejdzd edzd e dzd ejdzd ejdzd e e de eje ejejfdzffdZxZS)ZambaMambaDecoderLayerrlrmct|t|||_t |j |j |_||_y)N)rlrmr) r%r&rmambar!r-rrrmr}s r/r&zZambaMambaDecoderLayer.__init__s> $FiH +F,>,>FDWDWX"r0Nr1rrW causal_maskr~r position_idstransformer_hidden_statesrdr#c  t|} | || zn|}|j|}|jd|||d| }| |z}|S)aX Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )r1rrWr)rr") r,r1rrmrWr#r~rr$r%rdresiduals r/r>zZambaMambaDecoderLayer.forward#sn0! :S9^M5 5dq ,,]; "  '()   !=0 r0)NNNNNFNN)rCrDrErrr&r(rGr r LongTensorrrr@rr>rHrIs@r/r r s#{#s#7; $.2+/(,!&049=*||*!& t 3*: *  t+ * \\D( **$;*&&-*$)<<$#6*+,* u  %(9(95;L;L(L"MPT"TT U*r0r cJeZdZdedej deffd Z ddejdejdzde dzd ejdzd ejdzd e dzd e dzd e edeej eej ej fdzffdZxZS)ZambaHybridLayer shared_transflinearr"cLt|||_||_||_yr)r%r&r+r, mamba_decoder)r,r+r,r"r.s r/r&zZambaHybridLayer.__init__Qs% * "r0Nr1rrmrWr#r~rrdr#c |j|f|||||d|} |j| } |j|f| |||d|}|S)aF Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )rrmrWr~r)r%rWr~r)r+r,r.) r,r1rrmrWr#r~rrdr%s r/r>zZambaHybridLayer.forwardWs2%7D$6$6 % #9&+ % % !%)KK0I$J!***  &?)+    r0)NNNNNF)rCrDrErrrxr r&r(rGrr rrrr@rr>rHrIs@r/r*r*Ps#&@#"))#\r#7; $.2+/(,!&-||-!& t 3-: -  t+ - \\D( --$;-+,- u  %(9(95;L;L(L"MPT"TT U-r0r*c~eZdZUeed<dZdZddgZdgZdZ dZ dZ e e dZej fdZxZS) ZambaPreTrainedModelrlmodelTr*r r~)r1 attentionsc|jj}t| |t |t rt j|jd||jjdz}t j|j| ||jj|jjz|jjz}tj tj"|jj|t%j&|jj(t%j&|jj*z zt%j&|jj*zj-|jj.}|tj&tj0|  z}t j2|j4|tj6d|j8dztj:dddf}|j=|j>djA}t j2|jBtj&|jE|j|jFdt jH|jJyy)Nr)r:stdro)minrrr4)&rlinitializer_ranger% _init_weights isinstancerinitnormal_rruniform_rrr-rr(rrandmathr time_step_max time_step_minclamptime_step_floorexpm1copy_rrrr8rLrrcrrMrones_r) r,rSr5 dt_init_stdrdtinv_dtrr.s r/r8z"ZambaPreTrainedModel._init_weightsskk++ f% fo . LL--CS A++33T9K MM&//+{ K![[55 8O8OOSWS^S^SlSllN 4;;44nE88DKK556$++B[B[9\\^((4;;4456e 33e4  %))U[["%5$566F JJv**F 3 Q 5 5 9OPTVWPWXA1126AACA JJv||UYYq\%9%9&:N:NPVPePegi%j k JJvxx % /r0)rCrDrEr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulr rk_can_record_outputsr(no_gradr8rHrIs@r/r1r1sd &*#+-EF#4"5NL/$ U]]_!!r0r1ceZdZdZdeffd Zeee dde jdzde jdzde jdzde dzd e jdzd edzd eed eezfd ZxZS) ZambaModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig rlcFt|||j|_|j|_t j |j|j|j|_|j|_ g}d|_ t|jD]\}}t||}|dk(rt j|jj|jjd}|jt!t#||||jd|dd|di|_ |j|t j$||_t)|j|j* |_d|_|j1y) N)rmhybridFrpz layers.(?!z\.)\d+.shared_transfzlayers.z.shared_transfr)r%r& pad_token_id padding_idx vocab_sizer Embeddingr- embed_tokenslayers_block_type_tied_weights_keys enumerater rxrlrr*r ModuleListrr!rfinal_layernormgradient_checkpointing post_init)r,rlrlayer_idrr"r,r.s r/r&zZambaModel.__init__sc  !.. ++LL):):F,>FDWDWX&+# r0N input_idsrWr$r~ inputs_embedsrrdr#c >|du|duz r td||j|}|}tj|} |r|t |j }|V||j nd} tj|jd|j| z}|jd}t|j ||||} t|jD]\} } | || | || f||d|}|j|}t||r|SdS) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either one)rlrrr)rlrerWr~r$)r~r)last_hidden_stater~)rr[r(rr rlget_seq_lengthrrArrrr^rr`r)r,rdrWr$r~rerrdr1rpast_seen_tokensr#rmlayers r/r>zZambaModel.forwards^  -t"; <s    --i8M% !&]!; 0*$++>O  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L(;;')+%  !*$++ 6  Iu!&  !0#  M ,,]; &+/8O  >B  r0)NNNNNN)rCrDrErrr&rrrr(r(rGr rrrrr@rr>rHrIs@r/rTrTs{8 .2.204(,26!%8 ##d*8  t+8 &&- 8  8 ((4/ 8 $;8 +,8  ( (8  8 r0rTcNeZdZddiZdeffd Zee ddejdzdejdzdejdzd e dzd ejdzd ejdzd e dzd eejzdeedeezfdZ dfd ZxZS)ZambaForCausalLMzlm_head.weightzmodel.embed_tokens.weightrlct||t||_|j|_t j |j|jd|_|jyr) r%r&rTr2rYrrxr-lm_headrbr s r/r&zZambaForCausalLM.__init__sU  '  ++yy!3!3V5F5FUS  r0NrdrWr$r~relabelsrlogits_to_keeprdr#c b|jd||||||d| } | j} t|tr t | dn|} |j | dd| ddf} d}||j | ||jfi| }t|| | j| j| jS)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rdrWr$r~rerNlosslogitsr~r1r3r) r2rhr9rslicero loss_functionrYrr~r1r3)r,rdrWr$r~rerprrqrdoutputsr1 slice_indicesrurts r/r>zZambaForCausalLM.forwardsH,64::, )%+' , ,  11 8B>SV8W~ot4]k mA}a,?@A  %4%% D&#33!//))   r0c h|jj|d<t | |f||||||d|} | S)Nrq)r~rWrer$ris_first_iteration)rlnum_logits_to_keepr%prepare_inputs_for_generation) r,rdr~rWrer$rr{rd model_inputsr.s r/r}z.ZambaForCausalLM.prepare_inputs_for_generationcsU$(;;#A#A w<   +)'%1    r0)NNNNNNNr)NNNNTF)rCrDrEr]rr&rrr(r(rGr rrrrrr@rr>r}rHrIs@r/rmrms0*,GH{.2.204(,26*.!%-.@ ##d*@  t+@ &&- @  @ ((4/ @   4'@ $;@ ell*@ +,@  ' '@ @ J r0rma The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introceZdZfdZee d dejdzdejdzdejdzde dzdejdzdejdzd e dzd e e d eezfd ZxZS)ZambaForSequenceClassificationct|||j|_t||_t j |j|jd|_|jyr) r%r& num_labelsrTr2rrxr-scorerbr s r/r&z'ZambaForSequenceClassification.__init__sS   ++' YYv114??O  r0NrdrWr$r~rerprrdr#c |j|f|||||d|} | j} |j| } ||jd} n|jd} |jj | dk7r t d|jj d} n|||jj k7j| jtj}tj|jd| jtj}||zjd} n.d} tj|jj d| tj| | j | f}d}||j| j}|jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}t7||| j8| j:| j< S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). )rWr$r~rerNrrz=Cannot handle batch sizes > 1 if no padding token is defined.r4rz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rg regressionsingle_label_classificationmulti_label_classificationrs)r2rhrrArlrWrr7rr(int32rargmaxrrr.rC problem_typerr6longrrrrrrrr~r1r3)r,rdrWr$r~rerprrdtransformer_outputsr1rurlast_non_pad_token non_pad_mask token_indices pooled_logitsrtloss_fcts r/r>z&ZambaForSequenceClassification.forwards&8Btzz 8 )%+' 8 8 ,== M*  "+J&,,Q/J ;; # # + a\] ] ;; # # +!#   "%)A)AAEEfmmUZU`U`aL!LL)!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6/ /??-;;*55   r0)NNNNNNN)rCrDrEr&rrr(r(rGr rrrrr@rr>rHrIs@r/rr}s.2.204(,26*.!%R ##d*R  t+R &&- R  R ((4/ R   4'R $;R +,R  1 1R R r0r)rmrrTr1)r)Err>collections.abcrr(rtorch.nnrrrr r: activationsr cache_utilsr r generationr integrations.hub_kernelsr masking_utilsrmodeling_layersrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.import_utilsrutils.output_capturingrconfiguration_zambar get_loggerrCrModuler!rGrrRrFrirkrrrr r*r1rTrmr__all__rr0r/rs& $ AA&!.)8/9qqF&RR795,   H %J299J* UU\\ U# U%,, U&% II% <<% % << % LL4' %  %%2C)RYYC)Ld]biid]P ryy  00f171h414n$!?$!$!N_ %_ _ Fg+_gT  ^ %9^  ^ B gr0