^jAdZddlZddlmZddlmZddlZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZmZmZddlm Z ejBe"Z#GddejHZ%GddejHZ&GddejHZ'GddejHZ(GddejHZ)GddejHZ* dzdejHdejVdejVdejVd ejVdzd!e,dzd"e,d#eefd$Z-Gd%d&ejHZ.Gd'd(ejHZ/Gd)d*ejHZ0Gd+d,ejHZ1Gd-d.ejHZ2Gd/d0ejHZ3Gd1d2ejHZ4eGd3d4e Z5Gd5d6ejHZ6 d{d7ejVd8e,d9e7dzd:e8d;e9f d<Z: d|d7ejVd=e7e9zd9e7dzd;e9fd>Z;Gd?d@ejHZ<GdAdBejHZ=GdCdDejHZ>GdEdFejHZ?GdGdHejHZ@edIJeGdKdLe ZAGdMdNe5ZBedOJeGdPdQe ZCedRJGdSdTe5ZDedUJeGdVdWe ZEedXJGdYdZe5ZFed[JeGd\d]e ZGed^JeGd_d`e ZHed^JeGdadbe ZIdcejjddejVdeejVfdfZLd}dgejVdhejVdzdeejVfdiZMGdjdke5ZNedlJeGdmdne ZOGdodpe5ZPedqJeGdrdse ZQGdtduejHZRedvJGdwdxe5ZSgdyZTy)~zPyTorch PatchTSMixer model.N)Callable) dataclass)PreTrainedModel) ModelOutput)initialization)FlashAttentionKwargs)ALL_ATTENTION_FUNCTIONS)Unpack)NegativeBinomialOutput NormalOutputStudentTOutput)TransformersKwargsauto_docstringcan_return_tuplelogging)PatchTSMixerConfigc2eZdZdZdedeffd ZdZxZS)PatchTSMixerGatedAttentionz Module that applies gated attention to input data. Args: in_size (`int`): The input size. out_size (`int`): The output size. in_sizeout_sizect|tj|||_tj d|_y)Ndim)super__init__nnLinear attn_layerSoftmax attn_softmax)selfrr __class__s /var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/patchtsmixer/modeling_patchtsmixer.pyrz#PatchTSMixerGatedAttention.__init__/s1 ))GX6JJ2.cP|j|j|}||z}|SN)r#r!)r$inputs attn_weights r&forwardz"PatchTSMixerGatedAttention.forward4s*''(?@ +% r')__name__ __module__ __qualname____doc__intrr, __classcell__r%s@r&rr&s //s/ r'rcHeZdZdZdeffd ZdejfdZxZ S)PatchTSMixerBatchNormzP Compute batch normalization over the sequence length (time) dimension. configct|tj|j|j |_y)Neps)rrr BatchNorm1dd_modelnorm_eps batchnormr$r6r%s r&rzPatchTSMixerBatchNorm.__init__@s( FOOLr'r*cl|jdd}|j|}|jddS)a Parameters: inputs (`torch.Tensor` of shape `(batch_size, sequence_length, d_model)`): input for Batch norm calculation Returns: `torch.Tensor` of shape `(batch_size, sequence_length, d_model)` r) transposer=)r$r*outputs r&r,zPatchTSMixerBatchNorm.forwardDs7!!!Q''1%%r' r-r.r/r0rrtorchTensorr,r2r3s@r&r5r5;s'M1M &ell &r'r5cveZdZdZdeffd ZededejfdZ de jfdZ xZ S)PatchTSMixerPositionalEncodingz' Class for positional encoding r6ct||jr|j||_yt j tj|j|j|_yr)) rruse_positional_encoding_init_pe position_encr ParameterrDzeros num_patchesr;r>s r&rz'PatchTSMixerPositionalEncoding.__init__VsN   ) ) $ f 5D  " U[[9K9KV^^-\ ]D r'returnc`|jdk(rAtjtj|j |j d}|S|jdk(r7tj|j |j }tjd|j jd}tjtjd|j dtjd|j z z}tj||z|dddddf<tj||z|dddddf<||jz }||j!d zz }tj|d }|St#|jd ) NrandomT requires_gradsincosrrr@g@ FzN is not a valid positional encoder. Available types are 'random' and 'sincos'.)positional_encoding_typerrLrDrandnrNr;rMarange unsqueezeexpmathlogsincosmeanstd ValueError)r6rKpositiondiv_terms r&rJz'PatchTSMixerPositionalEncoding._init_pe^sv  * *h 6<< F4F4F(WgklL , , 8 ;;v'9'96>>JL||Av'9'9:DDQGHyya!CQXHY\b\j\jHjFk!klH$)IIh.A$BLADqD !$)IIh.A$BLADqD !',*;*;*==L'<+;+;+=+BCL<< EJL 2234BC r' patch_inputc$||jz}|Sr))rK)r$rd hidden_states r&r,z&PatchTSMixerPositionalEncoding.forwardrs"T%6%66 r')r-r.r/r0rr staticmethodrrLrJrDrEr,r2r3s@r&rGrGQsN^1^+ &5<<r'rGcHeZdZdZdeffd ZdejfdZxZ S)PatchTSMixerNormLayerzeNormalization block Args: config (`PatchTSMixerConfig`): Configuration. r6ct||j|_d|jjvrt ||_yt j|j|j|_y)Nbatchr8) rrnorm_mlplowerr5normr LayerNormr;r<r>s r&rzPatchTSMixerNormLayer.__init__sT  foo++- --f5DI V^^IDIr'r*cfd|jjvrtj||jd|jdz|jd|jdf}|j |}tj||j}|S|j |}|S)a Args: inputs (`torch.Tensor` of shape `((batch_size, num_channels, num_patches, d_model))`): Input to the normalization layer. Returns: `torch.Tensor` of shape `((batch_size, num_channels, num_patches, d_model))` rkrrr@r)rlrmrDreshapeshapern)r$r*inputs_reshapeds r&r,zPatchTSMixerNormLayer.forwards dmm))+ +#mmLLOfll1o5LLOLLOO#ii8O]]?FLLAF  YYv&F r'rCr3s@r&ririxs'J1Jellr'ric>eZdZfdZdej fdZxZS)PatchTSMixerMLPc<t|||jz}tj|||_tj |j|_tj|||_ tj |j|_ yr)) rrexpansion_factorrr fc1Dropoutdropoutdropout1fc2dropout2)r$ in_features out_featuresr6 num_hiddenr%s r&rzPatchTSMixerMLP.__init__sj  6#:#:: 99[*5 6>>2 99Z6 6>>2 r'r*c|jtjj|j |}|j |}|j |}|S)z Args: inputs (`torch.Tensor` of shape `((batch_size, num_channels, num_patches, d_model))`): Input to the MLP layer. Returns: `torch.Tensor` of the same shape as `inputs` )r{r functionalgelurxr|r})r$r*s r&r,zPatchTSMixerMLP.forwardsKr}}11$((62BCD&!v& r')r-r.r/rrDrEr,r2r3s@r&rurus3 ell r'rucHeZdZdZdeffd ZdejfdZxZ S)$PatchTSMixerChannelFeatureMixerBlockzThis module mixes the features in the channel dimension. Args: config (`PatchTSMixerConfig`): Configuration. r6ct|t||_|j|_t |j |j ||_|jr't|j |j |_ yyNr~rr6rr) rrrirn gated_attnrunum_input_channelsmlpr gating_blockr>s r&rz-PatchTSMixerChannelFeatureMixerBlock.__init__sv )&1  ++"1122     :11FL==((6??([L,,|U3K''1-88:K  $$r'c(eZdZdZ ddedededededed edzffd Z dd e jd e jdzd e jdzdedzde e de e je jdze e jdzff dZxZS)PatchTSMixerAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsrz is_decoderbias is_causalr6c t|||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ tj||||_ tj||||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: ).r)r)rrrrrzhead_dimr6rarrrrr k_projv_projq_projout_proj) r$rrrzrrrr6r%s r&rzPatchTSMixerAttention.__init__s "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r' hidden_stateskey_value_statesroutput_attentionsrrOc |du}|jdd}g|d|j}|j|j|j dd} |r|n|} g| jddd|j} |j | j| j dd} |j | j| j dd} tj|jjt}||| | | |f|jsdn |j|j|d|\}}|jg|dj!}|j#|}||dfS)z#Input shape: Batch x Time x ChannelNrrr@)rzrr)rrrrviewrArrr get_interfacer6_attn_implementationrrrzrrqrr)r$rrrrris_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacerrs r&r,zPatchTSMixerAttention.forward0s.T9$))#2. 88b8$--8 {{=166|DNNqRST -?)]B^))#2.BBDMMB[[055h?II!QO {{>277AKKAqQ (?(M(M KK , ,.E) %8      % $}}C$,,LL/ %  % ! \*k));;;;FFH mmK0 L$..r')rFTFN)NNF)r-r.r/r0r1floatboolrrrDrEr r tupler,r2r3s@r&rrsG  ,0CCC C  C  CC#T)CD15.2). 0/||0/ ,,-0/ t+ 0/  $; 0/-.0/ u||U\\D0% 2E2LL M0/r'rc.eZdZdZdeffd ZdZxZS)PatchMixerBlockzxThis module mixes the patch dimension. Args: config (`PatchTSMixerConfig`): Configuration. r6ct|t||_|j|_|j |_t |j|j||_|j r&t|j|j|_ |jrCt|j|j|j||_t||_yy)Nrr)rrrzr6)rrrirn self_attnrrurNrrrrr;self_attn_headsrzself_attn_layer norm_attnr>s r&rzPatchMixerBlock.__init__ks )&1 )) ++"**++     :6CUCU`f`r`r sD    #8 .. 00 $D 36:DN r'c|}|j|}|jrR|j\}}}}|j||z||}|j |d\}} } |j||||}|j dd}|j |}|jr|j|}|j dd}|jr|j|z}||z} | S)z Args: hidden_state (`torch.Tensor`): Input tensor. Returns: `torch.Tensor`: Transformed tensor. F)rr@r) rnrrrrqrrArrrr) r$rfr batch_sizen_varsrNr;hidden_state_reshapedx_attn_rs r&r,zPatchMixerBlock.forwards yy. >>7C7I7I 4J W$0$8$8f9Lk[b$c !//0EY^/_LFAq^^J WMF$--a3 xx - ??,,\:L$--a3 >>>>,*?@LX% r'r-r.r/r0rrr,r2r3s@r&rrcs;1;4!r'rcHeZdZdZdeffd ZdejfdZxZ S)FeatureMixerBlockzThis module mixes the hidden feature dimension. Args: config (`PatchTSMixerConfig`): Configuration. r6ct|t||_|j|_t |j |j ||_|jr't|j |j |_ yyr) rrrirnrrur;rrrr>s r&rzFeatureMixerBlock.__init__sn )&1  ++"     :6>>\b\j\j kD  r'hiddenc|}|j|}|j|}|jr|j|}||z}|S) Args: hidden (`torch.Tensor` of shape `(batch_size, num_patches, d_model)`): Input tensor to the layer. Returns: `torch.Tensor`: Transformed tensor. )rnrrr)r$rrrs r&r,zFeatureMixerBlock.forwardsK6"&! ??&&v.Fx r'rCr3s@r&rrs'l1l ellr'rcHeZdZdZdeffd ZdejfdZxZ S)PatchTSMixerLayerz The `PatchTSMixer` layer that does all three kinds of mixing. Args: config (`PatchTSMixerConfig`): Configuration. r6ct|t||_t ||_|j |_|j dk(rt||_yy)Nr6 mix_channel) rrr patch_mixerr feature_mixermoderchannel_feature_mixerr>s r&rzPatchTSMixerLayer.__init__sR *&9.f=KK ;;- ')MU[)\D & (r'rc|jdk(r|j|}|j|}|j|}|S)rr)rrrr)r$rs r&r,zPatchTSMixerLayer.forwardsE 99 %//7F!!&)##F+ r'rCr3s@r&rrs' ]1 ]ellr'rc6eZdZdZdeffd ZddefdZxZS)PatchTSMixerBlockzThe main computing framework of the `PatchTSMixer` model. Args: config (`PatchTSMixerConfig`): Configuration. r6ct||j}tjt |Dcgc]}t |c}|_ycc}wNr)rr num_layersr ModuleListrangermixers)r$r6rrr%s r&rzPatchTSMixerBlock.__init__sC && mmuU_O`$a!%6f%E$ab $asAoutput_hidden_statescxg}|}|jD]}||}|s|j| |r||fS|dfS)as Args: hidden_state (`torch.Tensor`): The input tensor. output_hidden_states (`bool`, *optional*, defaults to False.): Whether to output the hidden states as well. Returns: `torch.Tensor`: The embedding. `list`: List of all hidden states if `output_hidden_states` is set to `True`. N)rappend)r$rfrall_hidden_states embeddingmods r&r,zPatchTSMixerBlock.forward sW ;; 4CII#!((3 4 // /d? "r'F) r-r.r/r0rrrr,r2r3s@r&rrs#c1c#$#r'rc0eZdZdZddeffd ZdZxZS)PatchTSMixerForPredictionHeadzqPrediction Head for Forecasting Args: config (`PatchTSMixerConfig`): Configuration. r6ct||j|_|j|jjt j |j |_|=t j|j|jz|j|_ n-|j|j|jz|_ t jd|_y)N start_dim)rrprediction_channel_indicessortrry head_dropout dropout_layerr rNr;prediction_lengthbase_forecast_blockget_parameter_projectionFlattenflatten)r$r6distribution_outputr%s r&rz&PatchTSMixerForPredictionHead.__init__-s *0*K*K'  * * 6  + + 0 0 2ZZ(;(;<  &')yy&2D2Dv~~2UX^XpXp'qD $':'S'S""V^^3(D $zzB/ r'c^j|}j|}j|}t|trt d|D}n|j dd}j 7t|trt fd|D}|S|dj f}|S)ar Args: hidden_features (`torch.Tensor` of shape `(batch_size, num_patch, d_model)` in `flatten` mode or `(batch_size, n_vars, num_patch, d_model)` in `common_channel`/`mix_channel` mode.): Input hidden features. Returns: `torch.Tensor` of shape `(batch_size, prediction_length, nvars)`. c3@K|]}|jddyw)rrN)rA).0zs r& z8PatchTSMixerForPredictionHead.forward..PsCQQ[[R0Csrrc3@K|]}|djfyw).N)r)rrr$s r&rz8PatchTSMixerForPredictionHead.forward..Vs! [Q3(G(G#G!H [s.)rrr isinstancerrArr$hidden_featuresforecasts` r&r,z%PatchTSMixerForPredictionHead.forward?s,,7,,_=++O< h &C(CCH))"b1H  * * 6(E* [RZ [[$C)H)H$HIr'r)rr3s@r&rr%s010$r'rc0eZdZdZddeffd ZdZxZS)PatchTSMixerLinearHeadzLinear head for Classification and Regression. Args: config (`PatchTSMixerConfig`): Configuration. r6c\t||j|_|j|_|j |j}nd}||_|@t j|j|jz|z|j|_ n0|j|j|jz|z|_ |jt jd|_nt jd|_t j|j |_y)Nrrr)rrhead_aggregation output_rangerNrrr r;r num_targets projectionrrrryrrz)r$r6r mul_factorr%s r&rzPatchTSMixerLinearHead.__init__es  & 7 7"//  " " *++JJ#6  & ii!:!::ZG""DO 2JJ!:!::ZGDO  " " *::3DL::3DLzz&"5"56 r'c&|jdd}|jdk(r|d}nM|jdk(r|jdj}n!|jdk(r|j d}|j r|j |}|j |}|j|}|jQ|jEtj||jd|jd z z|jd z}|S) ai Args: hidden_features (`torch.Tensor` of shape `(batch_size x num_patch x d_model)` in `flatten` mode or `(batch_size x n_vars x num_patch x d_model)` in `common_channel`/`mix_channel` mode.): Input hidden features. Returns: `torch.Tensor` of shape `(batch_size x num_targets)`. rruse_last).rmax_poolravg_poolrr) rArmaxvaluesr_rrzrrrrDsigmoid)r$r s r&r,zPatchTSMixerLinearHead.forwards *33B;  J .-g6O  " "j 0-11b19@@O  " "j 0-22r2:O <<"ll?;O,,7///:  $ $ ,43D3D3P o.$2C2CA2FIZIZ[\I]2]^aeararstauu r'r)rr3s@r&rr]s7178 r'rcbeZdZUeed<dZdZdZdZe jfdZ xZ S)PatchTSMixerPreTrainedModelr6model past_values)timeFct||t|tr=|jj dk(r#t j|jddyyt|trSt j|jjt j|jjyy)zInitialize weightsrQrg?)r_r`N)r _init_weightsr rGr6rVinitnormal_rKr5zeros_r=rones_weight)r$rr%s r&r#z)PatchTSMixerPreTrainedModel._init_weightss f% f< ={{33x? V00sD@  5 6 KK((-- . JJv''.. /7r') r-r.r/r__annotations__base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointingrDno_gradr#r2r3s@r&rrs< #O &+#U]]_ 0 0r'rc.eZdZdZdeffd ZdZxZS)PatchTSMixerPretrainHeadzcPretraining head. Args: config (`PatchTSMixerConfig`): Configuration. r6ct|tj|j|_tj |j|j|_ yr)) rrrryrrr r; patch_length base_pt_blockr>s r&rz!PatchTSMixerPretrainHead.__init__sB ZZ(;(;<YYv~~v7J7JKr'cJ|j|}|j|}|S)a Args: hidden_features (`torch.Tensor` of shape `(batch_size x num_patch x d_model)` in `flatten` mode or `(batch_size x n_vars x num_patch x d_model)` in `common_channel`/`mix_channel` mode.): Input hidden features. Returns: `torch.Tensor` of shape `(batch_size x n_vars x num_patch x patch_length)`. )rr3r s r&r,z PatchTSMixerPretrainHead.forwards),,_=%%o6r'rr3s@r&r0r0sL1L r'r0r* mask_ratiounmasked_channel_indiceschannel_consistent_masking mask_valuec|dks|dk\rtd|d|j\}}}}|j} t|d|z z} |r-t j |d|| } | j d|d} nt j |||| } t j|||| } d| ddddd| f<t j| d} t j| d}t j| d| } | jdj ddd|} |d| dd|ddddf<|j| j|}|| d fS) arandom_masking: Mask the input considering the control variables. Args: inputs (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, num_features)`): The input tensor to mask. mask_ratio (`float`): Masking ratio applied to mask the input data during random pretraining. It is the number between 0 and 1. unmasked_channel_indices (list, *optional*): Indices of channels that will not be masked. channel_consistent_masking (bool, *optional*, defaults to `False`): When true, masking will be same across all channels of a timeseries. Otherwise, masking positions will vary across channels. mask_value (int, *optional*, defaults to 0): Define the value of masked patches for pretraining. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as input Tensor and mask tensor of shape [bs x c x n] rrz Mask ratio z has to be between 0 and 1.deviceNrr)rindex.r) rarrr;r1rDrandrepeatonesargsortgatherrY masked_fillr)r*r5r6r7r8r num_channelssequence_length num_featuresr;len_keepnoisemask ids_shuffle ids_restore inputs_masks r&random_maskingrMsQ4A~q;zl2MNOO>Dll;J o| ]]F?a*n56H! :q/&I Q a0 :|_VT ::j, ODDAyy--2.K-- 4K <<"K 8D >>"  $ $Q1l ;D+23Q(!Q ./$$TYY[*=K V $$r'num_forecast_mask_patchescPt|tr|g}|Dcgc]}d}}|j\}}}} tj||||j } g} d} t |} t||D]H\}}|dks||k\rtd|dt||z| z }| j|||g| |z } Jt| d} | |kr| dd|| z z| dd<n| |kDr| d d| |z z| d d<d}| D]\}}}||z}d| ||d d | d f<|}tj| jd}| |} | jd jddd| } |d| d d |d d d d f<|j| j|}|| d fScc}w) aForecast masking that masks the last K patches where K is from the num_forecast_mask_patches. If num_forecast_mask_patches is a list, samples in the batch will be randomly masked by numbers defined in the list. Parameters: inputs (`torch.Tensor`): Input of shape `(bs, num_channels, num_patch, patch_length)` num_forecast_mask_patches (`list`): Number of patches to be masked at the end of each batch sample. e.g. 4 or [3, 5]. unmasked_channel_indices (`list`, *optional*): Indices of channels that are not masked. mask_value (`int`, *optional*, defaults to 0): Values in the masked patches will be filled by `mask_value`. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as inputs Tensor and Mask tensor of shape `(bs, num_channels , num_patch)` or `(bs, tsg1, tsg2, num_channels, num_patch)` rr:rznum_forecast_mask_patches z6 should be greater than 0 and less than total patches.c |dSNr@)xs r&z"forecast_masking..@s !A$r')rr@rNr=)r r1rrrDrMr;sumziprarsortedrandpermrYr?rCr)r*rNr6r8rforecast_mask_ratiosrrDrErFrIt_list total_length total_ratior2ratiotemp_lenbatch1 patch_lenbatch2permrLs r&forecast_maskingrcs 0+S1%>$?!'@A!AAA>Dll;J o| ;;z< WD FL*+K"#<>RS! e 1   ?,\N:pq zE)K78 |UH56 !F /Fj ay|zL'@Aq !  "r 1  )BCr 1 F"( 1h("./VF]A z{ *+ >>$**Q- (D :D >>"  $ $Q1l ;D+23Q(!Q ./$$TYY[*=K V $$OBs F#cHeZdZdZdeffd ZdejfdZxZ S)PatchTSMixerPatchifyz A class to patchify the time series sequence into different patches Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` r6ct||j|_|j|_|j |_|j|jkr&t d|jd|jdt|j|j|jz |j zdz|_|j|j |jdz zz}|j|z |_ y)NzSequence length (z+) has to be greater than the patch length ()r) rrcontext_lengthrEr2 patch_striderarrNsequence_start)r$r6new_sequence_lengthr%s r&rzPatchTSMixerPatchify.__init__as %44"//"//   4#4#4 4#D$8$8#99deievevdwwxy    4 4d6G6GH4K\K\\aeararruvv"//$2C2CtGWGWZ[G[2\\"225HHr'r c:|jd}||jk7rtd|d|jd|dd|jdddf}|j d|j |j }|jddj}|S)a! Parameters: past_values (`torch.Tensor` of shape `(batch_size, sequence_length, num_channels)`, *required*): Input for patchification Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` rzInput sequence length (z%) doesn't match model configuration (rN) dimensionrstepr) rrrErarjunfoldr2rirAr)r$r rErBs r&r,zPatchTSMixerPatchify.forwardrs&++B/ d22 2)/)::_`d`t`t_uuwx Q 3 3 5q89$2C2C$J[J[\!!"b)446 r'rCr3s@r&rereYs'I1I"5<<r'recHeZdZdZdeffd ZdejfdZxZ S)PatchTSMixerMaskinga Class to perform random or forecast masking. Parameters: config (`PatchTSMixerConfig`): model config Returns: x_mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points r6c<t||j|_|j|_|j|_|j |_|j |_|j|_|j t|j |_yyr)) rrrandom_mask_ratior7 mask_typerNr6r8rWr>s r&rzPatchTSMixerMasking.__init__s !'!9!9*0*K*K')))/)I)I&(.(G(G% ++  ( ( 4,243P3P,QD ) 5r'rdcr|jdk(r>X %!/"11)-)F)F+/+J+J?? " L$^^z )!1"*.*H*H)-)F)F?? " L$1$..1ACD Dyy{T!!r'rCr3s@r&rqrqs'  R1 R!"5<<!"r'rqc eZdZdZdeffd ZdejdejdeejejejffdZ xZ S)PatchTSMixerStdScalerz Standardize features by calculating the mean and scaling along the first dimension, and then normalizes it by subtracting from the mean and dividing by the standard deviation. r6ct|t|dr |jnd|_t|dr |j nd|_t|dr|j |_yd|_y)N scaling_dimrkeepdimT minimum_scalegh㈵>)rrhasattrr{rr|r}r>s r&rzPatchTSMixerStdScaler.__init__s[ )0)G6%%Q)0)Cv~~ 5s r&rzPatchTSMixerMeanScaler.__init__su )0)G6%%Q)0)Cv~~ 5s r&rzPatchTSMixerNOPScaler.__init__'s@ )0)G6%%Q)0)Cv~~  r'NrrrOctj|dj|j|j}tj |dj|j|j}|||fS)a Parameters: data (`torch.Tensor` of shape `(batch_size, sequence_length, num_input_channels)`): input for Batch norm calculation Returns: tuple of `torch.Tensor` of shapes (`(batch_size, sequence_length, num_input_channels)`,`(batch_size, 1, num_input_channels)`, `(batch_size, 1, num_input_channels)`) FrR)rr|)rDrr_rr|r)r$rrrrs r&r,zPatchTSMixerNOPScaler.forward,siE:??DHHVZVbVb?ct59>>488UYUaUa>bS%r'r)rr3s@r&rr"s`N1N MQ LL 6;llT6I u||U\\5<<7 8 r'rzS Base class for `PatchTSMixerEncoderOutput`, with potential hidden states. ) custom_introcheZdZUdZdZej dzed<dZe ej dzed<y)PatchTSMixerEncoderOutputa- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, num_patches, d_model)`): Hidden-state at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*): Hidden-states of the model at the output of each layer. Nlast_hidden_stater) r-r.r/r0rrD FloatTensorr)rrrRr'r&rr=s937u((4/659M5**+d29r'rc neZdZdZdeffd Zee d dejde dzde fdZ xZ S) PatchTSMixerEncoderz Encoder for PatchTSMixer which inputs patched time-series and outputs patched embeddings. Args: config (`PatchTSMixerConfig`): Configuration. r6ct||tj|j|j |_|jrt||_ nd|_ t||_ |jyr) rrrr r2r;patcherrIrGpositional_encoderrmlp_mixer_encoder post_initr>s r&rzPatchTSMixerEncoder.__init__Xsc  yy!4!4fnnE  ) )&DF&SD #&*D #!2&!A r'Nr rrOc ||n|jj}|j|}|j|j|}|j ||\}}t ||S)a past_values (`torch.FloatTensor` of shape `(batch_size, seq_length, num_input_channels)`): Context values of the time series. For a pretraining task, this denotes the input time series to predict the masked portion. For a forecasting task, this denotes the history/past time series values. Similarly, for classification or regression tasks, it denotes the appropriate context values of the time series. For univariate time series, `num_input_channels` dimension should be 1. For multivariate time series, it is greater than 1. Returns: `torch.FloatTensor` of shape `(batch_size, n_vars, num_patches, d_model)` r)rr)r6rrrrr)r$r rrpatchesrrs r&r,zPatchTSMixerEncoder.forwardesw.%9$D $++JjJj  ,,{+  " " .--g6G+/+A+A'`t+A+u(=(;L\ijjr'r))r-r.r/r0rrrrrDrErrr,r2r3s@r&rrOs_ 1 -1!k\\!k#Tk!k # !k!kr'rzG Base class for model's outputs, with potential hidden states. ceZdZUdZdZej dzed<dZe ej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<y) PatchTSMixerModelOutputa last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, num_patches, d_model)`): Hidden-state at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*): Hidden-states of the model at the output of each layer. patch_input (`torch.FloatTensor` of shape `(batch_size, num_channels, num_patches, patch_length)`): Patched input data to the model. mask (`torch.FloatTensor` of shape `(batch_size, num_channels, num_patches)`, *optional*): Bool Tensor indicating True in masked patches and False otherwise. loc (`torch.FloatTensor` of shape `(batch_size, 1, num_channels)`, *optional*): Gives the mean of the context window per channel. Used for revin denorm outside the model, if revin enabled. scale (`torch.FloatTensor` of shape `(batch_size, 1, num_channels)`, *optional*): Gives the std dev of the context window per channel. Used for revin denorm outside the model, if revin enabled. NrrrdrIrr)r-r.r/r0rrDrr)rrrdrIrrrRr'r&rrs"37u((4/659M5**+d29,0K""T)0%)D%  d ")$(C  T !(&*E5  t #*r'rz= The PatchTSMixer Model for time-series forecasting. c eZdZd dedeffd Zee d dejdejdzdedzde fd Z xZ S) PatchTSMixerModelr6 mask_inputcxt||t||_t ||_|durt ||_nd|_|jdk(rt||_ n>|jdk(s|jdurt||_ nt||_ |jy)z mask_input (bool, *optional*, defaults to `False`): Whether to mask the input using the [`PatchTSMixerMasking`] module. TNr_r`)rrrencoderrepatchingrqmaskingrrscalerryrr)r$r6rr%s r&rzPatchTSMixerModel.__init__s  *62 ,V4  .v6DLDL >>V #08DK ^^u $$(>/7DK/7DK r'Nr  observed_maskrrOc ||n|jj}d}|tj|}|j ||\}}}|j |} | } |j |j | \} }|j| |} t| trt| } t| j| j| |||S)a past_values (`torch.FloatTensor` of shape `(batch_size, seq_length, num_input_channels)`): Context values of the time series. For a pretraining task, this denotes the input time series to predict the masked portion. For a forecasting task, this denotes the history/past time series values. Similarly, for classification or regression tasks, it denotes the appropriate context values of the time series. For univariate time series, `num_input_channels` dimension should be 1. For multivariate time series, it is greater than 1. observed_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). Nr)rrrdrIrr)r6rrDrrrrrr rrrrr) r$r rrrrIscaled_past_valuesrr patched_x enc_inputencoder_outputs r&r,zPatchTSMixerModel.forwards0%9$D $++JjJj   !OOK8M)-[-)P&CMM"45  << #"ll95OIt !5&  ne ,6GN&,>>(66!   r'rNN) r-r.r/rrrrrrDrErr,r2r3s@r&rrsr 1t2.2,0 5 \\5 ||d*5 #Tk 5 ! 5 5 r'rz> Output type of [`PatchTSMixerForPreTrainingOutput`]. ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e ej dzed<y) PatchTSMixerForPreTrainingOutputa@ loss (*optional*, returned when `y` is provided, `torch.FloatTensor` of shape `()`): Total loss prediction_outputs (`torch.FloatTensor` of shape `(batch_size, num_input_channels, num_patches, patch_length)`): Prediction output from the pretrain head. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_input_channels, num_patches, d_model)`): Backbone embeddings before passing through the head. hidden_states (`tuple(torch.FloatTensor)`, *optional*): Hidden-states of the model at the output of each layer. Nlossprediction_outputsrr r-r.r/r0rrDrr)rrrrrRr'r&rrd &*D%  d ")37))D0726u((4/659M5**+d29r'rz. `PatchTSMixer` for mask pretraining. c eZdZdeffd Zee d dejdejdzde dzde de f d Z xZ S) PatchTSMixerForPretrainingr6ct||t|d|_t ||_|j |_|jy)NT)rr)rrrrr0head masked_lossrr>s r&rz#PatchTSMixerForPretraining.__init__"sD  &v$? ,F; !-- r'Nr rr return_lossrOc ||n|jj}|jdur!tjj d}n tjj d}|j |||}t|trt|}|j|j}|dur|||j} nd} |jdurM| K| jd|jzj|jjd zz } t!| ||j|j" S) aT past_values (`torch.FloatTensor` of shape `(batch_size, seq_length, num_input_channels)`): Context values of the time series. For a pretraining task, this denotes the input time series to predict the masked portion. For a forecasting task, this denotes the history/past time series values. Similarly, for classification or regression tasks, it denotes the appropriate context values of the time series. For univariate time series, `num_input_channels` dimension should be 1. For multivariate time series, it is greater than 1. observed_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). return_loss (`bool`, *optional*): Whether to return the loss in the `forward` call. NTnone reductionr_rrrrrrrrr)r6rrrDrMSELossrr rrrrrdr_rIrUrr) r$r rrrrr model_outputx_hatloss_vals r&r,z"PatchTSMixerForPretraining.forward+s;6%9$D $++JjJj    t #88##f#5D88##f#5Dzz '!5" lE *2LAL ,889 $ E<#;#;]>]9]+^_H''#77DD|//|7I7I E  !,1D#L-@H/9H 2 22\5E5EE ,1D#E=9H  * * 6""3(G(G#GHC &&sD,K,K'KLE""C &&E.$*<<&44   r'cF|j}||d|d}|jj|j|j|j }t |Dcgc]}|j}}tj|d}t|Scc}w)a Generate sequences of sample predictions from a model with a probability distribution head. Args: past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_input_channels)`): Past values of the time series that serves as context in order to predict the future. observed_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_input_channels)`, *optional*): Boolean mask to indicate which `past_values` were observed and which were missing. Mask values selected in `[0, 1]`: - 1 for values that are **observed**, - 0 for values that are **missing** (i.e. NaNs that were replaced by zeros). Return: [`SamplePatchTSMixerPredictionOutput`] where the outputs `sequences` tensor will have shape `(batch_size, number of samples, prediction_length, num_input_channels)`. NF)r rrrrrrr) rrrrrrrsamplerDstackr)r$r rroutputsrrsampless r&generatez"PatchTSMixerForPrediction.generatebs2 $88#'!&  //<<  & &GKKw}}= 388L2MNQ<&&(NN++g1-1GDD Os"B)NNNTr))r-r.r/r0rrrrrDrErrr,r.rr r2r3s@r&rrs 1<.2-1,0 k \\k ||d*k ||d* k #Tk k  k  )k k ZU]]_.2-E\\-E||d*-E , -E-Er'rzK Output type of [`PatchTSMixerForTimeSeriesClassificationOutput`]. ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e ej dzed<y)-PatchTSMixerForTimeSeriesClassificationOutputaP loss (*optional*, returned when `y` is provided, `torch.FloatTensor` of shape `()`): Total loss. prediction_outputs (`torch.FloatTensor` of shape `(batch_size, num_labels)`): Prediction output from the classification head. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_input_channels, num_patches, d_model)`): Backbone embeddings before passing through the head. hidden_states (`tuple(torch.FloatTensor)`, *optional*): Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. NrrrrrrRr'r&r r rr'r c eZdZdZdeffd Zee d dejdejdzde dzde d e f d Z xZ S) 'PatchTSMixerForTimeSeriesClassificationz `PatchTSMixer` for classification application. Args: config (`PatchTSMixerConfig`): Configuration. Returns: `None`. r6ct||t||_t ||_|j dvr't|j|j|_ nd|_ |jy)Nrr`r_Tr;rN) rrrrrrrInjectScalerStatistics4Dr;rN inject_scalerr>s r&rz0PatchTSMixerForTimeSeriesClassification.__init__sh  &v. *   >>2 2 8]c]o]o pD  $D  r'Nr  target_valuesrrrOc tjj}||n|jj}|j ||}t |trt|}|j7|j|j|j|j|_ |j|j}||dur |||} nd} t| ||j|jS)a past_values (`torch.FloatTensor` of shape `(batch_size, seq_length, num_input_channels)`): Context values of the time series. For a pretraining task, this denotes the input time series to predict the masked portion. For a forecasting task, this denotes the history/past time series values. Similarly, for classification or regression tasks, it denotes the appropriate context values of the time series. For univariate time series, `num_input_channels` dimension should be 1. For multivariate time series, it is greater than 1. target_values (`torch.FloatTensor` of shape `(batch_size, target_len, num_input_channels)` for forecasting, `(batch_size, num_targets)` for regression, or `(batch_size,)` for classification, *optional*): Target values of the time series, that serve as labels for the model. The `target_values` is what the Transformer needs during training to learn to output, given the `past_values`. Note that, this is NOT required for a pretraining task. For a forecasting task, the shape is be `(batch_size, target_len, num_input_channels)`. Even if we want to forecast only specific channels by setting the indices in `prediction_channel_indices` parameter, pass the target data with all channels, as channel Filtering for both prediction and target will be manually applied before the loss computation. For a classification task, it has a shape of `(batch_size,)`. For a regression task, it has a shape of `(batch_size, num_targets)`. return_loss (`bool`, *optional*): Whether to return the loss in the `forward` call. NrrTr)rDrCrossEntropyLossr6rrr rrrrrrrr r) r$r rrrrrrrrs r&r,z/PatchTSMixerForTimeSeriesClassification.forwardsJxx((*%9$D $++JjJj zz !5"  lE *2LAL    (-1->->.. $$"((.?.L *  ,889  $)<E=1HH<$*<<&44   r'r)r-r.r/r0rrrrrDrErr r,r2r3s@r&r r s  1 .2,0 C \\C ||d*C #Tk C  C  7C C r'r z= Output type of [`PatchTSMixerForRegressionOutput`]. ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e ej dzed<y)PatchTSMixerForRegressionOutputaM loss (*optional*, returned when `y` is provided, `torch.FloatTensor` of shape `()`): Total loss. regression_outputs (`torch.FloatTensor` of shape `(batch_size, num_targets)`): Prediction output from the regression head. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_input_channels, num_patches, d_model)`): Backbone embeddings before passing through the head. hidden_states (`tuple(torch.FloatTensor)`, *optional*): Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. Nrregression_outputsrr) r-r.r/r0rrDrr)rrrrrRr'r&rrrr'rc~eZdZd dededeffd Zdej dej dej fdZxZS) rr;rN expansionc&t|tj|dz||z|_tj||z||_tjdd|z|_tjd|zd|_||_yrQ) rrrr inverse_trans_expansioninverse_trans_compressionmap_scale_expansionmap_scale_compressionrN)r$r;rNrr%s r&rz!InjectScalerStatistics4D.__init__'sx ')yy1i'>Q'R$)+9w3F)P&#%99QI #> %'YYq9}a%@"&r'r*rrc|jdd}|jd}|jdd|jd}|jdd}|jd}|jdd|jd}t j ||gd}|j |}|j|}t j ||gd}|j|}|j|}|S)a Args: inputs (`torch.Tensor` of shape `(batch_size, num_input_channels, num_patch, d_model)`) loc (`torch.Tensor` of shape `(batch_size, 1, num_input_channels)`) scale (`torch.Tensor` of shape `(batch_size, 1, num_input_channels)`) Returns: `torch.Tensor` of shape `(batch_size, num_input_channels, num_patch, d_model)` rrrr) rArYr?rNrDcatrrrr)r$r*rrr_stdev concat_statss r&r,z InjectScalerStatistics4D.forward0s}}R$~~b!{{1a!1!115B'# Q4#3#3Q7yy$B7 // = 11,? FL1r:--f5//7 r')r@) r-r.r/r1rrDrEr,r2r3s@r&rr&sC''#'#'ellellr'rz4 `PatchTSMixer` for regression application. c eZdZdeffd Zee d dejdejdzde dzde de f d Z ejdejde fd ZxZS) PatchTSMixerForRegressionr6cht||t||_|j|_|j |_|j |_|jdk(rd|_n`tttd}|j|j }|||j|_ntd|j |jdvr't|j|j |_nd|_t%||j |_|j)y)Nrrrrrrr)rrrrrrrrr r rrrarrr;rNrrrr)r$r6rrr%s r&rz"PatchTSMixerForRegression.__init__Ts  &v. KK #)#=#= $*$?$?! ;;% '+D $,&%;' # 366v7Q7QRL'+7F>2 2 8]c]o]o pD  $D * $ 8 8  r'Nr rrrrOc Fjdk(rtjd}n!jdk(rt}n t d||nj j }j||}t|trt|}j7j|j|j|j|_ j|j}||d urj rvj d k(r#t#j$|d kr t'd j j)|} tfd |D}|| |} t+| } n |||} nd} t-| ||j|j.S)a past_values (`torch.FloatTensor` of shape `(batch_size, seq_length, num_input_channels)`): Context values of the time series. For a pretraining task, this denotes the input time series to predict the masked portion. For a forecasting task, this denotes the history/past time series values. Similarly, for classification or regression tasks, it denotes the appropriate context values of the time series. For univariate time series, `num_input_channels` dimension should be 1. For multivariate time series, it is greater than 1. target_values (`torch.FloatTensor` of shape `(batch_size, target_len, num_input_channels)` for forecasting, `(batch_size, num_targets)` for regression, or `(batch_size,)` for classification, *optional*): Target values of the time series, that serve as labels for the model. The `target_values` is what the Transformer needs during training to learn to output, given the `past_values`. Note that, this is NOT required for a pretraining task. For a forecasting task, the shape is be `(batch_size, target_len, num_input_channels)`. Even if we want to forecast only specific channels by setting the indices in `prediction_channel_indices` parameter, pass the target data with all channels, as channel Filtering for both prediction and target will be manually applied before the loss computation. For a classification task, it has a shape of `(batch_size,)`. For a regression task, it has a shape of `(batch_size, num_targets)`. return_loss (`bool`, *optional*): Whether to return the loss in the `forward` call. rr_rrrNrrTrrzDtarget_values cannot be negative for negative_binomial distribution.c3jK|]*}|jdjj,yw)rN)rr6r)ritemr$s r&rz4PatchTSMixerForRegression.forward..s&WdiiDKK,C,CDWs03)rrrr)rrrrrar6rrr rrrrrrrrrDany Exceptionrrrr) r$r rrrrrrrrrs ` r&r,z!PatchTSMixerForRegression.forwardysH 99 ::/D YY% DQR R%9$D $++JjJj zz !5"  lE *2LAL    (-1->->.. $$"((.?.L *  ,889  $)<''++/BBuyyQ^abQbGc#$jkk#77DDUK WQVWW m<+H5}5H.$*<<&44   r'c`|j}||dd}|jj|j}t |Dcgc]}|j }}t j|djd||jj}t|Scc}w)a  Generate sequences of sample predictions from a model with a probability distribution head. Args: past_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_input_channels)`): Past values of the time series that serves as context in order to predict the target values. Return: [`SamplePatchTSMixerRegressionOutput`] where the outputs `sequences` tensor will have shape `(batch_size, number of samples, num_targets)`. NF)r rrrrrr) rrrrrrrDrrr6rr)r$r rrrrrs r&r z"PatchTSMixerForRegression.generates" $88#!& //<r@sS" $! 7*&B5&UURR:   H %*&BII&,$RYY$N.BII.bbii.-299-n!% II% <<% % << % LL4' % T\ %%'(%:R/BIIR/jCbiiCL* *Z# #L&# &#R5BII5pDRYYDN0/00*ryyD-1', 7% LL7%7%#Tk7%!% 7%  7%|-1 A% LLA%#czA%#TkA% A%J-299-b9""))9"z 0BII 0H3;RYY3;n BII 6   : :  :9k59kx  +k+  +4 Q 3Q  Q h  :{:  :$ F !<F  F R  +k+  +0  // /  // /#u""//##%,,#*5<<*%,,:M*chcoco*0GE ;GET  :K:  :$` .I` F  :k:  :$%ryy%P ^E ;^E  ^EB r'