^jbdZddlmZddlZddlmZddlmZddlmZddl m Z m Z d d l m Z e jeZe d eGd deZGddej$ZGddej$ZGddej$ZGddej$Ze GddeZdgZy)zPyTorch UnivNetModel model.) dataclassN)nn) ModelOutput)PreTrainedModel)auto_docstringlogging) UnivNetConfigz Output class for the [`UnivNetModel`], which includes the generated audio waveforms and the original unpadded lengths of those waveforms (so that the padding can be removed by [`UnivNetModel.batch_decode`]). ) custom_introcbeZdZUdZdZej dzed<dZej dzed<y)UnivNetModelOutputa" waveforms (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Batched 1D (mono-channel) output audio waveforms. waveform_lengths (`torch.FloatTensor` of shape `(batch_size,)`): The batched length in samples of each unpadded waveform in `waveforms`. N waveformswaveform_lengths) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rw/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/univnet/modeling_univnet.pyrrs4+/Iu  4'.15e''$.5rrcTeZdZdZdeffd ZdejfdZdZ dZ xZ S)#UnivNetKernelPredictorResidualBlockz Implementation of the residual block for the kernel predictor network inside each location variable convolution block (LVCBlock). Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. configc t||j|_|j|_|j |_|j|_|j dz dz}tj|j|_ tj|j|j|j |d|_ tj|j|j|j |d|_y)Nr Tpaddingbias)super__init__model_in_channelschannelskernel_predictor_conv_size kernel_sizekernel_predictor_dropout dropout_probleaky_relu_sloperDropoutdropoutConv1dconv1conv2)selfrr! __class__s rr$z,UnivNetKernelPredictorResidualBlock.__init__;s 00 !<<";; & 7 7##a'A-zz$"3"34 YYt}}dmmT=M=MW^eij YYt}}dmmT=M=MW^eij r hidden_statesc|}|j|}|j|}tjj ||j }|j |}tjj ||j }||zSN)r-r/r functional leaky_relur+r0)r1r3residuals rforwardz+UnivNetKernelPredictorResidualBlock.forwardKst  ]3  =1  00@U@UV  =1  00@U@UV x''rctjj}ttjjdr$tjjj}||j ||j yN weight_norm)rutilsr<hasattrparametrizationsr/r0r1r<s rapply_weight_normz5UnivNetKernelPredictorResidualBlock.apply_weight_normUsRhh** 288,,m <((33??KDJJDJJrctjj|jtjj|jyr5)rr=remove_weight_normr/r0r1s rrCz6UnivNetKernelPredictorResidualBlock.remove_weight_norm]s. ##DJJ/ ##DJJ/r) rrrrr r$rrr9rArC __classcell__r2s@rrr1s3kk (U%6%6( 0rrcbeZdZdZ d dededeffd ZdejfdZ dZ d Z xZ S) UnivNetKernelPredictora  Implementation of the kernel predictor network which supplies the kernel and bias for the location variable convolutional layers (LVCs) in each UnivNet LVCBlock. Based on the KernelPredictor implementation in [maum-ai/univnet](https://github.com/maum-ai/univnet/blob/9bb2b54838bb6d7ce767131cc7b8b61198bc7558/model/lvcnet.py#L7). Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. conv_kernel_size (`int`, *optional*, defaults to 3): The kernel size for the location variable convolutional layer kernels (convolutional weight tensor). conv_layers (`int`, *optional*, defaults to 4): The number of location variable convolutional layers to output kernels and biases for. rconv_kernel_size conv_layersct||j|_d|jz|_||_||_|j|jz|j z|j z|_|j|j z|_|j|_ |j|_ |j|_|j|_|j"|_|jdz dz}t%j&|j|jddd|_t%j*t-|j Dcgc] }t/|c}|_t%j&|j|j|j|d|_t%j&|j|j|j|d|_ycc}w)Nrr Tr )r#r$model_hidden_channelsconv_in_channelsconv_out_channelsrIrJkernel_channels bias_channels num_mel_binsresnet_in_channels kernel_predictor_hidden_channelsresnet_hidden_channelsr'resnet_kernel_sizekernel_predictor_num_blocks num_blocksr+rr. input_conv ModuleListranger resblocks kernel_conv bias_conv)r1rrIrJr!_r2s rr$zUnivNetKernelPredictor.__init__ss  & < <!"V%A%A!A 0&  ! !D$:$: :T=R=R RUYUeUe e "33d6F6FF"("5"5&,&M&M#"("C"C << & 7 7**Q.14))D$;$;T=X=XZ[efmqr]bcgcrcr]s'tXY(KF(S'tu99  ' ')=)=t?V?V`gnr   ' '););T=T=T^elp  (usG2 spectrogramc|j\}}}|j|}tjj ||j }|j D] }||} |j|}|j|}|j||j|j|j|j|j} |j||j|j|j} | | fS)a Maps a conditioning log-mel spectrogram to a tensor of convolutional kernels and biases, for use in location variable convolutional layers. Note that the input spectrogram should have shape (batch_size, input_channels, seq_length). Args: spectrogram (`torch.FloatTensor` of shape `(batch_size, input_channels, seq_length)`): Tensor containing the log-mel spectrograms. Returns: tuple[`torch.FloatTensor, `torch.FloatTensor`]: tuple of tensors where the first element is the tensor of location variable convolution kernels of shape `(batch_size, self.conv_layers, self.conv_in_channels, self.conv_out_channels, self.conv_kernel_size, seq_length)` and the second element is the tensor of location variable convolution biases of shape `(batch_size, self.conv_layers. self.conv_out_channels, seq_length)`. )shaperYrr6r7r+r\r]r^viewrJrNrOrI contiguous) r1r` batch_sizer_ seq_lengthr3resblockkernel_hidden_statesbias_hidden_stateskernelsbiasess rr9zUnivNetKernelPredictor.forwards "%0$5$5! Az 4  00@U@UV  4H$]3M 4 $// >!^^M:'++      ! !  " "  ! !    *, $((      " "   *, rcvtjj}ttjjdr$tjjj}||j |j D]}|j||j||jyr;) rr=r<r>r?rYr\rAr]r^r1r<layers rrAz(UnivNetKernelPredictor.apply_weight_normshh** 288,,m <((33??KDOO$^^ &E  # # % &D$$%DNN#rc<tjj|j|jD]}|jtjj|j tjj|j yr5)rr=rCrYr\r]r^r1rns rrCz)UnivNetKernelPredictor.remove_weight_normse ##DOO4^^ 'E  $ $ & ' ##D$4$45 ##DNN3r)r rrrrr intr$rrr9rArCrErFs@rrHrHbsO&!" $ $ $  $ L,5#4#4,\ $4rrHc eZdZdZdededeffd ZddZ ddejdejd ejded ef d Z d Z d Z xZ S)UnivNetLvcResidualBlocka Implementation of the location variable convolution (LVC) residual block for the UnivNet residual network. Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. kernel_size (`int`): The kernel size for the dilated 1D convolutional layer. dilation (`int`): The dilation for the dilated 1D convolutional layer. rr(dilationcNt||j|_||_||_|j |_|j |jdz zdz}tj|j|j|j||j |_ y)Nr r)r!rv) r#r$rMhidden_channelsr(rvr+rr.conv)r1rr(rvr!r2s rr$z UnivNetLvcResidualBlock.__init__s %;;&  & 7 7--4#3#3a#78A=II     ]]   rc|}tjj||j}|j |}tjj||j}|j ||||}t j|ddd|jddft j|dd|jdddfz}||z}|SN)hop_size) rr6r7r+rylocation_variable_convolutionrsigmoidrxtanh)r1r3kernelr"r|r8s rr9zUnivNetLvcResidualBlock.forwards  00@U@UV  -0  00@U@UV ::=&RVai:j  mA7M9M9M7Mq4P&QRUZU_U_ !T113Q6 7V  !=0 rr3rr"r|c|j\}}}|j\}}} } } || |zk7rtd| |zd|d|t| dz dz z} tjj || | fdd}|j d|d| zz|}||kr$tjj |d|fdd}|j d||}|d d d d d d d d d |f}|jdd }|j d | d}tjd ||} |jd jd }| |z} | j|| d } | S) u Performs location-variable convolution operation on the input sequence (hidden_states) using the local convolution kernel. This was introduced in [LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation](https://huggingface.co/papers/2102.10815) by Zhen Zheng, Jianzong Wang, Ning Cheng, and Jing Xiao. Time: 414 μs ± 309 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each), test on NVIDIA V100. Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, in_channels, in_length)`): The input sequence of shape (batch, in_channels, in_length). kernel (`torch.FloatTensor` of shape `(batch_size, in_channels, out_channels, kernel_size, kernel_length)`): The local convolution kernel of shape (batch, in_channels, out_channels, kernel_size, kernel_length). bias (`torch.FloatTensor` of shape `(batch_size, out_channels, kernel_length)`): The bias for the local convolution of shape (batch, out_channels, kernel_length). dilation (`int`, *optional*, defaults to 1): The dilation of convolution. hop_size (`int`, *optional*, defaults to 256): The hop_size of the conditioning sequence. Returns: `torch.FloatTensor`: the output sequence after performing local convolution with shape (batch_size, out_channels, in_length). z#Dim 2 of `hidden_states` should be z ) but got zX. Please check `hidden_states` or `kernel` and `hop_size` to make sure they are correct.r rconstantrrNrqzbildsk,biokl->bolsd) rb ValueErrorrsrr6padunfold transposereinsum unsqueezerc)r1r3rr"rvr|batchr_ in_length out_channelsr( kernel_lengthr!output_hidden_statess rr}z5UnivNetLvcResidualBlock.location_variable_convolution s<,11q)=C\\:q, ] 1 25mh6N5OzZcYde]]  S+/Q!677 ))-'79KZYZ[ %,,Q1w;0FQ h MM--ma]JXYZM%,,Q(C %aAq)8)&;< %//15 %,,Q Q?  %||,A=RXY~~b!++B/3d:388 bQ##rctjj}ttjjdr$tjjj}||j yr;)rr=r<r>r?ryr@s rrAz)UnivNetLvcResidualBlock.apply_weight_normMsFhh** 288,,m <((33??KDIIrcVtjj|jyr5)rr=rCryrDs rrCz*UnivNetLvcResidualBlock.remove_weight_normTs ##DII.r)r r)rrrrr rsr$r9rrr}rArCrErFs@rrurus      , * >$((>$!!>$ >$  >$  >$@/rrucxeZdZdZ d dededeffd ZdejdejfdZ d Z d Z xZ S) UnivNetLvcBlocka# Implementation of the location variable convolution (LVC) residual block of the UnivNet residual block. Includes a `UnivNetKernelPredictor` inside to predict the kernels and biases of the LVC layers. Based on LVCBlock in [maum-ai/univnet](https://github.com/maum-ai/univnet/blob/9bb2b54838bb6d7ce767131cc7b8b61198bc7558/model/lvcnet.py#L98) Parameters: config (`UnivNetConfig`): Config for the `UnivNetModel` model. layer_id (`int`): An integer corresponding to the index of the current LVC resnet block layer. This should be between 0 and `len(config.resblock_stride_sizes) - 1)` inclusive. lvc_hop_size (`int`, *optional*, defaults to 256): The hop size for the location variable convolutional layers. rlayer_id lvc_hop_sizec t||j|_|j||_|j ||_|j||_ ||_ |j|_ t|j|_ tj|j|jd|jz|j|jdz|jdzz|jdz|_t#||j |j|_tj&t)|jDcgc]&}t+||j |j|(c}|_ycc}w)Nr)strider!output_padding)r#r$rMrxresblock_kernel_sizesr(resblock_stride_sizesrresblock_dilation_sizes dilationscond_hop_lengthr+lenrXrConvTranspose1d convt_prerHkernel_predictorrZr[rur\)r1rrrir2s rr$zUnivNetLvcBlock.__init__js< %;;!77A228< 77A+ & 7 7dnn-++    O;;KK1$t{{Q6;;?  !7vt?O?OQUQ`Q` a[`aeapap[q rVW $VT-=-=t~~a?P Q r  rs+Fr3r`c Rtjj||j}|j |}|j |\}}t |jD]?\}}|dd|ddddddddf}|dd|ddddf}|||||j}A|Sr{) rr6r7r+rr enumerater\r) r1r3r`rjrkrrgrr"s rr9zUnivNetLvcBlock.forwards 00@U@UV }5 // <$T^^4 aKAxQ1aA-.F!Q1*%D$]FD4K_K_`M a rcbtjj}ttjjdr$tjjj}||j |j j|jD]}|jyr;) rr=r<r>r?rrrAr\rms rrAz!UnivNetLvcBlock.apply_weight_normswhh** 288,,m <((33??KDNN# //1^^ &E  # # % &rctjj|j|jj|j D]}|jyr5)rr=rCrrr\rps rrCz"UnivNetLvcBlock.remove_weight_normsI ##DNN3 002^^ 'E  $ $ & 'rrrrrFs@rrrXsW*      < U%6%6 UEVEV &'rrceZdZUeed<dZdZdeffd Ze dde jde jdzde jdzde jdzd e dzd e e jezf d Zd Zd ZxZS) UnivNetModelrinput_featuresaudioc ^t||t|j|_|j |_t j|j|jdddd|_ t|j}d}g}|jD]}||z}|j|t jt|Dcgc]}t||||c}|_t j|jdddd|_|j%ycc}w)Nr rreflect)r(rr! padding_mode)rr)r!r)r#r$rr num_kernelsr+rr.r%rMconv_prerappendrZr[rr\ conv_post post_init)r1r num_layers hop_length hop_lengthsrrr2s rr$zUnivNetModel.__init__s  v;;< & 7 7  $ $  ( ("   556   22 +F#f,J   z * +z*     !,Q   6#?#?Aq_hi  s D*Nnoise_sequence padding_mask generator return_dictreturnc ||n|jj}|jdk(}|s|jd}|j\}} } |'|jdk(} | sX|jd}nF|| |jj f} t j| ||j|j}|jd} |dkDr| dk(r|j|dd}n| dkDr|dk(r|j| dd}| |k7rtd| d|d|J|jdk(r|jd}|jd}||k7rtd |d|d|jd d}|jd d}|j|}|jD] }|||} tj j#||j$}|j'|}t j(|}|j+d}d}|t j,|d }|s||f}|St/|| S) a noise_sequence (`torch.FloatTensor`, *optional*): Tensor containing a noise sequence of standard Gaussian noise. Can be batched and of shape `(batch_size, sequence_length, config.model_in_channels)`, or un-batched and of shape (sequence_length, config.model_in_channels)`. If not supplied, will be randomly generated. padding_mask (`torch.BoolTensor`, *optional*): Mask indicating which parts of each sequence are padded. Mask values are selected in `[0, 1]`: - 1 for tokens that are **not masked** - 0 for tokens that are **masked** The mask can be batched and of shape `(batch_size, sequence_length)` or un-batched and of shape `(sequence_length,)`. generator (`torch.Generator`, *optional*): A [torch generator](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. return_dict: Whether to return a [`~utils.ModelOutput`] subclass instead of a plain tuple. Example: ```python >>> from transformers import UnivNetFeatureExtractor, UnivNetModel >>> from datasets import load_dataset, Audio >>> model = UnivNetModel.from_pretrained("dg845/univnet-dev") >>> feature_extractor = UnivNetFeatureExtractor.from_pretrained("dg845/univnet-dev") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> # Resample the audio to the feature extractor's sampling rate. >>> ds = ds.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate)) >>> inputs = feature_extractor( ... ds[0]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="pt" ... ) >>> audio = model(**inputs).waveforms >>> list(audio.shape) [1, 140288] ``` Nrr)rdtypedevicer z&The batch size of `noise_sequence` is z+ and the batch size of `input_features` is z', but the two are expected to be equal.z$The batch size of `padding_mask` is r)dim)rr)rrrrrbr%rrandnrrrepeatrrrr\rr6r7r+rrsqueezesumr)r1rrrrrkwargsspectrogram_batchedspectrogram_batch_sizespectrogram_lengthr_noise_sequence_batchednoise_sequence_shapenoise_sequence_batch_sizepadding_mask_batch_sizer3rgwaveformroutputss rr9zUnivNetModel.forwardsb&1% ")!/!9!9!!<%; >89R8ST((>'??fh   #!Q&+55a8 &2&8&8&; #&*@@ :;R:ST,,B+CCjl '00A6 '11!Q7 m4  DH$]NCM D 00@U@UV }5  =1 !((+   #$yy1= !12GN!-  rcRtjj}ttjjdr$tjjj}||j |j D]}|j||jyr;) rr=r<r>r?rr\rArrms rrAzUnivNetModel.apply_weight_normQsqhh** 288,,m <((33??KDMM"^^ &E  # # % &DNN#rctjj|j|jD]}|jtjj|j yr5)rr=rCrr\rrps rrCzUnivNetModel.remove_weight_norm[sM ##DMM2^^ 'E  $ $ & ' ##DNN3r)NNNN)rrrr rmain_input_nameinput_modalitiesr$rrr Generatorbooltuplerr9rArCrErFs@rrrs &O%}%N4815,0#' y ))y ))D0y ''$. y ??T) y D[ y  u  !$6 6y y v$4rr)r dataclassesrrrmodeling_outputsrmodeling_utilsrr=rr configuration_univnetr get_loggerrloggerrModulerrHrurr__all__rrrrs"! +-,0   H %   6 6  6.0")).0bu4RYYu4p{/bii{/|M'biiM'`v4?v4v4r  r