^jXdZddlmZddlZddlmZmZmZm Z ddl m Z ddl m Z ddlmZmZmZej$eZGd d e Zd gZy) z)Feature extractor class for UnivNetModel.)AnyN)mel_filter_bankoptimal_fft_length spectrogramwindow_function)SequenceFeatureExtractor) BatchFeature)PaddingStrategy TensorTypeloggingc)eZdZdZgdZ d.dededededed ed ed ed edzd edededzdedededededededef(fd Z dZ dZ de jde jfdZ d/dede jj dzde jfd Zd/dee jfd!Z d0d"e jeezee jzeeezdedzd#eezezd$edzd%ed&edzd'ede jj dzd(ed)edzdedzd*edzd+eezdzdefd,Zdeeefffd- ZxZS)1UnivNetFeatureExtractora Constructs a UnivNet feature extractor. This class extracts log-mel-filter bank features from raw speech using the short time Fourier Transform (STFT). The STFT implementation follows that of TacoTron 2 and Hifi-GAN. This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: feature_size (`int`, *optional*, defaults to 1): The feature dimension of the extracted features. sampling_rate (`int`, *optional*, defaults to 24000): The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). padding_value (`float`, *optional*, defaults to 0.0): The value to pad with when applying the padding strategy defined by the `padding` argument to [`UnivNetFeatureExtractor.__call__`]. Should correspond to audio silence. The `pad_end` argument to `__call__` will also use this padding value. do_normalize (`bool`, *optional*, defaults to `False`): Whether to perform Tacotron 2 normalization on the input. Normalizing can help to significantly improve the performance for some models. num_mel_bins (`int`, *optional*, defaults to 100): The number of mel-frequency bins in the extracted spectrogram features. This should match `UnivNetModel.config.num_mel_bins`. hop_length (`int`, *optional*, defaults to 256): The direct number of samples between sliding windows. Otherwise referred to as "shift" in many papers. Note that this is different from other audio feature extractors such as [`SpeechT5FeatureExtractor`] which take the `hop_length` in ms. win_length (`int`, *optional*, defaults to 1024): The direct number of samples for each sliding window. Note that this is different from other audio feature extractors such as [`SpeechT5FeatureExtractor`] which take the `win_length` in ms. win_function (`str`, *optional*, defaults to `"hann_window"`): Name for the window function used for windowing, must be accessible via `torch.{win_function}` filter_length (`int`, *optional*, defaults to 1024): The number of FFT components to use. If `None`, this is determined using `transformers.audio_utils.optimal_fft_length`. max_length_s (`int`, *optional*, defaults to 10): The maximum input length of the model in seconds. This is used to pad the audio. fmin (`float`, *optional*, defaults to 0.0): Minimum mel frequency in Hz. fmax (`float`, *optional*): Maximum mel frequency in Hz. If not set, defaults to `sampling_rate / 2`. mel_floor (`float`, *optional*, defaults to 1e-09): Minimum value of mel frequency banks. Note that the way [`UnivNetFeatureExtractor`] uses `mel_floor` is different than in [`transformers.audio_utils.spectrogram`]. center (`bool`, *optional*, defaults to `False`): Whether to pad the waveform so that frame `t` is centered around time `t * hop_length`. If `False`, frame `t` will start at time `t * hop_length`. compression_factor (`float`, *optional*, defaults to 1.0): The multiplicative compression factor for dynamic range compression during spectral normalization. compression_clip_val (`float`, *optional*, defaults to 1e-05): The clip value applied to the waveform before applying dynamic range compression during spectral normalization. normalize_min (`float`, *optional*, defaults to -11.512925148010254): The min value used for Tacotron 2-style linear normalization. The default is the original value from the Tacotron 2 implementation. normalize_max (`float`, *optional*, defaults to 2.3143386840820312): The max value used for Tacotron 2-style linear normalization. The default is the original value from the Tacotron 2 implementation. model_in_channels (`int`, *optional*, defaults to 64): The number of input channels to the [`UnivNetModel`] model. This should match `UnivNetModel.config.model_in_channels`. pad_end_length (`int`, *optional*, defaults to 10): If padding the end of each waveform, the number of spectrogram frames worth of samples to append. The number of appended samples will be `pad_end_length * hop_length`. return_attention_mask (`bool`, *optional*, defaults to `True`): Whether or not [`~UnivNetFeatureExtractor.__call__`] should return `attention_mask`. )input_featuresnoise_sequence padding_maskN feature_size sampling_rate padding_value do_normalize num_mel_bins hop_length win_length win_function filter_length max_length_sfminfmax mel_floorcentercompression_factorcompression_clip_val normalize_min normalize_maxmodel_in_channelspad_end_lengthc t|d||||d|||_||_||_||_||_| |_| |_| t|dz } | |_ | |_ | |_ | |z|_ |jt|j |_n|j|_|jdzdz|_t#|j |j d|_t'|j |j|j|j|j(dd|_||_||_||_||_||_||_||_y) N)rrrreturn_attention_maskT) window_lengthnameperiodicslaney)num_frequency_binsnum_mel_filters min_frequency max_frequencyrnorm mel_scale)super__init__rrrrrrrfloatrrrnum_max_samplesrn_fftn_freqsrwindowrr mel_filtersr r!r"r#r$r%r&)selfrrrrrrrrrrrrrr r!r"r#r$r%r&r(kwargs __class__s /var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/univnet/feature_extraction_univnet.pyr7z UnivNetFeatureExtractor.__init__esf2  %''"7    )($$(* <'!+D "(+m;    %+DOOrs rA normalizez!UnivNetFeatureExtractor.normalizes4[4#5#55$:L:LtOaOa:abcfgggrBc`|j|j|jz |dzdz zzS)Nr*r)rDrEs rA denormalizez#UnivNetFeatureExtractor.denormalizes6!!T%7%7$:L:L%LR]`aRaefQf$gggrBwaveformreturnc tj|t|j|jz dz t|j|jz dz fd}t ||j |j|j|jd|jdd }tjtj|dztj|dzz|jz}tj|jj|}tjtj ||j"d|j$z}|jS)a Calculates log MEL spectrograms from a batch of waveforms. Note that the input waveform(s) will be padded by `int(self.n_fft - self.hop_length) / 2` on both sides using the `reflect` padding mode. Args: waveform (`np.ndarray` of shape `(length,)`): The input waveform. This must be a single real-valued, mono waveform. Returns: `numpy.ndarray`: Array containing a log-mel spectrogram of shape `(num_frames, num_mel_bins)`. r)reflect)modeN)r< frame_lengthr fft_lengthpowerr r=r)a_mina_max)nppadintr:rrr<r sqrtrealimagrmatmulr=Tlogclipr"r!)r>rIcomplex_spectrogramamplitude_spectrogrammel_spectrogramlog_mel_spectrograms rAr_z'UnivNetFeatureExtractor.mel_spectrograms/66  $**t.!3 4c4::;W[\:\6] ^ * ;;zz;;  !# GG' (A -8K0LPQ0Q QTXTbTb b! ))D$4$4$6$68MN!ff GGO4+D+DD QTXTkTk k  #$$$rB noise_length generatorc|tjj}||jf}|j |tj }|S)a Generates a random noise sequence of standard Gaussian noise for use in the `noise_sequence` argument of [`UnivNetModel.forward`]. Args: spectrogram_length (`int`): The length (dim 0) of the generated noise. model_in_channels (`int`, *optional*, defaults to `None`): The number of features (dim 1) of the generated noise. This should correspond to the `model_in_channels` of the [`UnivNetGan`] model. If not set, this will default to `self.config.model_in_channels`. generator (`numpy.random.Generator`, *optional*, defaults to `None`) An optional `numpy.random.Generator` random number generator to control noise generation. If not set, a new generator with fresh entropy will be created. Returns: `numpy.ndarray`: Array containing random standard Gaussian noise of shape `(noise_length, model_in_channels)`. dtype)rSrandom default_rngr%standard_normalfloat32)r>rarb noise_shapenoises rAgenerate_noisez&UnivNetFeatureExtractor.generate_noisesJ0   --/I#T%;%;< ))+RZZ)H rBc|Dcgc]1}|jjddj3}}|#t|Dcgc] \}}|d||}}}|Scc}wcc}}w)a Removes padding from generated audio after running [`UnivNetModel.forward`]. This returns a ragged list of 1D audio waveform arrays and not a single tensor/array because in general the waveforms will have different lengths after removing padding. Args: waveforms (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): The batched output waveforms from the [`UnivNetModel`]. waveform_lengths (`torch.FloatTensor` of shape `(batch_size,)`, *optional*): The batched lengths of each waveform before padding. Returns: `list[np.ndarray]`: A ragged list of 1D waveform arrays with padding removed. cpuT)devicecopyN)detachtonumpy enumerate)r> waveformswaveform_lengthsrIis rA batch_decodez$UnivNetFeatureExtractor.batch_decodes| \eexX__&))T)BHHJe e  'LUV_L`a[Q"7$4Q$78aIa fbs 6A$ A) raw_speechpadding max_length truncationpad_to_multiple_of return_noisepad_end pad_lengthr(return_tensorsc | | n |j} |O||jk7rmtd|jjd|jd|jd|d t j d|jjdt|tjxrt|jdkD}|r&t|jd kDrtd ||xs@t|ttfxr(t|d tjttf}|r3|Dcgc]'}tj|tj )}}n|s@t|tjs&tj|tj }nht|tjrN|j tj tj"ur|j%tj}|s&tj|tj g}| rN| | n |j&} |Dcgc]3}tj(|d | |j*zf|j, 5}}t/d|i}|j)||||n |j0||| }|j3d}|Dcgc]}|j5|}}t|d tr7|Dcgc]'}tj|tj )c}|d<n0|Dcgc]!}|j%tj#c}|d<|j3d}|6|Dcgc]'}tj|tj6 )c}|d<|r4|dDcgc]!}|j9|jd |#}}||d<| r%|dDcgc]}|j;|c}|d<| |j=| }|Scc}wcc}wcc}wcc}wcc}wcc}wcc}wcc}w)a Main method to featurize and prepare for the model one or several sequence(s). Args: raw_speech (`np.ndarray`, `list[float]`, `list[np.ndarray]`, `list[list[float]]`): The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. sampling_rate (`int`, *optional*): The sampling rate at which the `raw_speech` input was sampled. It is strongly recommended to pass `sampling_rate` at the forward call to prevent silent errors and allow automatic speech recognition pipeline. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the input `raw_speech` waveforms (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). If `pad_end = True`, that padding will occur before the `padding` strategy is applied. max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). truncation (`bool`, *optional*, defaults to `True`): Activates truncation to cut input sequences longer than `max_length` to `max_length`. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. return_noise (`bool`, *optional*, defaults to `True`): Whether to generate and return a noise waveform for use in [`UnivNetModel.forward`]. generator (`numpy.random.Generator`, *optional*, defaults to `None`): An optional `numpy.random.Generator` random number generator to use when generating noise. pad_end (`bool`, *optional*, defaults to `False`): Whether to pad the end of each waveform with silence. This can help reduce artifacts at the end of the generated audio sample; see https://github.com/seungwonpark/melgan/issues/8 for more details. This padding will be done before the padding strategy specified in `padding` is performed. pad_length (`int`, *optional*, defaults to `None`): If padding the end of each waveform, the length of the padding in spectrogram frames. If not set, this will default to `self.config.pad_end_length`. do_normalize (`bool`, *optional*): Whether to perform Tacotron 2 normalization on the input. Normalizing can help to significantly improve the performance for some models. If not set, this will default to `self.config.do_normalize`. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific feature_extractor's default. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'pt'`: Return PyTorch `torch.np.array` objects. - `'np'`: Return Numpy `np.ndarray` objects. z3The model corresponding to this feature extractor: z& was trained using a sampling rate of zI. Please make sure that the provided `raw_speech` input was sampled with z and not .zDIt is strongly recommended to pass the `sampling_rate` argument to `zN()`. Failing to do so can result in silent errors that might be hard to debug.r*r)z2Only mono-channel audio is supported for input to rrd)constant_valuesr)rzr{r|r}r(attention_maskrr)rr ValueErrorr@__name__loggerwarning isinstancerSndarraylenshapelisttupleasarrayrirefloat64astyper&rTrrr r9getr_int32rlrFconvert_to_tensors)r>ryrrzr{r|r}r~rbrrrr(ris_batched_numpy is_batchedspeechrIbatched_speech padded_inputsrmel_spectrogramsmelrarrayrrks rA__call__z UnivNetFeatureExtractor.__call__sV(4'?|TEVEV  $ 2 22 I$..JaJaIbc))-););(<=))-););(XJXJz2::$FJbjjAJ  BJJ /J4D4DQSQ[Q[H\4\#**2::6J**ZrzzBCJ '1'=4CVCVJ!+x!Z$//%A!BTXTfTfgJ &'7&DE %/%;zAUAU!1"7 ! '**+;<KYZxD00:ZZ nQ' .]m/nVY 3bjj0Q/nN+ ,Rb/c3 2::0F/cN+ ,'**+;<  %]k-lTYbjjbhh.O-lN> * $22B#C##K$5$5a$8)DE05N+ , ?MN^?_00;{+0N+ ,  %+>>~NNyY([0o/c .m0s05,P 8P! P&>,P+5&P08,P53&P:)P?cJt|}gd}|D] }||vs||= |S)N)r<r=r:r;r9)r6to_dict)r>outputnamesr,r@s rArzUnivNetFeatureExtractor.to_dicts;"Q !Dv~4L ! rB)r*i]Fd hann_windowr rNg& .>Fg?gh㈵>g 'gă@@rT)N) NTNTNTNFNNNN)r __module__ __qualname____doc__model_input_namesrUr8boolstrr7rFrHrSrr_rf Generatorrlrrxr r r rdictrr __classcell__)r@s@rArrsCJM""")$(!$'&*21!# "-J-J-J- J-  J-  J-J-J-J-TzJ-J-J-dlJ-J-J- "!J-"$#J-$%J-&'J-()J-*+J-Xhh.% .%rzz.%f1599&&-  @RZZ@P4%)04!%)-!04!%#'-126^JJe,tBJJ/??$tE{BSS^Tz^o- ^ $J ^  ^ $J^^99&&-^^$J^Dj^ $d{^j(4/^ ^@ c3h  rBr)rtypingrrsrS audio_utilsrrrr!feature_extraction_sequence_utilsr feature_extraction_utilsr utilsr r r get_loggerrrr__all__r5rBrArsP0\\I499   H %j6jZ % %rB