^jddlmZddlmZmZmZddlZddlZddl m Z ddl m Z ddl mZmZmZmZdd lmZdd lmZerdd lmZdd lmZdd lmZej8eZer ddlZddl m!Z!dZ"ddZ#dZ$GddeZ%y)) defaultdict) TYPE_CHECKINGAnyUnionN)GenerationConfig)PreTrainedTokenizer)is_torch_availableis_torchaudio_availableis_torchcodec_availablelogging) ffmpeg_read) ChunkPipeline)BeamSearchDecoderCTC)SequenceFeatureExtractor)PreTrainedModel)(MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMEScg}|D]g\}}}tt||z}tt||z |z}tt||z |z}|||f}|j|i|S)z Rescales the stride values from audio space to tokens/logits space. (160_000, 16_000, 16_000) -> (2000, 200, 200) for instance. )introundappend)strideratio new_stridesinput_nleftrighttoken_n new_strides ~/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/pipelines/automatic_speech_recognition.pyrescale_strider")sK &'ueGeO,-5'123E%'/G345tU+ :& ' c#TK|jd}||z |z }td||D]}}||z} ||| } || |jdd} || j|} |dk(rdn|} | |k\} | rdn|}| jd}|| |f}| jd| kDr | |d| | s}yyw)NrptT sampling_ratereturn_tensorsreturn_attention_maskdtype)is_lastr)shaperanger'to)inputsfeature_extractor chunk_len stride_left stride_rightr+ inputs_lenstepchunk_start_idx chunk_end_idxchunk processed _stride_leftr, _stride_rightrs r! chunk_iterr==saJ { "\ 1D J5')3 }5% +99"&   ! 5 1I+q0qk :-$, KKN \=9 ;;q>L (%E9E E  )s B!B($B(c D|ddjDcgc]}||jvs|}}|ddD]}|djDcgc]}||jvs|}}d}d}tdt|dzD]d}|dz } t j t j || dt j |d|k(} | |z | z} | dkDs[| |kDsa|}| }f|j||dt j |Scc}wcc}w)Nrrgg@)tolistall_special_idsr.lennpsumarrayextend) sequences tokenizertok_idsequencenew_seq new_sequenceindexmax_iepsmatchesmatchings r!_find_longest_common_sequencerRWs3&/q\!_%;%;%=i6yOhOhAhiHiQR= .-4QZ->->-@l6FR[RkRkDkl lq#l+a/0 Ag+CffRXXhrsm4bqAQ8RRSG{S(H{x$   UV,- . 88H jlsDDD$Dc 8eZdZdZdZdZdZdZdZe ddZ ddd d e d e fdzd e dzd e de fdzde edfdzf fd Zdej"eze zezdedeee efffd Z ddZedZddZddZ d dedzfdZxZS)!"AutomaticSpeechRecognitionPipelineao Pipeline that aims at extracting spoken text contained within some audio. The input can be either a raw waveform or a audio file. In case of the audio file, ffmpeg should be installed for to support multiple audio formats Unless the model you're using explicitly sets these generation parameters in its configuration files (`generation_config.json`), the following default values will be used: - max_new_tokens: 256 - num_beams: 5 Example: ```python >>> from transformers import pipeline >>> transcriber = pipeline(model="openai/whisper-base") >>> transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac") {'text': ' He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick, peppered flour-fatten sauce.'} ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) Arguments: model ([`PreTrainedModel`]): The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from [`PreTrainedModel`]. feature_extractor ([`SequenceFeatureExtractor`], *optional*): The feature extractor that will be used by the pipeline to encode waveform for the model. tokenizer ([`PreTrainedTokenizer`], *optional*): The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from [`PreTrainedTokenizer`]. decoder (`pyctcdecode.BeamSearchDecoderCTC`, *optional*): [PyCTCDecode's BeamSearchDecoderCTC](https://github.com/kensho-technologies/pyctcdecode/blob/2fd33dc37c4111417e08d89ccd23d28e9b308d19/pyctcdecode/decoder.py#L180) can be passed for language model boosted decoding. See [`Wav2Vec2ProcessorWithLM`] for more information. device (Union[`int`, `torch.device`], *optional*): Device ordinal for CPU/GPU supports. Setting this to `None` will leverage CPU, a positive will run the model on the associated CUDA device id. TF)max_new_tokens num_beamsNmodelrr1rrGdecoderrdevicez torch.devicec D|jjdk(rd|_nj|jjt j vrd|_n8|jjdvrd|_n|||_d|_nd|_t|$|||fd|i|y) Nwhisperseq2seq_whisperseq2seq) parakeet_tdt parakeet_rnntnemotron_asr_streamingnemotron3_5_asrtdt ctc_with_lmctcr[) config model_typetype __class____name__rvaluesrZsuper__init__)selfrYr1rGrZr[kwargsrjs r!rnz+AutomaticSpeechRecognitionPipeline.__init__s << " "i /)DI __ % %)Q)X)X)Z Z!DI \\ $ $(v vDI  "DL%DIDI  +<VVVvVr#r0rpreturnc $t||fi|S)ae Transcribe the audio sequence(s) given as inputs to text. See the [`AutomaticSpeechRecognitionPipeline`] documentation for more information. Args: inputs (`np.ndarray` or `bytes` or `str` or `dict`): The inputs is either : - `str` that is either the filename of a local audio file, or a public URL address to download the audio file. The file will be read at the correct sampling rate to get the waveform using *ffmpeg*. This requires *ffmpeg* to be installed on the system. - `bytes` it is supposed to be the content of an audio file and is interpreted by *ffmpeg* in the same way. - (`np.ndarray` of shape (n, ) of type `np.float32` or `np.float64`) Raw audio at the correct sampling rate (no further check will be done) - `dict` form can be used to pass raw audio sampled at arbitrary `sampling_rate` and let this pipeline do the resampling. The dict must be in the format `{"sampling_rate": int, "raw": np.array}` with optionally a `"stride": (left: int, right: int)` than can ask the pipeline to treat the first `left` samples and last `right` samples to be ignored in decoding (but used at inference to provide more context to the model). Only use `stride` with CTC models. return_timestamps (*optional*, `str` or `bool`): Only available for pure CTC models (Wav2Vec2, HuBERT, etc) and the Whisper model. Not available for other sequence-to-sequence models. For CTC models, timestamps can take one of two formats: - `"char"`: the pipeline will return timestamps along the text for every character in the text. For instance, if you get `[{"text": "h", "timestamp": (0.5, 0.6)}, {"text": "i", "timestamp": (0.7, 0.9)}]`, then it means the model predicts that the letter "h" was spoken after `0.5` and before `0.6` seconds. - `"word"`: the pipeline will return timestamps along the text for every word in the text. For instance, if you get `[{"text": "hi ", "timestamp": (0.5, 0.9)}, {"text": "there", "timestamp": (1.0, 1.5)}]`, then it means the model predicts that the word "hi" was spoken after `0.5` and before `0.9` seconds. For the Whisper model, timestamps can take one of two formats: - `"word"`: same as above for word-level CTC timestamps. Word-level timestamps are predicted through the *dynamic-time warping (DTW)* algorithm, an approximation to word-level timestamps by inspecting the cross-attention weights. - `True`: the pipeline will return timestamps along the text for *segments* of words in the text. For instance, if you get `[{"text": " Hi there!", "timestamp": (0.5, 1.5)}]`, then it means the model predicts that the segment "Hi there!" was spoken after `0.5` and before `1.5` seconds. Note that a segment of text refers to a sequence of one or more words, rather than individual words as with word-level timestamps. generate_kwargs (`dict`, *optional*): The dictionary of ad-hoc parametrization of `generate_config` to be used for the generation call. For a complete overview of generate, check the [following guide](https://huggingface.co/docs/transformers/en/main_classes/text_generation). Return: `Dict`: A dictionary with the following keys: - **text** (`str`): The recognized text. - **chunks** (*optional(, `list[Dict]`) When using `return_timestamps`, the `chunks` will become a list containing all the various text chunks identified by the model, *e.g.* `[{"text": "hi ", "timestamp": (0.5, 0.9)}, {"text": "there", "timestamp": (1.0, 1.5)}]`. The original full text can roughly be recovered by doing `"".join(chunk["text"] for chunk in output["chunks"])`. )rm__call__)ror0rprjs r!rsz+AutomaticSpeechRecognitionPipeline.__call__srw1&11r#c |i}i} i} |@|jdvr-|s+d} |jdk(r| dz } tj| ||d<|||d<d|vr | j|j d| j|t |dd|j | d<t |d d|j| d <|j| d <||| d <|$|jdk7r td || d <|| d <t|dr0t|jdr|xs|jj}||jdk(r |r td|jdk(r|dk7r td|jdk(r|dvr td|jdk(r|dk(r td|| d<|| d<|| | fS)N)r_r^aUsing `chunk_length_s` is very experimental with seq2seq models. The results will not necessarily be entirely accurate and will have caveats. More information: https://github.com/huggingface/transformers/pull/20104. Ignore this warning with pipeline(..., ignore_warning=True).r^z To use Whisper for long-form transcription, use rather the model's `generate` method directly as the model relies on it's own chunking mechanism (cf. Whisper original paper, section 3.8. Long-form Transcription).chunk_length_sstride_length_sgenerate_kwargsassistant_modelassistant_tokenizerrGdecoder_kwargsz)Only Whisper can return language for now.return_languagegeneration_configreturn_timestampsr_zEWe cannot return_timestamps yet on non-CTC models apart from Whisper!rewordzRCTC with LM can only predict word level timestamps, set `return_timestamps='word'`rf)charr~zCTC can either predict character level timestamps, or word level timestamps. Set `return_timestamps='char'` or `return_timestamps='word'` as required.rzWhisper cannot return `char` timestamps, only word level or segment level timestamps. Use `return_timestamps='word'` or `return_timestamps=True` respectively.) riloggerwarningupdatepopgetattrrxrGry ValueErrorhasattrr|r}) rorurvignore_warningrzr}r{rwpreprocess_paramsforward_paramspostprocess_params type_warnings r!_sanitize_parametersz7AutomaticSpeechRecognitionPipeline._sanitize_parameterss;  %yy::>- 99 11 4L |,2@ . /  &3B / 0  /  ! !/"5"56G"H Io. 4*D 1 =040D0DN, - 4. 5 A*...N; '484L4LN0 1  %3A / 0  &yy-- !LMM4C 0 10?N, - 4, -'$:P:PRe2f 1 ]T5K5K5]5]   (yyI%*; !hiiyyM).?6.I !uvvyyE!&7?O&O `yy--2Cv2M _3DN. /6G 2 3 .2DDDr#ct|jjdd}|jjjdk(r||jj z}|S)zSample stride per output.inputs_to_logits_ratiorlasr_ctc)rrYrgrhr1 hop_length)roalign_tos r! _align_toz,AutomaticSpeechRecognitionPipeline._align_toKsR 4::,,.FJ ::   ' ': 5 ..99 9Hr#c# Kt|tri|jds|jdr"tj|dj }n%t |d5}|j}dddt|tr t||jj}d}i}tr8ddl }t||jr|jj!}t#rwddl}t||j&j(rS|j+} | j,} | j.dk(r| j0ddk(r| dn| } | | j2d }t|t4r|j7d d}d |vrd |vsd |vs t9d|j7d d} | $|j7dd|j7d d} |j7d } |}| }| |jjk7rt;rddlm} n tAd| jCt|tDjFrjH|n|| |jjj!}|jj| z }nd}|h|d|dz|j0dkDr t9d|j0dtKtM|d|ztKtM|d|zf}t|tDjFjfstOdtQ|d|j.dk7r5tRjUd|j.d|jWd}|r ||dz }t|tJtXfr||g}|jZ}tKtM||jjz|z |z}tKtM|d|jjz|z |z}tKtM|d|jjz|z |z}|||zkr t9dt]||j||||j^D] }i|| y|jPdk(rR|j0d|jj`kDr,|j||jjdddd}nd|jPdk(r,|*|j||jjdd}n)|j||jjdd}|j^|jc|j^ }||jPd!k(r t9d"||d <d#di||y#1swYMxYww)$Nzhttp://zhttps://T)follow_redirectsrbrrr)rDr'rr'rawrDzWhen passing a dictionary to AutomaticSpeechRecognitionPipeline, the dict needs to contain a "raw" key containing the numpy array or torch tensor representing the audio and a "sampling_rate" key, containing the sampling_rate associated with that arraypath) functionalztorchaudio is required to resample audio samples in AutomaticSpeechRecognitionPipeline. The torchaudio package can be installed through: `pip install torchaudio`.zStride is too large for inputz9We expect a numpy ndarray or torch tensor as input, got ``zSWe expect a single channel audio input for AutomaticSpeechRecognitionPipeline, got z6. Taking the mean of the channels for mono conversion.axisz.Chunk length must be superior to stride lengthr^Flongestr%)r' truncationpaddingr(r)r&r*r_z8Stride is only usable with CTC models, try removing it !r,)2 isinstancestr startswithhttpxgetcontentopenreadbytesrr1r'r torchTensorcpunumpyr torchcodecdecoders AudioDecoderget_all_samplesdatandimr- sample_ratedictrrr torchaudior ImportErrorresamplerBndarray from_numpyrr TypeErrorrirrmeanfloatrr=r+ n_samplesr/)ror0rurvfrextrarr_audio_samples_array_inputsin_sampling_rateFrrr2r3r4itemr:s r! preprocessz-AutomaticSpeechRecognitionPipeline.preprocessYs fc "  +v/@/@/L6DAII&$'&1VVXF& fe $ )?)?)M)MNF   &%,,/++- " $ &*"5"5"B"BC!'!7!7!9(,,&,kkQ&66<<?a;OU[#)NrfrerelogitsdimrrdzUnsupported model type .)rrrikeysrr|boolrrYrgeneratercatrrrsetrrlr?tensorrargmaxrtupler"rF)ro model_inputsr}r{rwrrrr,r0rrout segment_listsegmentrrrfull_seq gen_configlang_idstoken_idoutputsrrrs r!_forwardz+AutomaticSpeechRecognitionPipeline._forwards%))*:DA!!(D1!%%lD9 ""9-  *"8OP P 996 6 </%))*:;</%)).9 LLXL]L]L_K`b !2 pWT=S=SUhjo5p  TYY2C%C7;9=O$56#/97;7M7M 34(( l"[77+=#>"[\($(&,K%8N^_CFD)kV.C !45(yy--%$*CM"z&$'?JRXDX &j1HHQK!)!Q!9:DVT:R6+#6X^%4%8%89LdNdNd%e ":|<'*:+@+@+G+G+I'JH,4OO,=*#+x#75:\\8*5MC N$)* YY0 0 **L,<,rfrer_r^rrr*rr)r}r{time_precisionrrr)r~ start_offset end_offsetrfrd group_tokensFskip_special_tokensT)routput_char_offsets char_offsetsr~rrr)text timestampchunksr,rr)#rir+rbfloat16float16r/float32rrrrRrGr1 chunk_lengthrYrgmax_source_positionsr'r unsqueezer _decode_asrrB concatenatesqueezerZ decode_beamsdecode_get_word_offsetsreplace_word_delimiter_charrrlistritems)(ro model_outputsrzr}r{optional final_itemskeyrrrtotal_nrrright_nrr'outputr2r3r4r lang_tokenrbeams chunk_offsetoffsetsr~rrr decode_kwargsrrrstartstoprkvs( r! postprocessz.AutomaticSpeechRecognitionPipeline.postprocessXs ))}4h($ &Gs|!!ennemm%DD  6<<> **,[[40F!dii3I&I'-$u "E/ago.   u % & dii9,1+t~~NE YY+ +!33@@4::CTCTCiCiiN 22@@M' Lv%;A(;K8I{L.I=0K M1L'0+|'KF8$ L+]F F*"("3";;=A-&-&7&7&:G%,%6%6q%9%<%<6(CSCYCY%<%Z +099j&BR5SY[+\x( ]"^^77"3 /- 8ND(NN;Q7EMM!$E 99 %%!#-DLL--eF~FE8A;D  %Qx{ 8Dk4D4<NND,^h#ijk YY+ +"&))u"4 8>((iDWi[hiD /$..///BX\`m "%."nn>>wHrHrsG 2P!PF~~H ]^,x7//===L)H4..<<< t,='>eUY][\ ]"(HX D!# #F JJx & JJx & JJy$ ' JJx & JJ)4 0 JJy$ '  #1a" # #22E22r#)NNNN)NNNNNN)rN)FN)NNN)rk __module__ __qualname____doc___pipeline_calls_generate_load_processor_load_image_processor_load_feature_extractor_load_tokenizerr_default_generation_configrrr rrnrBrrrrrrsrpropertyrrrr __classcell__)rjs@r!rTrTpsP'R $O!"O!1"LP04=A48 W W!!;S!@ADHW'- W -s23d: W c>)*T1 W292rzzE1C7$>92#92RVW[\_ad\dWeRf92zPEd  H:Ts4ljnn3-1D[n3r#rT)N)& collectionsrtypingrrrrrrB generationrtokenization_pythonr utilsr r r r audio_utilsrbaser pyctcdecoder!feature_extraction_sequence_utilsrmodeling_utilsr get_loggerrkrrmodels.auto.modeling_autorr"r=rRrTrr#r!r0sq$,, )5aa$0L0   H %T(42V 3V 3r#