^j9dZddlZddlmZddlmZddlZddlmZddlm Z ddl m Z dd l mZmZmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZmZmZm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4m5Z5m6Z6e!jne8Z9edeGddeZ:GddejvZ<GddejvZ=Gd d!e*Z>Gd"d#e,Z?Gd$d%ejvZ@Gd&d'eZAeGd(d)eZBed*Gd+d,eBZCeGd-d.eZDeGd/d0eDZEed1Gd2d3eBeZFGd4d5ejvZGGd6d7ejvZHeGd8d9eZIed:Gd;deHZKed?Gd@dAe6eJZLgdBZMy)CzPyTorch Parakeet model.N)Callable) dataclass)nn)initialization)ACT2FN) CompileConfigGenerationMixinGenerationMode)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingCausalLMOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)capture_outputs) AutoModel)%FastSpeech2ConformerConvolutionModule)LlamaAttentioneager_attention_forward)ParakeetCTCConfigParakeetEncoderConfigParakeetRNNTConfigParakeetTDTConfig)ParakeetRNNTDecoderCacheParakeetRNNTGenerationMixinParakeetTDTGenerationMixinz Extends [~modeling_outputs.BaseModelOutputWithPooling] to include the output attention mask since sequence length is not preserved in the model's forward. ) custom_introc:eZdZUdZdZej dzed<y)ParakeetEncoderModelOutputa attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices after sequence compression. Returned because the sequence length may differ from the input sequence length. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Nattention_mask)__name__ __module__ __qualname____doc__r+torchTensor__annotations__x/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/parakeet/modular_parakeet.pyr*r*5s+/NELL4'.r4r*ceZdZUejed<d deffd Ze d dedzdejfdZ ejdejfdZ xZ S) $ParakeetEncoderRelPositionalEncodinginv_freqNconfigct||j|_||_|j ||}|j d|dy)Ndevicer8F) persistent)super__init__max_position_embeddingsr9.compute_default_relative_positional_parametersregister_buffer)selfr9r<r8 __class__s r5r?z-ParakeetEncoderRelPositionalEncoding.__init__LsN '-'E'E$ FFvV\F] ZeDr4returncd}d|tjd|jdtjj |tj |jz zz }|S)Ng@?rrdtype)r<rI)r0arange hidden_sizeint64tofloat)r9r<baser8s r5rAzSParakeetEncoderRelPositionalEncoding.compute_default_relative_positional_parametersSsd   Q 2 2AU[[ILLTZbgbmbmLn$$%  r4 hidden_statesc|jd}tj|dz | d|j}|jddddfj j |jdddj|j}|ddddfj }t|jjtr/|jjdk7r|jjnd}t|d5|j |j zjdd }|j}|j} tj|| gd } | j g| jdd d} ddd j|j" S#1swY%xYw) Nr r;rmpscpuF) device_typeenabledrdimrH)shaper0rJr<r8rNexpandrM isinstancetypestrr transposesincosstackreshaperI) rCrP seq_length position_idsinv_freq_expandedposition_ids_expandedrUfreqsr`ra pos_embeds r5forwardz,ParakeetEncoderRelPositionalEncoding.forwardbs"((+ ||JNZKML`L`a MM$4- ( . . 0 7 7 8K8KA8NPRTU V Y YZgZnZn o !-T4] ; A A C-..33S9m>R>R>W>W[`>`  % %   U C E&,,.1F1L1L1NNYYZ[]^_E))+C))+C S#JB7I) ))D9??3B+?DDI  E||-"5"5|66 E Es B F::GNNN) r,r-r.r0r1r2r"r? staticmethodrAno_gradrj __classcell__rDs@r5r7r7IsxllE4E/3 %,    U]]_7U\\77r4r7c*eZdZdeffd ZdZxZS)ParakeetEncoderFeedForwardr9c`t|tj|j|j |j |_t|j|_ tj|j |j|j |_ |j|_ y)Nbias) r>r?rLinearrKintermediate_sizeattention_biaslinear1r hidden_act activationlinear2activation_dropoutrCr9rDs r5r?z#ParakeetEncoderFeedForward.__init__|s| yy!3!3V5M5MTZTiTij  !2!23yy!9!96;M;MTZTiTij "(";";r4c|j|j|}tjj ||j |j }|j|}|S)Nptraining)r{ryr functionaldropoutr}rr|)rCrPs r5rjz"ParakeetEncoderFeedForward.forwardsU ](CD  --mt?V?Vaeanan-o  ]3 r4)r,r-r.r"r?rjrorps@r5rrrr{s<4<r4rrc&eZdZddeffd ZxZS) ParakeetEncoderConvolutionModuler9c&t|||yrk)r>r?)rCr9 module_configrDs r5r?z)ParakeetEncoderConvolutionModule.__init__s /r4rk)r,r-r.r"r?rorps@r5rrs0400r4rc eZdZdZdedeffd Z d dejdejdzdejdzd e e d e ejejff d Z d Z xZS)ParakeetEncoderAttentionztMulti-head attention with relative positional encoding. See section 3.3 of https://huggingface.co/papers/1901.02860.r9 layer_idxct|||d|_tj|j |j |jzd|_tjtj|j |j|_ tjtj|j |j|_ y)N)rFrt)r>r? is_causalrrvrKnum_attention_headshead_dimrelative_k_proj Parameterr0zerosbias_ubias_vrCr9rrDs r5r?z!ParakeetEncoderAttention.__init__s 95!yy););V=W=WZ^ZgZg=gnstll5;;v/I/I4==#YZ ll5;;v/I/I4==#YZ r4NrPposition_embeddingsr+kwargsrEc |jdd}|\}}||d|jf}|j|j|j dd} |j |j|j dd} |j |j|j dd} tj|jjt} | |jjd|jjd|jz} | |jjd|jjd|jz}|j|}|j|d|jj|j}||j!ddddz}|j#|}|dd|f}||j$z}|)|j'|j)t+d}| |f| | | ||j,sdn |j.|j$d |\}}|j0g|dj3}|j5|}||fS) NrRr rrr.z-inf)querykeyvaluer+rscaling)rZrq_projviewr_k_projv_projr get_interfacer9_attn_implementationrrrrrpermute _rel_shiftr masked_fill_ logical_notrNrattention_dropoutrc contiguouso_proj)rCrPrr+r input_shape batch_sizerd hidden_shape query_states key_states value_statesattention_interfacequery_states_with_bias_uquery_states_with_bias_vrelative_key_states matrix_bd attn_output attn_weightss r5rjz ParakeetEncoderAttention.forwardsj$))#2. !, J"JDMMB {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST (?(M(M KK , ,.E) $0$++2B2B t{{..4==3 $  $0$++2B2B t{{..4==3 $  #223FG166z2t{{GfGfhlhuhuv-/B/J/J1aQRTU/VV OOI. c;J;./  ,  %"..~/I/I/KUSY][I%8  % *$#}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r4c|j\}}}}tjj|d}|j ||d|}|ddddddfj ||||}|S)ztRelative position shift for Shaw et al. style attention. See appendix B of https://huggingface.co/papers/1901.02860.)r r)padrRNr )rZrrrr)rCattention_scoresr num_heads query_lengthposition_lengths r5rz#ParakeetEncoderAttention._rel_shiftst?O?U?U< I|_==,,-=6,J+00YLY+Aq!"H5:::yR^`opr4rk)r,r-r.r/r"intr?r0r1rrtuplerjrrorps@r5rrs~[4[[/3 7)||7)#\\D07) t+ 7) +, 7) u||U\\) * 7)r r4rceZdZdeffd Zdej dejfdZ d dej dej fdZ xZ S) ParakeetEncoderSubsamplingConv2Dr9c t||j|_|j|_|j |_|jdz dz|_ttj|j|_ tj|_|j j#tj$d|j|j|j |j|j j#tj&t)|jdz D]}|j j#tj$|j|j|j|j |j|j|j j#tj$|j|jd|j j#tj&|j*|j |jzz}tj,|j |z|j.d|_y)Nr r) kernel_sizestridepadding)rrrgroupsrTrt)r>r?subsampling_conv_kernel_sizersubsampling_conv_stridersubsampling_conv_channelschannelsrrmathlog2subsampling_factor num_layersr ModuleListlayersappendConv2dReLUrange num_mel_binsrvrKlinear)rCr9i out_lengthrDs r5r?z)ParakeetEncoderSubsamplingConv2D.__init__s !>>44 88 ((1,2 dii(A(ABCmmo   IIaD4D4DT[[bfbnbn o  2779%t*+ *A KK   MMMM $ 0 0;; LL==   KK  ryy STU V KK  rwwy ) *"((T[[$//-IJ ii @ @: MvOaOahlm r4 input_lengths conv_layerct|drR|jdk7rC|j}|jd}|jd}||dz|dz|z |zdz}|S|S)Nr)r r rr )hasattrrrr)rCrrrrroutput_lengthss r5_get_output_lengthz3ParakeetEncoderSubsamplingConv2D._get_output_lengthsx :x (Z->->&-H ((G$003K&&q)F+gaj871:E SX^^abbN! !r4input_featuresr+c|jd}||jdnd}|jD]}||}t|tj s&|)|j ||}|jd}tj||j|dddfk}||ddddddfz}|jddj|jd|jdd}|j|}|S)Nr rRrr;r) unsqueezesumrr\rrrrZr0rJr<r_rcr)rCrr+rPcurrent_lengthslayercurrent_seq_length channel_masks r5rjz(ParakeetEncoderSubsamplingConv2D.forward s&003 4B4N.,,R0TX[[ @E!-0M%+0J"&"9"9/5"Q%2%8%8%;"LL!3N!??  @&//15==m>Q>QRS>TVcViVijkVlnpq  M2 r4rk) r,r-r.r"r?r0r1rrrrjrorps@r5rrsI!n4!nF  ")) ellELLr4rc eZdZd dededzffd Z d dejdejdzdejdzdee d ejf d Z xZ S) ParakeetEncoderBlockNr9rct|d|_t||_t |||_t||_t||_ tj|j|_ tj|j|_tj|j|_tj|j|_tj|j|_y)NF)r>r?gradient_checkpointingrr feed_forward1r self_attnrconv feed_forward2r LayerNormrKnorm_feed_forward1 norm_self_att norm_convnorm_feed_forward2norm_outrs r5r?zParakeetEncoderBlock.__init__$s &+#7?1&)D4V< 7?"$,,v/A/A"B\\&*<*<=f&8&89"$,,v/A/A"B V%7%78 r4rPr+rrrEc x|}|j|j|}|d|zz}|j|}|jd|||d|\}}||z}|j |j ||} || z}|j |j|} |d| zz}|j|}|S)Ng?)rPr+r)r+r3) rrrrrrrrr) rCrPr+rrresidualnormalized_hidden_statesr_ conv_output ff2_outputs r5rjzParakeetEncoderBlock.forward3s!**4+B+B=+QR  3#66 #'#5#5m#D ' 2) 3    Q & 3 ii} =ni] % 3 ''(?(? (NO %j(88  m4 r4rkrl) r,r-r.r"rr?r0r1rrrjrorps@r5rr#sx 94 9t 9$/337 || t+#\\D0  +,   r4rceZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ dZdZeedZej(fd Zd ej,fd Zdd ej,ded zfdZxZS)ParakeetPreTrainedModelr9modelraudioTrF)rP attentionsct||t|jdd}t |t rEt j|jd|t j|jd|yt |tr<|j|j}t j|j|yy)Ninitializer_rangeg{Gz?r)meanstd)r> _init_weightsgetattrr9r\rinitnormal_rrr7rAcopy_r8)rCmoduler buffer_valuerDs r5rz%ParakeetPreTrainedModel._init_weightshs f%dkk#6= f6 7 LLSc : LLSc :  D E!PPQWQ^Q^_L JJv 5Fr4rct|jd|j}|j}|j}t t j |j}|dz dzdz}||z }|}t|D]Q} tj|jtj|z|dz}tj|}S|jtjS)Nencoder_configr rrHrG)rr9rrrrrrrr0divrMrNfloor) rCrr rrr all_paddingsadd_padlengthsrs r5_get_subsampling_output_lengthz6ParakeetPreTrainedModel._get_subsampling_output_lengthts .> L$AA 77>#D#DEF #aA-1 ,z" +Aii  = GPSVVGkk'*G +zz z**r4Nr+ target_lengthc|j|jd}||n|j}tj||j |dddfk}|S)z Convert the input attention mask to its subsampled form. `target_length` sets the desired output length, useful when the attention mask length differs from `sum(-1).max()` (i.e., when the longest sequence in the batch is padded) rRNr;)rrmaxr0rJr<)rCr+rr max_lengths r5_get_output_attention_maskz2ParakeetPreTrainedModel._get_output_attention_masksc <<^=O=OPR=ST&3&?]^EWEWEY j9N9NOR`abdhahRiir4rk)r,r-r.r!r2base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_flat_attention_mask_supports_sdpa_supports_flex_attn_supports_flash_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr0rnrr1rrrrorps@r5rrRs &O&*#/0$(!N!!"&-. U]]_ 6 6+ELL+"  VY\`V` r4rz{ The Parakeet Encoder model, based on the [Fast Conformer architecture](https://huggingface.co/papers/2305.05084). ceZdZUeed<dZdeffd Zeee e d de jde jdzde deed ef d ZxZS) ParakeetEncoderr9encoderc t||||_d|_|j|_|j |_|j |_|jrtj|jnd|_ t||_ t||_t!j"t%|j&Dcgc]}t)||c}|_|j-ycc}w)NFrG)r>r?r9rrdropout_positions layerdrop scale_inputrsqrtrK input_scaler subsamplingr7encode_positionsrrrnum_hidden_layersrr post_initrs r5r?zParakeetEncoder.__init__s   &+#~~ !'!9!9))>> from transformers import AutoProcessor, ParakeetEncoder >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> encoder = ParakeetEncoder.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> encoder_outputs = encoder(**inputs) >>> print(encoder_outputs.last_hidden_state.shape) ``` rNr rrRrFT)r+r)last_hidden_stater+)r.r-r/rrrrr)rrZrr[r_rr0randr*r*r) rCrr+r2rrPr output_mask encoder_layerto_dropdropout_probabilitys r5rjzParakeetEncoder.forwardsF((H %(8(88 "33MB --mt||VZVcVc-d  mm33 4#9#9DMM4   %99.XeXkXklmXn9oK(2215<0JOd;??,  jn  r4)NT)r,r-r.r"r2rr?rrrrr0r1boolrrr rjrorps@r5r&r&s "!!4&/3&* B  B  t+B  $ B +, B  B  B r4r&ceZdZUdZej ed<dZeejdzed<dZ eeejdzed<dZ eeejdzed<y)ParakeetCTCGenerateOutputaz Outputs of Parakeet CTC model generation. Args: sequences (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, generated_length, hidden_size)`. sequencesNlogitsrrP) r,r-r.r/r0 LongTensorr2r?r FloatTensorrrPr3r4r5r=r=sm&.2FE%## $t +29=JeE--./$6=<@M5u0012T9@r4r=c"eZdZdZfdZxZS)ParakeetGenerateOutputz` Deprecated alias for ParakeetCTCGenerateOutput. Use ParakeetCTCGenerateOutput instead. cNt||i|tjdy)Nz`ParakeetGenerateOutput` is deprecated and removed starting from version 5.11.0; please use `ParakeetCTCGenerateOutput` instead.)r>r?logger warning_once)rCargsrrDs r5r?zParakeetGenerateOutput.__init__s) $)&) O r4)r,r-r.r/r?rorps@r5rCrCs  r4rCzS Parakeet Encoder with a Connectionist Temporal Classification (CTC) head. cXeZdZUeed<deffd Zee d dejdejdzdejdzde e de f d Z ej ddejdejdzd ed edzde e deej$zf d ZxZS)ParakeetForCTCr9ct||tj|j|_t j|jj|jd|_ |jy)Nr r) r>r?r from_configr r'rConv1drK vocab_sizectc_headr1r~s r5r?zParakeetForCTC.__init__&sY   ,,V-B-BC  &"7"7"C"CVEVEVdef  r4Nrr+labelsrrEc ||jdd|jd||d|}|j}|j|j ddj dd}d}|+|j j d} ||jjk7} | j d} |j| } tjj|dtjj d d} tjj j#d 5tjj%| | | | |jj|jj&|jj( }dddt+|||j,|j. S#1swY,xYw)a Example: ```python >>> from transformers import AutoProcessor, ParakeetForCTC >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> outputs = model(**inputs) >>> print(outputs.loss) ```Nr2Trr+r rrR)rXrIrF)rV)blank reduction zero_infinity)lossr?rPrr3) setdefaultr'r5rNr_r+rr9 pad_token_id masked_selectrr log_softmaxr0float32backendscudnnflagsctc_lossctc_loss_reductionctc_zero_infinityrrPr)rCrr+rOrencoder_outputsrPr?rUencoder_lengths labels_masktarget_lengthsflattened_targets log_probss r5rjzParakeetForCTC.forward.s:     5t <&$,, ))   (99 }66q!<=GG1M  -<<@@DO!DKK$<$<>> from transformers import AutoProcessor, ParakeetForCTC >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> predicted_ids = model.generate(**inputs) >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> print(transcription) ``` T return_dictrQrRrWr r4)r>r?rrPr3) get_compiled_call__call__r?argmaxrrZr9rWr=rrP) rCrr+rgrhr model_forwardoutputsr>s r5generatezParakeetForCTC.generatetsBCQB\..~>bfbobo $}"/# ))# # NN))b)1   %!<<^[d[j[jkl[m<nN)-)A)AI~o & ",#~~"--%33  r4rl)NFN)r,r-r.r!r2r?rrr0r1rrrrjrnr;r r=r@rprorps@r5rIrIs 0/3&* B  B  t+B  t# B +, B  B B HU]]_/3(-/3 9 9 t+9"& 9 &, 9 +, 9 #U%5%5 599r4rIcneZdZdZdeffd Z d dejdedzdejfdZ xZ S) ParakeetRNNTDecoderz'LSTM-based prediction network For RNN-Tr9cxt||j|_tj|j |j |_tj|j |j |jd|_ tj|j |j |_ y)NT) input_sizerKr batch_first) r>r?blank_token_idr EmbeddingrMdecoder_hidden_size embeddingLSTMnum_decoder_layerslstmrvdecoder_projectorr~s r5r?zParakeetRNNTDecoder.__init__s $33f&7&79S9STGG112200   "$6+E+EvGaGa!br4N input_idscacherEc|>|dddf|jk(}|jr|jr |jS|j |}|8|j}|s|j ||j |jf}nd}|j||\}\}} |j|} |(rnd} |j| || | |jS| S)NrR)mask) rvis_initializedallrrylazy_initialization hidden_state cell_stater|r}update) rCr~r blank_mask embeddingswas_initializedhidden_cell_states lstm_outputrrdecoder_outputrs r5rjzParakeetRNNTDecoder.forwards  "1b5)T-@-@@J## (8{{"^^I.   #22O"))*5"'"4"4e6F6F!G !% 26))JHZ2[/ /lJ// <  "1J;tD LLzL M;; r4rk) r,r-r.r/r#r?r0r@r%r1rjrorps@r5rrrrsJ1 c1 c26##($.  r4rrceZdZdZdeffd ZdejdejdeejejffdZ xZ S)ParakeetRNNTJointNetworkzPJoint network that combines encoder and decoder outputs to predict token logits.r9ct|t|j|_t j |j|j|_ |j|_yrk) r>r?rrzr{rrvrxrMheadr~s r5r?z!ParakeetRNNTJointNetwork.__init__sK  !2!23IIf88&:K:KL  ++r4decoder_hidden_statesencoder_hidden_statesrEcL|j||z}|j|Srk)r{r)rCrr joint_outputs r5rjz ParakeetRNNTJointNetwork.forwards( 'r?rrKr r'rrvrKrxencoder_projectorrrdecoderrjointmax_symbols_per_stepr1r~s r5r?zParakeetForRNNT.__init__s}   ,,V-B-BC !#6+@+@+L+LfNhNh!i*62 -f5 $*$?$?! r4Nrr+rrEc p|jd||d|}|j|j|_|S)NrQr3)r'rr5 pooler_output)rCrr+rras r5get_audio_featuresz"ParakeetForRNNT.get_audio_featuressJ'$,, ))   )-(>(>?`?`(a%r4decoder_input_idsruse_decoder_cacherarOc ||jd ||d|}|r|t|j}|j||} |j |j dddddddf| dddddddfj d} d} ||jjd} |jd | dddt| jf|| ||jjk7jd|jjd|} t| | |j|j |j|j |S) a? decoder_input_ids (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Decoder input token ids for single-step inference. decoder_cache (`ParakeetRNNTDecoderCache`, *optional*): Decoder LSTM cache. When provided and initialized, the cached `decoder_output` is reused (e.g. during blank-skipping) instead of running the decoder. When `input_ids` is provided, the decoder runs and the cache is updated in-place. use_decoder_cache (`bool`, *optional*): Whether to use a decoder cache. When `True` and `decoder_cache` is `None`, a new cache is created automatically during the forward pass. encoder_outputs (`tuple(torch.FloatTensor)`, *optional*): Pre-computed encoder outputs (last_hidden_state, pooler_output, hidden_states, attentions, attention_mask). Can be a tuple or `ParakeetEncoderModelOutput`. Example: ```python >>> from transformers import AutoProcessor, ParakeetForRNNT >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-rnnt-0.6b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForRNNT.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> outputs = model(**inputs) ``` NrQrrrrrR)r?rO logit_lengths label_lengthsrvrUr?r5rrPrrr3)rr%r9rrrsqueezer+r loss_functionrrrvrr5rPr) rCrr+rrrrarOrrr?rUrs r5rjzParakeetForRNNT.forward*svX  "5d55--O !64T[[AM $ ->m T"1"?"?1dA "N"74A "F  '!*    +::>>rBM%4%%a!;3}'8'8':#;!;;<+%)C)CCHHL#{{99   D"-??)77)77&11'  r4rkNNNNNNN)r,r-r.r#r2rr GREEDY_SEARCH_supported_generation_modesr?rr0r1rrr*rrr@r%r;rrArrjrorps@r5rrsc ./#1#?#?"@1/3    t+ +,  $  /3.2599=)-X\&*N  t+N  t+N !++d2 N 0$6 N  $; N 4eEr?rrvrxrMlen durationsrr~s r5r?z ParakeetTDTJointNetwork.__init__s?  IIf88&:K:KcRXRbRbNc:cd r4)r,r-r.r/r$r?rorps@r5rr}s e0eer4rzG Parakeet Encoder with a TDT (Token Duration Transducer) head. c eZdZUeed<deffd Zee ddejdzdejdzdejdzde dzde dzd e eejzdzd ejdzd eed efd ZxZS)ParakeetForTDTr9cdt||t||_|j yrk)r>r?rrr1r~s r5r?zParakeetForTDT.__init__s&  ,V4  r4Nrr+rrrrarOrrEc ||jd ||d|}|r|t|j}|j||} |j |j dddddddf| dddddddfj d} d} ||jd | dd|jjf| d|jjdf||jjd||jjk7jd|jj|jjd|} t| | |j|j |j |j"| S) a? decoder_input_ids (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Decoder input token ids for single-step inference. decoder_cache (`ParakeetRNNTDecoderCache`, *optional*): Decoder LSTM cache. When provided and initialized, the cached `decoder_output` is reused (e.g. during blank-skipping) instead of running the decoder. When `input_ids` is provided, the decoder runs and the cache is updated in-place. use_decoder_cache (`bool`, *optional*): Whether to use a decoder cache. When `True` and `decoder_cache` is `None`, a new cache is created automatically during the forward pass. encoder_outputs (`tuple(torch.FloatTensor)`, *optional*): Pre-computed encoder outputs (last_hidden_state, pooler_output, hidden_states, attentions, attention_mask). Can be a tuple or `ParakeetEncoderModelOutput`. Example: ```python >>> from transformers import AutoProcessor, ParakeetForTDT >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-tdt-0.6b-v3" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForTDT.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> outputs = model(**inputs) ``` NrQrrr.rR) token_logitsduration_logitsrOrrrvrrr3)rr%r9rrrrrrMr+rrWrvrrr5rPr) rCrr+rrrrarOrrr?rUs r5rjzParakeetForTDT.forwardsX  "5d55--O !64T[[AM $ ->m T"1"?"?1dA "N"74A "F  '!*    %4%% #C)A4;;+A+A)A$AB &sDKK,B,B,D'D E-<<@@D%)A)AAFFrJ#{{99++//  D"-??)77)77&11'  r4r)r,r-r.r$r2r?rrr0r1r@r%r;r*rrArrrrjrorps@r5rrs 0 /3.2599=)-X\&*O  t+O  t+O !++d2 O 0$6 O  $; O 4eErs[ $! &!HH9[[F&H5hJss   H %   /!; /  //7299/7d  0'L0 L ~L ^BryyBJ,5,^;o;;| ] -]  ] @ A A A4   6      K,oK  K\-"))-`'ryy'$ :3: :& n -/Jn  n b e6 e Z /Z  Z z pr4